Pipeline

A pipeline project is a directory of .sql files containing CREATE SOURCE, CREATE INCREMENTAL VIEW, CREATE SINK, and START PIPELINE statements. The CLI loads, validates, and runs them as a unit.

init — scaffold a project

krishiv pipeline init --name my-pipeline --dir ./my-pipeline

Produces:

my-pipeline/
├── pipeline.sql          # entry point (sources → views → sinks)
├── sources/
│   └── orders.sql
├── views/
│   └── order_totals.sql
└── sinks/
    └── totals_to_iceberg.sql

Edit the scaffolded files, then run.

dry-run — validate without executing

krishiv pipeline dry-run ./my-pipeline

Loads every .sql file, splits on ;, ignores -- comment lines, normalises whitespace, and validates each statement against the live catalogs. Does not execute START PIPELINE.

run — execute

krishiv pipeline run ./my-pipeline
krishiv pipeline run ./my-pipeline --full-refresh    # reset IVM first

Runs all statements in order. --full-refresh calls reset_ivm_job(name) for every incremental view before the first tick. START PIPELINE returns the sink's memory output (for sinks that emit to memory); use --full-refresh to force a state reset.

See also

• Pipeline DDL
• Parquet → SQL aggregation recipe
• CLI Overview