SQLMesh: The Open-Source Analytics Framework Bringing Real Software Engineering to the Data Warehouse

I have spent twenty years building data infrastructure, and the number of times I have watched someone accidentally clobber a production dbt model with a bad dbt run is genuinely embarrassing. The pipeline fails. The dashboards go dark. The oncall engineer scrambles. Eventually you trace it back to a developer who ran dbt run --select my_model+ against production because their local environment did not exist. This is not a people problem. It is a tooling problem.

SQLMesh exists to fix this class of problem. It is an open-source analytics engineering framework built by Tobias Mao and team, drawing on hard experience at Lyft and LinkedIn. It does what dbt does (SQL transformations, data modeling, lineage) but with a fundamentally different architectural philosophy: treat the data warehouse the way Terraform treats cloud infrastructure. Declare state. Plan before you apply. Never modify production without an explicit promotion step.

If you have been following the analytics engineering space, you know dbt went from scrappy startup favorite to near-universal standard in about four years. That dominance created a complacency that SQLMesh is now exploiting. I am not saying dbt is bad. I use it on several projects right now. But SQLMesh solves real problems that dbt has punted on for years, and if you are running a data team at any reasonable scale, you owe it to yourself to understand the difference.

What SQLMesh Actually Is

SQLMesh is a data transformation framework, an analytics engineering tool, and a CI/CD system for your data warehouse all in one. You define models in SQL or Python, you declare their relationships, and SQLMesh figures out how to build, test, and promote those models through environments.

The core data model is similar to dbt: you have models (SQL SELECT statements saved as files), you have tests, you have macros, and you have a DAG that SQLMesh builds from your ref() calls. If you have a dbt project, SQLMesh can read most of it directly. That compatibility is intentional; the team did not want to make migration a cliff.

What is different is everything underneath. Where dbt is essentially a jinja-templated SQL runner that lets the warehouse figure out what to build, SQLMesh maintains state. It knows which models have run. It knows which models changed since the last run. It knows which downstream models need to be backfilled as a result. It can answer the question “if I change this model, what exactly will I need to recompute?” before you run anything.

Virtual Environments: The Feature That Changes Everything

The single biggest architectural difference in SQLMesh is virtual environments. Understanding this feature is the key to understanding why SQLMesh exists.

In dbt, an “environment” is really just a target: a schema name or a database connection. When you run dbt run --target dev, dbt creates tables in your dev_schema and you test against those. Your production models live in prod_schema. The problem is that these are physically separate objects. If you have a large model that costs $50 to build in Snowflake, you are paying $50 every time you build it in dev. If the model is based on a huge fact table, your dev environment either runs against a sample or it runs the full cost.

SQLMesh’s virtual environments are different. They use views to create logical representations of your models. When you create a dev environment in SQLMesh, it does not rebuild all your models. Instead, it creates views that point to either your existing production physical tables (for unchanged models) or to new physical tables (for models you have changed). Changed models get built in isolation. Unchanged models are just views pointing to the production objects.

This means your dev environment is cheap. You only pay compute for the models you actually changed. Everything else is zero-cost views. And because the views are logically separate per environment, you cannot accidentally pollute production by running in dev.

I cannot overstate how much this changes the daily workflow. In dbt, running in dev against full-scale production data costs real money and real time. In SQLMesh, a dev run that touches three models out of a 200-model DAG rebuilds three models and views everything else. The cost drops by 98%. The iteration cycle drops from “go get coffee” to “wait a few seconds.”

The Plan/Apply Workflow

If virtual environments are SQLMesh’s best feature, the plan/apply workflow is its most distinctive behavior. It is the same concept as Terraform: before you execute anything, you generate a plan that shows exactly what will happen.

When you run sqlmesh plan, SQLMesh computes:

• Which models changed since the last run (based on fingerprints of the SQL, not file modification times)
• Which downstream models are affected by those changes and need to be backfilled
• What the categorized impact is: breaking change (downstream tables need full backfill), non-breaking change (downstream tables do not need to be rebuilt), or metadata-only change (no recompute required at all)

The categorization is important. SQLMesh distinguishes between a breaking change (you changed the join condition or removed a column) and a non-breaking change (you added a WHERE clause that filters differently but the output schema is the same). Non-breaking changes do not require downstream backfills. Breaking changes do. This is information dbt simply does not have, because dbt does not understand the semantic impact of your changes.

You review the plan, confirm it looks right, and then run sqlmesh apply to execute it. Nothing touches production until you say so. The workflow looks like this in practice:

# create a dev environment
sqlmesh plan dev

# review the plan: what changed, what will be rebuilt, what is the cost estimate
# SQLMesh shows you the full impact graph

# apply the changes to dev (rebuilds only what changed)
sqlmesh apply dev

# when you are happy, promote to production
sqlmesh plan prod
sqlmesh apply prod

This is fundamentally safer than dbt’s model. In dbt, dbt run is always a side-effectful operation. In SQLMesh, sqlmesh plan is a read-only operation you can run as many times as you want.

State Management and Incremental Models

Incremental models are where dbt causes the most grief at scale. In dbt, an incremental model runs a SELECT with a WHERE clause filtered to recent data, then merges or appends that to an existing table. The logic for detecting what counts as “recent” is entirely your responsibility. If your pipeline skips a day, you might miss data. If the watermark logic is wrong, you might double-load. If you need to do a full historical backfill, you have to manually run dbt run --full-refresh and hope for the best.

SQLMesh handles incremental models through a concept called intervals. You declare the granularity of your model (daily, hourly, by event time) and SQLMesh maintains state about which intervals have been processed. If a pipeline run fails, SQLMesh knows exactly which intervals are missing and will pick up exactly where it left off. If you need to backfill a date range, you specify --start and --end and SQLMesh will process every unprocessed interval in that range, in order, without you having to reason about watermarks.

This matters enormously for late-arriving data scenarios and for teams that have unreliable pipelines (which is most teams). Instead of “I think my incremental logic is correct,” you get “SQLMesh knows what it processed and will fill the gaps.”

The state is stored in a SQLMesh state database: by default a local SQLite file for development, and a shared backend (Postgres, MySQL, or cloud database) for production. This is conceptually similar to Terraform’s state file. It is the source of truth for what has and has not been processed.

Python Models and the Semantic Layer

One area where SQLMesh genuinely surpasses dbt is first-class Python model support. In dbt, Python models are an afterthought: they run as Snowflake Python worksheets or Databricks notebooks, with limited integration into the broader model graph. The developer experience is rough.

In SQLMesh, Python models are proper citizens. You define them as Python functions that return a DataFrame or write directly to a table, and they participate in the same dependency graph, the same virtual environments, and the same plan/apply workflow as SQL models. You can mix SQL and Python freely.

This matters for data teams doing feature engineering, model preprocessing, or complex transformations that SQL cannot express cleanly. The ability to have a SQL model feed into a Python model that feeds back into a SQL model, all within the same framework and CI/CD pipeline, is something I have wanted for years.

SQLMesh also has a built-in semantic layer, though this is more nascent than the transformation engine. You can define metrics and dimensions that get materialized as views or tables and queried through a consistent interface. This is the same space that dbt’s semantic layer occupies, though SQLMesh’s approach is more integrated with the execution engine.

For teams building data lakehouse architectures, SQLMesh has strong support for Apache Iceberg tables across Spark, Trino, and Snowflake Open Catalog, which makes it a viable option even if your data lives outside a traditional warehouse.

Platform Support: Where SQLMesh Actually Runs

SQLMesh supports essentially the same platforms as dbt: Snowflake, BigQuery, Databricks, Redshift, DuckDB, Postgres, Trino, Spark, and more. The DuckDB support is particularly good; SQLMesh uses DuckDB extensively for local development and testing, and running your full model suite against a DuckDB instance locally (without any cloud warehouse) is a first-class workflow.

For teams on Snowflake, BigQuery, or Databricks, the virtual environment feature has a meaningful cost implication. On Snowflake, SQLMesh virtual environments use clone-on-write semantics where available, which means changed models get dedicated storage but unchanged models reference shared physical data. On BigQuery, virtual environments are implemented as authorized views. The exact behavior varies by platform, but the conceptual model is consistent.

SQLMesh also integrates with orchestration tools like Airflow and Dagster. If you are already running Airflow or Dagster for pipeline orchestration, SQLMesh can plug in as the transformation layer within your existing DAGs. There is a native Airflow operator and a Dagster integration that respects SQLMesh’s execution model.

The CI/CD Story

The area where SQLMesh most clearly outcompetes dbt is CI/CD. With dbt, a CI run typically creates a separate schema, runs your models against it, runs your tests, and tears down the schema. This is expensive for large models and gives you no meaningful indication of production impact.

SQLMesh’s CI story is built around the plan/apply model. In a CI pipeline, you run sqlmesh plan against a staging environment. SQLMesh generates the impact analysis: which models would change, which need backfills, what the estimated row counts are. A human reviews this plan as part of the code review process. When the PR merges, sqlmesh apply promotes changes from staging to production.

Because the plan output shows the exact impact, reviewers can catch issues before they hit production. If a model change would trigger a full backfill of a 100-billion-row fact table, that shows up in the plan before anyone approves the PR. This is the kind of pre-flight check that dbt simply cannot provide.

For teams with data contracts as part of their governance process, SQLMesh’s plan output gives contract consumers visibility into what changes are coming before they land. The breaking change detection in SQLMesh is not perfect (it is still semantic analysis of SQL, not a runtime guarantee), but it is far better than nothing.

Migration from dbt: What It Actually Takes

If you have an existing dbt project and want to evaluate SQLMesh, the migration path is more reasonable than you might expect. SQLMesh has a dbt project loader that can read your existing dbt_project.yml, your model SQL files, and your ref() calls. For many projects, you can run sqlmesh init --dialect trino dbt://path/to/dbt/project and have SQLMesh parse your existing models.

The compatibility is not perfect. Some advanced Jinja macros that are dbt-specific will not translate directly. Custom materializations require rewriting. The dbt utils package has to be partially replaced. But for a project that uses the standard dbt model types (table, view, incremental, snapshot), the migration can be mostly mechanical.

The honest assessment: plan for two to four weeks of migration work for a medium-sized project (50-150 models). The early investment pays back quickly in saved CI costs and the elimination of the “who ran what against production” class of incidents.

If you are starting a new project and do not have dbt lock-in, SQLMesh is the clear choice today. If you have a large, mature dbt project with heavy use of packages and custom macros, evaluate carefully before committing to migration.

Data Observability Integration

SQLMesh generates lineage metadata that integrates with data catalog and lineage tools like OpenMetadata and DataHub. The integration is similar to what dbt provides through dbt docs: a model graph with column-level lineage, owner metadata, and test results.

For data observability platforms like Monte Carlo or Soda, SQLMesh emits the same types of events that dbt does: model run start, model run complete, test pass/fail. If you have observability tooling built around dbt events, the integration path to SQLMesh is straightforward because the observability platforms have added SQLMesh adapters in the past year.

One area where SQLMesh genuinely helps observability is the interval-based incremental model tracking. Because SQLMesh knows which intervals have been processed, you can query the SQLMesh state database to see exactly what data has landed and what has not. This is a significant improvement over trying to infer completeness from watermark columns in your raw tables.

When SQLMesh Wins, When dbt Is Still Fine

SQLMesh is not automatically better in every scenario. Here is my honest breakdown after running both in production contexts:

SQLMesh clearly wins when:

• You have large, expensive models where rebuilding everything in CI is cost-prohibitive
• Your team has been burned by prod/dev isolation failures in dbt
• You have complex incremental models with late-arriving data
• You are starting a new project with no dbt debt
• You have Python-heavy transformation logic that does not fit SQL well
• Your team practices infrastructure-as-code and values the plan/apply mental model

dbt is still the better choice when:

• You are a small team with a well-understood, small dataset where cost is not an issue
• You have a large investment in dbt packages (dbt-utils, dbt-expectations, etc.) that would need replacing
• Your organization has standardized on dbt and switching costs outweigh the benefits
• You rely heavily on the dbt Cloud platform’s hosted CI and documentation features
• Your team uses dbt Semantic Layer and MetricFlow deeply and the SQLMesh semantic layer is not mature enough for your needs

The ecosystem gap is real. dbt has four years of packages, integrations, and community tooling that SQLMesh is still building. If your workflow depends on third-party dbt packages for things like date spine generation, audit helpers, or unit testing utilities, you will need to reimplement or find substitutes in SQLMesh.

The Change Data Capture Integration Question

One thing teams ask about is how SQLMesh fits with change data capture pipelines. The typical pattern is: Debezium captures changes from Postgres, pushes them to Kafka, and a sink connector lands them in your data warehouse as append-only or upsert tables. Then your transformation layer (dbt or SQLMesh) picks up from there.

SQLMesh’s interval model works well with this pattern. If your CDC pipeline delivers data in hourly batches, you declare your source models with hourly granularity and SQLMesh will process each hour’s worth of data as an interval. Gap detection means that if the CDC pipeline missed an hour due to a Kafka rebalance, SQLMesh will flag that interval as unprocessed and wait (or alert) rather than silently producing incorrect results.

Looking at 2026 and Where This Goes

The SQLMesh roadmap includes deeper semantic layer capabilities, more robust dbt compatibility, and a SaaS hosted offering that would compete directly with dbt Cloud. The project is growing fast; the GitHub star count has roughly tripled in the past year and several major data teams (including some well-known fintech and e-commerce companies) have publicly documented their migrations.

The pattern I am seeing in 2026 is: teams that have outgrown dbt’s limitations are evaluating SQLMesh seriously, while teams that are happy with dbt are staying put. The tipping point usually comes when one of three things happens: the CI costs on large Snowflake or BigQuery warehouses become untenable, a production data quality incident is traced back to dbt’s lack of environment isolation, or a new data engineer joins from a software engineering background and asks why the data workflow does not have the same discipline as application code.

That last one is increasingly common. As software engineers have moved into data engineering roles, they bring expectations about environments, state management, and pre-flight validation that the dbt model struggles to satisfy. SQLMesh speaks their language.

My Recommendation

If you are building a new analytics engineering stack today, start with SQLMesh. The virtual environments and plan/apply model are better abstractions that will serve you as you scale. The dbt compatibility means you can always read dbt projects if you inherit them.

If you are already on dbt and things are working, do not migrate for migration’s sake. But if you are fighting the dev/prod isolation problem, drowning in CI costs, or building a platform for a team that needs reliable incremental processing, put SQLMesh on your evaluation list. Run it against a subset of your models first, see how the workflow feels, and make a data-driven decision.

Twenty years of building data infrastructure teaches you one thing above all: the workflow matters as much as the tool. SQLMesh’s workflow is simply more sound. Whether that is worth a migration depends entirely on your team’s specific pain points, and only you can answer that.