We’ve just cut a first release candidate of dbt Core v0.21 (Louis Kahn), which includes some long-sought-after additions:
dbt buildcommand for multi-resource runs (watch Staging!)
- Defining configs in all the places you’d expect
- Handling for column schema changes in incremental models
A new minor version of dbt Core is exciting enough, but there’s something even more exciting lurking just beyond. The version after v0.21 will not be v0.22; instead, it’s going to be the big one (point oh).
In the 5+ years since dbt’s very first commit, we have been iterating on and solidifying its core functionality:
- SQL compilation and execution
- powerful, extensible materializations for models, tests, and the rest
- constructing DAGs of idempotent transformations
- collecting and exposing valuable metadata all along the way
Over that time, we’d reserved the right to make big changes when called for, and to revisit foundational assumptions if they weren’t scaling to meet the needs of a fast-growing community.
Now, we feel the time is right to lock in the foundational product. We’re ready to call dbt a mature tool at the heart of the modern data ecosystem. We will be releasing dbt Core v1.0.0 in December 2021.
What does v1 mean for you?
- Commitments to stability for the most crucial interfaces, so you can feel confident building projects and tooling on top of dbt
- More intuitive naming for familiar features, to smooth the learning curve for new users and colleagues
- Improved performance, turned on for everyone, ensuring quick development cycles in dbt projects of all sizes
- Easier upgrades to newer versions, so you’re always getting the latest & greatest that dbt has to offer
This is a rite of passage, and you’re all invited to the ceremony: I’ll have much more to say about all of the above at Coalesce in December. Today, I want to share some more thoughts about versions and upgrades in particular.
The best preparation: upgrading now
You’re able to get a lot of things right now by upgrading to newer version of dbt Core:
- Exciting, net-new functionality
- More intuitive ways to use existing features
- Bug fixes, both in
dbt-coreand in your warehouse adapter
You might also encounter a handful of breaking changes along the way. During the version-zero era of dbt, we’ve iterated on every aspect of the core product—making its interfaces intuitive, its logic sensible, and its architecture one that can perform at scale. These have been necessary changes, and we’ve communicated them accordingly. To those who’ve seen `archives` become `snapshots`, who have a distinct memory of `version: 1` yml files: thank you for your patience and collaboration over the past few years. We couldn’t have done it without you.
After v1.0, no more. There will be no breaking changes to documented features—the code in your projects—in minor versions. There will be no reason not to upgrade. We can all take a deep breath, loosen up our `require-dbt-version` configs, and focus on building. (Package maintainers: that goes for you, too! Thanks for all you contribute to the dbt ecosystem; I’m excited to make your jobs easier.)
By upgrading your project to the latest version of dbt now, you’re making it that much easier for your team to upgrade to v1.0 this December.
Why we care about version numbers (and why you should, too)
In semantic versioning, a version is composed of three numbers: Major.Minor.Patch. Today, dbt is “major version zero,” which means that the core interfaces are not yet locked down. Once we release v1.0, they will be. Afterward, each of these numbers will start having a new, subtly different meaning:
- Major: dbt is going to look, feel, and work like this for a long time. You can know with confidence that, if the code in your project works on v1.0.0, it will keep working on every v1.x.y after that.
- Minor version releases can still include big new things, but changes to existing interfaces will be behind-the-scenes and backwards-compatible. Simply put, the code in your project will not break when you upgrade. In the unlikely case that this happens, we will do everything in our power to fix it, and turn around a patch release with the fix, as quickly as possible.
- Patch releases are for bug fixes only; they will never include net-new features. You can upgrade to a newer patch release with full confidence, and without a moment’s hesitation.
We are going to keep working on dbt. There are still so many features to build, and always plenty of bugs and corner cases to fix. While we do that, you can feel certain that these things will not break in new minor versions:
- The code in your dbt projects
- The CLI commands, arguments, and flags in your dbt deployments
At the same time, I want to be clear about the things that can change, after v1, in minor versions only. We’ll offer lots of prior warning, and aim for backwards compatibility wherever possible. The most important ones are:
- Adapter plugins: If we add, change, or remove internal methods that require changes by maintainers of adapter code, we will call it out clearly in prerelease notes and documentation. At the same time, patch releases will not break these internal contracts. Adapter plugins can start
~=pinning dbt-core minor versions, and get patches for free.
- Metadata artifacts: If we add, change, or remove a field in
sources.json, we will bump the version number of its artifact schema, and clearly document the changes in a migration guide.
There are still no guarantees about things that aren’t guaranteed today: importing dbt-core as a python library; less mature functionality that wraps dbt-core, such as the dbt-rpc server; or any other uses of dbt that are not already documented.
Upgrade, today & always
We believe that:
- You should always be using the latest & greatest version of dbt
- You should especially always upgrade to a newer patch release as soon as it’s out, without a second thought (e.g. v0.20.1 → v0.20.2, a.k.a v0.20.latest)
- You should upgrade to dbt v1.0 when it’s available in December
The teams behind dbt Cloud will be rolling out tooling to help with all of the above, and to ensure you’re running a stable, full-featured version of dbt Core. If you’re a Cloud user, stay tuned for more details.
There’s so much we still want to build in dbt Core. We also know that thousands of companies are running dbt in production, and building critical data infrastructure with dbt at the center. It’s a responsibility we take seriously, and we’re making commitments to reflect it. That, friends, is the real meaning of v1.0.
I’ll have even more to say at Coalesce—see you there!