Why I’m going to enforce CI in dbt Cloud as soon as I get home (thanks Coalesce)​

You want your production environment to work smoothly and never break. And who doesn’t?

So you set a standard for yourself and your team to run a dbt build before every commit/merge request. However

• it unnecessarily wastes time and computing credits (maybe you even need to take a coffee break every time you run dbt build?)
• you have no oversight of whether the rest of the team actually runs a successful dbt build before committing their changes

The solution? Continuous Integration (CI).

A quick definition: Continuous Integration (CI) is a set of automated tests and runs that take place on your code base before merging development and production.

The purpose of these tests and runs in to catch any failure before merging, effectively ensuring that your production environment keeps running smoothly and only allowing you to merge once it’s safe.

As a heavy dbt Cloud user, I am pleased to see a new set of features for CI in dbt Cloud that make it easy to set up the project. Let me walk you through how to do it

How to set up CI in dbt Cloud

1. Create a new Job through the dbt Cloud interface and select the option “Continuous Integration Job”. This will create a job with the pre-filled command: dbt build –select state:modified+, so that the tests will run only the models you’ve changed and their children (saving time and resources!).

2. Select the environment you want to defer to! This means that it can defer to the most recent job of the selected environment. 

By “deferring”, we mean looking at the difference between the two jobs and only running what is necessary based on that difference. 

Thus, by using deferring and state:modified at the same time, we will only run what has been modified between the most recent job and the code we are trying to merge.

3. Select the feature “Triggered by pull requests” if you want even more automation! This means that whenever a developer in your team opens a pull request, a CI check will automatically be triggered; with the option to even run this on Draft Pull Request (which you can switch off at will).

Just a note on this last point: you will need to be connected to one of the native repositories to enable this feature (e.g. GitLab, GitHub): I tried to set this up on a mock account we used at Coalesce and wasn’t able to:

Unlimited Concurrent CI checks

Another very appealing feature is the possibility to run an unlimited amount of CI checks at the same time.

So….go try it out!

In conclusion, the newly introduced dbt Cloud features make for a smooth CI experience and I expect more and more project to adopt this standard – I know I will.

Read more TIL NL blogs here: https://theinformationlab.nl/blogs/

or check out my YouTube channel: @downtoearthdatatip