How you can enhance the code high quality of your dbt fashions with unit exams and TDD | by Pablo Porto | Might, 2023
[ad_1]
The next mannequin enriches a weight measurement with the newest top measurement recorded previous to the burden one.
Let’s now create a dbt unit take a look at to show the transformation logic is appropriate.
Wanting on the earlier take a look at, we are able to see a number of of the dbt-unit-testing macros in use:
- dbt_unit_testing.take a look at: This macro permits us outline the mannequin underneath take a look at and the title of the take a look at. In our instance we reference to int_weight_measurements_with_latest_height.
- dbt_unit_testing.mock_ref: This macro permits us to mock references to different dbt fashions. In our instance, we’re mocking the burden (stg_gym_app__weight) and the peak (stg_gym_app__height) staging information.
- dbt_unit_testing.anticipate: And this macro permits us to say on the results of the transformation. Within the instance, we assert that the burden measurement will get enriched with the newest top.
Let’s now run our mannequin’s unit take a look at. We will name the same old dbt take a look at command:
dbt take a look at
Opps, that instructions runs the entire take a look at suite together with different dbt information high quality exams. However we wish to solely run our unit exams. No downside, we are able to leverage dbt tags performance to isolate our unit exams. Within the instance, we tagged our take a look at with two tags:
{{ config(tags=['unit-test', 'unit-tests']) }}
The primary one is a boilerplate tag required by the dbt-unit-testing library. The second is the one we are going to use to execute our unit take a look at.
dbt take a look at --select tag:unit-tests
Thus far we’ve got seen how we are able to write a unit take a look at to confirm the logic of a single mannequin. After creating just a few of those, our crew began discussing the potential of implementing new forms of exams like we often do for operational software program like microservices.
“A part take a look at limits the scope of the exercised software program to a portion of the system underneath take a look at, manipulating the system by means of inside code interfaces and utilizing take a look at doubles to isolate the code underneath take a look at from different parts.” — Toby Clemson
In a microservices context, a part is a service that exposes sure functionalities. If we apply the identical idea to the info context, part exams for dbt apps could be applied as exams that validate whether or not the dbt app supplies the performance that it guarantees to by mocking the info sources.
When implementing the part take a look at, the scope of the take a look at will increase. We take a look at our dbt app as an entire mocking solely its sources.
Such a take a look at ensures that the completely different fashions combine appropriately and the anticipated information transformation end result is created. Let’s take a look at an instance:
Within the part take a look at above, we’re testing our output mannequin body_mass_indexes. The mannequin makes use of the enriched weight measurements to calculate the physique mass of the consumer. We mock the sources straight (raw_weight and raw_height) with the dbt_unit_testing.mock_source macro. Lastly, we assert the ultimate transformation of the output mannequin verifying that the physique mass index (BMI) is calculated appropriately.
We will additionally run such a exams in isolation utilizing the tag title we specified within the take a look at configuration.
dbt take a look at --select tag:component-test
Now that we’ve got the flexibility to check our fashions in isolation, what if we begin by writing the exams earlier than we write any transformation logic?
Take a look at-driven improvement or TDD is a software program engineering observe that helps enhance the design of the code by forcing the builders to put in writing a take a look at first after which write the minimal quantity of code to make that take a look at move.
Our information crew had expertise in making use of TDD in operational techniques so we determined to provide it a strive.
Beginning by defining the outcomes of a given transformation in a take a look at felt fairly pure. Oh, what’s the BMI I might anticipate if I’ve this weight and this top as an enter? Let’s write a take a look at for that. After working towards TDD for some time, the crew nonetheless proceed to make use of this system when including new enterprise logic into the transformations.
I simply pictured an ideal situation the place you’ll be able to simply add the unit testing dbt package deal and begin creating exams right away. The reality is that the library remains to be underneath improvement and we discovered some gotchas that you simply also needs to in all probability concentrate on:
- The dbt-unit-testing macros break the precept of not permitting testing code to pollute manufacturing code. There’s a simple hack to repair this. You may create a macro to patch the unique ref() and supply() and name the testing macros. You may see an example here.
- We discovered that typically looks as if modifications in a take a look at aren’t picked up. There’s an choice to disable caching however we haven’t strive it.
- When mocking sources you want to outline the supply within the dbt .yml file if not it doesn’t compile.
- Typically the take a look at error messages are fairly cryptic. On this scenario, we discovered ourselves wanting on the compiled SQL code within the construct folder.
- Be additionally conscious of different limitations listed in the library docs.
We’ve seen how we are able to add unit and part exams to our dbt initiatives to extend the code high quality and subsequently the maintainability of our transformation logic. We additionally noticed how we are able to tag the several types of exams in order that we are able to run them in isolation each regionally and within the CI/CD pipeline. Lastly, we additionally take a look at how we might additionally observe TDD
Hope this text helps you and your crew begin adopting unit exams and creating extra maintainable and scalable dbt apps as your codebase scale to satisfy new information use instances.
If you’re curious, you’ll be able to test a completely useful instance in this Github repo. I additionally ready some katas in case you wish to observe TDD and unit testing with a easy instance.
Are you prepared to provide it a strive?
[ad_2]
