4 Airflow Internals You Most likely Don’t Know About


4 Apache Airflow internals you might need missed

Picture generated through DALL-E

I’ve been working with Airflow for greater than three years now and total, I’m fairly assured with it. It’s a strong orchestrator that helps me construct information pipelines shortly and in a scalable trend whereas for many issues I’m seeking to implement it comes with batteries included.

Lately, and whereas getting ready myself to get a certification for Airflow, I’ve come throughout many various issues I had actually no clue about. And this was primarily my motivation to put in writing this text and share with you a number of Airflow internals which have completely blown my thoughts!

1. Scheduler solely parses information containing sure key phrases

The Airflow Scheduler will parse solely information containing airflow or dag within the code! Sure, you’ve heard this proper! If a file underneath the DAG folder doesn’t include not less than considered one of these two key phrases, it would merely not be parsed by the scheduler.

If you wish to modify this rule such that that is not a requirement for the scheduler, you may merely set DAG_DISCOVERY_SAFE_MODE configuration setting to False. In that case, the scheduler will parse all information underneath your DAG folder (/dags).

I wouldn’t suggest disabling this verify although, since doing so doesn’t actually make any sense. A correct DAG file could have Airflow imports and DAG definition which suggests the necessities for parsing that file are met) however it’s value understanding that this rule exists.

2. Variables with sure key phrases of their title have their values hidden

We all know that by default, Airflow will disguise delicate data saved in a Connection (and extra particularly within the password subject), however what about Variables?

Effectively, that is certainly attainable and the thoughts blowing factor is that Airflow can do that routinely for you. If a variable accommodates sure key phrases, that may presumably point out delicate data, then its worth will routinely be hidden.

Right here’s a listing of key phrases that may make a Variable qualify for having delicate data retailer as…


Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button