Apache Airflow tips

How to solve EmrStepSensor returns

I encountered that problem when I had been trying to add a watcher step to an EMR job. Unfortunately, my first tries ended with a Cluster id '' is not valid message. First, I checked whether the clust...

Continue Reading โ†’

How to use an execution branch depending on the previous results ?

Apache Airflow is a very flexible orchestration framework. You can execute the operations depending on the conditional branches or, as you will see below, on the previous tasks results. The execut...

Continue Reading โ†’

How to wait for a specific task in Apache Airflow?

A common way to control task sequentiality consists on using data sensors. The idea is to wait for the data generated by the previous DAG execution. Actually, there is also a second way to implement s...

Continue Reading โ†’

Airflow: no module named '...' problem

Once I encountered a mysterious error in the operator using some XCom variables: ERROR - No module named 'commons' Traceback (most recent call last): File "/usr/local/lib/python3.6/site-package...

Continue Reading โ†’