Once I encountered a mysterious error in the operator using some XCom variables:
ERROR - No module named 'commons' Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1642, in _run_raw_task self.render_templates() File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 1998, in render_templates rendered_content = rt(attr, content, jinja_context) File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 2909, in render_template return self.render_template_from_field(attr, content, context, jinja_env) File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 2877, in render_template_from_field result = jinja_env.from_string(content).render(**context) File "/usr/local/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render return original_render(self, *args, **kwargs) File "/usr/local/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render return self.environment.handle_exception(exc_info, True) File "/usr/local/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise raise value.with_traceback(tb) File "", line 1, in top-level template code File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 2105, in xcom_pull return pull_fn(task_id=task_ids) File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 73, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/airflow/models.py", line 4813, in get_one return pickle.loads(result.value) ModuleNotFoundError: No module named 'commons'
Initially I supposed some missing module in my code base but I was wrong. In fact the problem came from the fact of corrupted XCom variable used in the operator. To solve it I had to clean up the state generating that XCom variable and, of course, the task using the failed operator. In my case it led to retriggering the whole pipeline execution.