![]() ![]() It would be great to see Airflow or Apache separate Airflow-esque task dependency into its own microservice, as it could be expanded to provide dependency management across all of your systems, not just Airflow. Airflow Cross-communication (XCom) XCom is an inbuilt dictionary that houses metadata from all the steps in the Airflow DAG. Tasks with dependencies on this legacy replication service couldn’t use Task Sensors to check if their data is ready. The usage of XCOM between tasks keeps the DAG clean and avoids un-intended side-effects of instantiating variables within the DAG. def trigger (context, dagrunobj): destpath context ti. While external services can GET Task Instances from Airflow, they unfortunately can’t POST them. Is there a way to push this variable to another DAG using TriggerDagRunOperator. However, what if the upstream dependency is outside of Airflow? For example, perhaps your company has a legacy service for replicating tables from microservices into a central analytics database, and you don’t plan on migrating it to Airflow.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |