Airflow DAG Execution and XCom Value Retrieval Automation - n8n Workflow

Data Engineers needing reliable orchestration monitoring and synchronization.
DevOps teams automating deployment steps using n8n and Airflow.

• Users looking for advanced examples of resilient polling loops within an n8n workflow.

This comprehensive n8n workflow provides a structured approach to integrating n8n with Apache Airflow's REST API. While Airflow handles complex data pipelines, n8n excels at external orchestration and monitoring. This specific n8n workflow template solves the crucial problem of synchronous execution—triggering a DAG, waiting for it to finish, handling various success, failure, and timeout states, and finally extracting meaningful output (XCom values).

This automation leverages multiple n8n node types to create a resilient monitoring mechanism, providing robust error handling essential for production systems and eliminating the need for external scripting. Utilizing this n8n workflow means you can reliably chain Airflow processes with other services managed by n8n.

1. The process begins with the in data n8n trigger node, which receives required input parameters such as dagid, taskid, the execution configuration (conf), polling interval (wait), and the maximum polling count (waittime).


2. The airflow-api n8n node sets the base URL for the Airflow API connection.


3. The Airflow: dagrun n8n node initiates the DAG execution via a secure HTTP POST request, passing the runtime configuration payload.


4. An initial check using the if state == queued n8n node verifies if the DAG was successfully initiated. If not queued, the n8n workflow stops with an error.


5. If the DAG is running, the n8n workflow enters a polling loop. It first pauses using the Wait n8n node (defaulting to 10 seconds).


6. The Airflow: dagrun - state n8n node executes a GET request to repeatedly check the status of the specific DAG run ID.


7. The Switch: state n8n node determines the subsequent flow based on the returned state:

1. The If count > waittime n8n node acts as a safety mechanism, checking if the number of polling attempts exceeds the configured limit. If exceeded, the dag run wait too long n8n node stops the automation.


2. If successful, the final Airflow: dagrun - get result n8n node retrieves the designated XCom value (returnvalue) from the specified task instance, completing the synchronous operation within this powerful n8n template.

To utilize this highly structured n8n workflow template, follow these steps:

1. Import Workflow: Copy the provided JSON data and paste it directly into your n8n instance using the "Import Workflow" function.


2. Airflow Credentials Setup: This n8n workflow requires HTTP Basic Auth credentials for secure access to the Airflow API endpoints.

1. API Endpoint Configuration: Edit the airflow-api Set n8n node. Update the prefix value to match your actual Airflow API base URL (e.g., https://airflow.mycompany.com).


2. Testing: Execute the in data n8n trigger node manually, supplying sample parameters like a valid dagid, task_id, and a JSON object for the conf payload to confirm the polling mechanism and error handling of this n8n template function correctly.

in data (Execute Workflow Trigger): This acts as the primary n8n trigger, defining required input parameters for the DAG execution, including DAG ID, task ID, execution configuration, and polling limits.
airflow-api (Set): An essential configuration n8n node used to dynamically set the base URL prefix for all subsequent Airflow API HTTP requests.
Airflow: dagrun (HTTP Request): The initial action n8n node responsible for sending a POST request to the Airflow API to commence a new DAG run instance.
Wait (Wait): A core logic n8n node used within the polling loop to introduce a configurable delay (default 10 seconds), managing API throttling and reducing load during the waiting period.
Airflow: dagrun - state (HTTP Request): This n8n node handles status polling by repeatedly querying the Airflow API using the GET method to retrieve the current state of the executing DAG run.
Switch: state (Switch): A key flow control n8n node that directs the n8n workflow path based on the current state (success, running, failed, etc.), ensuring the workflow either proceeds, loops, or terminates.
count (Code): A custom JavaScript n8n node that increments a counter on each loop iteration. This counter is vital for implementing the maximum timeout logic defined in this n8n workflow.
Airflow: dagrun - get result (HTTP Request): The final retrieval n8n node, which fetches the specified XCom value (returnvalue) once the DAG execution is successfully completed.