Creating a DAG (Directed Acyclic Graph) is the first step in building workflows in Apache Airflow.<\/p>\n\n\n\n
A DAG defines:<\/p>\n\n\n\n
DAGs are written in Python and saved as A simple DAG includes:<\/p>\n\n\n\n Default arguments are settings applied to all tasks.<\/p>\n\n\n\n Key Parameters:<\/p>\n\n\n\n Example Python function:<\/p>\n\n\n\n Create a task using PythonOperator:<\/p>\n\n\n\n Dependencies define execution order.<\/p>\n\n\n\n For multiple tasks:<\/p>\n\n\n\n This means: Example:<\/p>\n\n\n\n This runs daily at 6 AM.<\/p>\n\n\n\n Creating a DAG in Apache Airflow involves defining a DAG object in Python, adding tasks using operators, and setting dependencies between tasks to control execution order.<\/p>\n\n\n\n Creating DAGs in Apache Airflow involves:<\/p>\n\n\n\n DAGs are the backbone of workflow orchestration in modern data engineering pipelines.<\/p>\n\n\n.py<\/code> files inside the Airflow dags<\/code> folder.<\/p>\n\n\n\nBasic Structure of a DAG<\/h1>\n\n\n\n
\n
Step 1: Import Required Libraries<\/h1>\n\n\n\n
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime<\/pre>\n\n\n\nStep 2: Define Default Arguments<\/h1>\n\n\n\n
default_args = {
'owner': 'airflow',
'retries': 1
}<\/pre>\n\n\n\nStep 3: Create the DAG Object<\/h1>\n\n\n\n
dag = DAG(
dag_id='simple_pipeline',
default_args=default_args,
start_date=datetime(2024, 1, 1),
schedule_interval='@daily',
catchup=False
)<\/pre>\n\n\n\n\n
Step 4: Define Tasks<\/h1>\n\n\n\n
def say_hello():
print(\"Hello from Airflow\")<\/pre>\n\n\n\ntask1 = PythonOperator(
task_id='hello_task',
python_callable=say_hello,
dag=dag
)<\/pre>\n\n\n\nStep 5: Set Task Dependencies<\/h1>\n\n\n\n
task1<\/pre>\n\n\n\n
task1 >> task2 >> task3<\/pre>\n\n\n\n
task1 runs first, then task2, then task3.<\/p>\n\n\n\nExample Complete Simple DAG<\/h1>\n\n\n\n
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetimedefault_args = {
'owner': 'airflow',
'retries': 1
}def extract():
print(\"Extracting data\")def transform():
print(\"Transforming data\")def load():
print(\"Loading data\")dag = DAG(
dag_id='etl_pipeline',
default_args=default_args,
start_date=datetime(2024, 1, 1),
schedule_interval='@daily',
catchup=False
)extract_task = PythonOperator(
task_id='extract',
python_callable=extract,
dag=dag
)transform_task = PythonOperator(
task_id='transform',
python_callable=transform,
dag=dag
)load_task = PythonOperator(
task_id='load',
python_callable=load,
dag=dag
)extract_task >> transform_task >> load_task<\/pre>\n\n\n\nCommon DAG Scheduling Options<\/h1>\n\n\n\n
\n
schedule_interval='0 6 * * *'<\/pre>\n\n\n\n
Best Practices for Creating DAGs<\/h1>\n\n\n\n
\n
How to Deploy a DAG<\/h1>\n\n\n\n
\n
Interview Answer (Short Version)<\/h1>\n\n\n\n
Final Summary<\/h1>\n\n\n\n
\n