Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

on_dag_run_created listener ignored in plugin #44071

Open
2 tasks done
WillRaphaelson opened this issue Nov 15, 2024 · 1 comment
Open
2 tasks done

on_dag_run_created listener ignored in plugin #44071

WillRaphaelson opened this issue Nov 15, 2024 · 1 comment
Labels
area:core area:Listeners kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet

Comments

@WillRaphaelson
Copy link

Apache Airflow version

2.10.3

If "Other Airflow 2 version" selected, which one?

No response

What happened?

on_dag_run_created hook is being ignored in plugin

What you think should happen instead?

on_dag_run_created should run the relevant logic in the function

How to reproduce

from airflow.plugins_manager import AirflowPlugin
from airflow.listeners import hookimpl
import logging

# Configure logger
logger = logging.getLogger(__name__)

class PreDagRunListener:
    """
    Listener to perform actions before any task in a DAG run starts.
    """

    @hookimpl
    def on_dag_run_created(self, dag_run, session):
        """
        Hook triggered when a DAG run is created, before tasks are executed.
        """
        dag_id = dag_run.dag_id
        run_id = dag_run.run_id

        logger.info(f"Pre-DAG execution logic triggered for DAG {dag_id}, run_id {run_id}.")

        # Add your custom pre-DAG logic here
        # Example: Block a specific DAG
        if dag_id == "blocked_dag":
            logger.error(f"DAG {dag_id} is blocked from running.")
            raise ValueError(f"DAG {dag_id} failed pre-run checks.")
        
        logger.info(f"DAG {dag_id} passed pre-run checks.")

class PreDagRunPlugin(AirflowPlugin):
    name = "pre_dag_run_plugin"
    listeners = [PreDagRunListener()]

Operating System

MacOS

Versions of Apache Airflow Providers

apache-airflow-providers-common-compat==1.2.1
apache-airflow-providers-common-io==1.4.2
apache-airflow-providers-common-sql==1.19.0
apache-airflow-providers-fab==1.5.0
apache-airflow-providers-ftp==3.11.1
apache-airflow-providers-http==4.13.2
apache-airflow-providers-imap==3.7.0
apache-airflow-providers-smtp==1.8.0
apache-airflow-providers-sqlite==3.9.0

Deployment

Virtualenv installation

Deployment details

Local setup - only change i made was to not lazy load plugins with lazy_load_plugins = False

Anything else?

I can never get an on_dag_run_created hook to run no matter what. I'm open ot anything that allows me to run dag level pre-flight checks.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@WillRaphaelson WillRaphaelson added area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Nov 15, 2024
Copy link

boring-cyborg bot commented Nov 15, 2024

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:core area:Listeners kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet
Projects
None yet
Development

No branches or pull requests

1 participant