Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"The key 'start_date' in args is a part of kwargs and therefore reserved." #46754

Open
1 of 2 tasks
atul-astronomer opened this issue Feb 14, 2025 · 4 comments · May be fixed by #46961
Open
1 of 2 tasks

"The key 'start_date' in args is a part of kwargs and therefore reserved." #46754

atul-astronomer opened this issue Feb 14, 2025 · 4 comments · May be fixed by #46961
Assignees
Labels
affected_version:3.0.0alpha For all 3.0.0 alpha releases area:core area:task-execution-interface-aip72 AIP-72: Task Execution Interface (TEI) aka Task SDK kind:bug This is a clearly a bug

Comments

@atul-astronomer
Copy link

atul-astronomer commented Feb 14, 2025

Apache Airflow version

3.0.0a3

If "Other Airflow 2 version" selected, which one?

No response

What happened?

The DAG has failed for the reason "The key 'start_date' in args is a part of kwargs and therefore reserved."

{"timestamp":"2025-02-14T07:25:07.539637","level":"error","event":"Task failed with exception","logger":"task","error_detail":[{"exc_type":"ValueError","exc_value":"The key 'start_date' in args is a part of kwargs and therefore reserved.","syntax_error":null,"is_cause":false,"frames":[{"filename":"/opt/airflow/task_sdk/src/airflow/sdk/execution_time/task_runner.py","lineno":592,"name":"run"},{"filename":"/opt/airflow/task_sdk/src/airflow/sdk/execution_time/task_runner.py","lineno":699,"name":"_execute_task"},{"filename":"/opt/airflow/airflow/models/baseoperator.py","lineno":173,"name":"wrapper"},{"filename":"/opt/airflow/providers/standard/src/airflow/providers/standard/operators/python.py","lineno":185,"name":"execute"},{"filename":"/opt/airflow/providers/standard/src/airflow/providers/standard/operators/python.py","lineno":205,"name":"determine_kwargs"},{"filename":"/opt/airflow/airflow/utils/operator_helpers.py","lineno":178,"name":"determine"}]}]}

What you think should happen instead?

The DAG should pass as its passing for AF2.

How to reproduce

Run below DAG on Airflow 3 tag 3.0.0a3

default_args = {
    "owner": "airflow",
}

with DAG(
    dag_id="compare_pendulum_datetime2",
    default_args=default_args,
    start_date=start_date,
    schedule=None,
    catchup=False,
    user_defined_macros={
        "future_date": pendulum.datetime(2135, 11, 25)
    },
    tags=['core'],
) as dag:

    t1 = PythonOperator(
        task_id="subtract_future_and_start_date",
        python_callable=subtractor,
        op_args=[str(start_date), '{{ future_date }}']
    )

t1

Operating System

Linux

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@atul-astronomer atul-astronomer added area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Feb 14, 2025
@ashb ashb added area:task-execution-interface-aip72 AIP-72: Task Execution Interface (TEI) aka Task SDK affected_version:3.0.0alpha For all 3.0.0 alpha releases and removed needs-triage label for new issues that we didn't triage yet labels Feb 14, 2025
@amoghrajesh
Copy link
Contributor

@atul-astronomer the DAG is incomplete here, could you provide the entire DAG? I cannot see subtractor callable

@atul-astronomer
Copy link
Author

@amoghrajesh

#This compares the datetime objects without timezones

#do something with time with both datetime and pendulum

#dags can do same thing

#give it a start date in the past

# proposed test plan:
# dag1:
#  - pendulum start time in the past
#  - standard datetime in the future as a user macro
#  - task that calculates the difference between the two and prints it
# dag2:
#  - standard start time in the past
#  - pendulum datetime in the future as a user macro
#  - task that calculates the difference between the two and prints it
# (future) outer dag:
# - run both and compare that they do the same things

#templated fields to calculate how long ago the start date is
from airflow.models.dag import DAG
from airflow.providers.standard.operators.python import PythonOperator
from airflow.operators.empty import EmptyOperator

#from pendulum import datetime, timezone
import pendulum
import datetime
import pytz


etc_gmt13 = 'Etc/GMT-13'

chuuk = 'Pacific/Chuuk'

#start_date = pendulum.datetime(2021, 11, 25)
#start_date = pendulum.datetime(2135, 11, 25)
start_date = datetime.datetime(2021, 11, 25)

def subtractor(start_date, future_date):
    print(type(start_date), type(future_date)) #both strings, now find a way to compare
    if "T" in start_date or "T" in future_date:
        start_date = start_date.replace("T", "")
        future_date = future_date.replace("T", "")
    if "00:00:00" in start_date or "00:00:00" in future_date:
        start_date = start_date.replace("00:00:00", "")
        future_date = future_date.replace("00:00:00", "")
    
    if "+00:00" in start_date or "+00:00" in future_date:
        start_date = start_date.replace("+00:00", "")
        future_date = future_date.replace("+00:00", "")

    print(start_date, future_date) #2021-11-25 00:00:00+10:00 2135-11-25 00:00:00+13:00

    for i in start_date:
        if "-"==i:
            start_date = start_date.replace("-", "_")
    
    for i in future_date:
        if "-"==i:
            future_date = future_date.replace("-", "_")

    print(f"The newly filtered start date is {start_date} and the newly filtered future date is {future_date}") 
    #2021_11_25 +10:00, 2135_11_25 +13:00
    start_date_obj = datetime.datetime.strptime(start_date.strip(), "%Y_%m_%d")
    future_date_obj = datetime.datetime.strptime(future_date.strip(), "%Y_%m_%d")
    print(f"The start date object is: {start_date_obj}")
    print(f"The future date object is: {future_date_obj}")
    #now that the filtering has commenced
    #compare the 2 datetimes to ensure the result is the same across the 2 dags
    #the actual comparison

    the_difference = future_date_obj - start_date_obj
    print(f"This is the difference between the 2 dates in days: {the_difference}")
    print(f"This is the datatype of the difference betweeen the  2 datetime objects: {type(the_difference)}")


default_args = {
    "owner": "airflow",
}

with DAG(
    dag_id="compare_pendulum_datetime2",
    default_args=default_args,
    #datetime start date, pendulum future date as user defined macro
    start_date=start_date,
    schedule=None,
    catchup=False,
    user_defined_macros={
        "future_date": pendulum.datetime(2135, 11, 25)
    },
    tags=['core'],
) as dag:

    t1 = PythonOperator(
        task_id="subtract_future_and_start_date",
        python_callable=subtractor,
        op_args=[str(start_date), '{{ future_date }}']
    )

t1

@amoghrajesh
Copy link
Contributor

@atul-astronomer can you please check this on 2.10 as well? I think it should fail there as well.

@atul-astronomer
Copy link
Author

@amoghrajesh its passing on 2.10

Image

@ashb ashb moved this from Todo to In Progress in AIP-72 - Task Execution Interface and SDK Feb 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affected_version:3.0.0alpha For all 3.0.0 alpha releases area:core area:task-execution-interface-aip72 AIP-72: Task Execution Interface (TEI) aka Task SDK kind:bug This is a clearly a bug
Projects
Development

Successfully merging a pull request may close this issue.

3 participants