Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

why dagster automatically convert job config of timestamp string into datetime #27855

Open
mathews opened this issue Feb 14, 2025 · 1 comment
Labels
type: bug Something isn't working

Comments

@mathews
Copy link

mathews commented Feb 14, 2025

What's the issue?

why dagster automatically convert job config of timestamp string into datetime?
I config a job with string of datetime, because dagster does not support datetime as config field.
but either start the job from UI or from graph api, the job runs with error:
Error 1: Invalid scalar at path root:end_time. Value "2025-02-13 15:00:00" of type "<class 'datetime.datetime'>" is not valid for expected type "String".
Error 2: Invalid scalar at path root:start_time. Value "2023-08-05 09:00:00" of type "<class 'datetime.datetime'>" is not valid for expected type "String".

my config in UI is :
{'symbol':'000034', 'start_time':'2023-08-05T09:00:00', 'end_time':'2025-02-13T15:00:00'}

and run from graph api with config:
run_job(
"job_bar_data",
{
"start_time": "2023-08-05T09:00:00",
"end_time": "2025-02-13T15:00:00",
"symbol": "000034",
"period": "1",
},
)
obviously, dagster automatically convert the start_time and end_time string into datetime.datetime.
this is just really acceptable, as dagster doesn't support datetime as config field.
please fix the bug.

What did you expect to happen?

No response

How to reproduce?

No response

Dagster version

1.9.9, 1.10.0

Deployment type

None

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

@mathews mathews added the type: bug Something isn't working label Feb 14, 2025
@mathews
Copy link
Author

mathews commented Feb 14, 2025

this is the stacktrace of bug:
dagster._core.errors.DagsterInvalidConfigError: Error in config mapping
Error 1: Invalid scalar at path root:end_time. Value "2025-02-13 15:00:00" of type "<class 'datetime.datetime'>" is not valid for expected type "String".
Error 2: Invalid scalar at path root:start_time. Value "2023-08-05 09:00:00" of type "<class 'datetime.datetime'>" is not valid for expected type "String".

File "/media/mathews/LENOVO/orchastack/quant-wp/quantenv/lib/python3.12/site-packages/dagster/_core/execution/plan/execute_plan.py", line 245, in dagster_event_sequence_for_step
yield from check.generator(step_events)
File "/media/mathews/LENOVO/orchastack/quant-wp/quantenv/lib/python3.12/site-packages/dagster/_core/execution/plan/execute_step.py", line 501, in core_dagster_event_sequence_for_step
for user_event in _step_output_error_checked_user_event_sequence(
File "/media/mathews/LENOVO/orchastack/quant-wp/quantenv/lib/python3.12/site-packages/dagster/_core/execution/plan/execute_step.py", line 184, in _step_output_error_checked_user_event_sequence
for user_event in user_event_sequence:
File "/media/mathews/LENOVO/orchastack/quant-wp/quantenv/lib/python3.12/site-packages/dagster/_core/execution/plan/execute_step.py", line 88, in _process_asset_results_to_events
for user_event in user_event_sequence:
File "/media/mathews/LENOVO/orchastack/quant-wp/quantenv/lib/python3.12/site-packages/dagster/_core/execution/plan/compute.py", line 190, in execute_core_compute
for step_output in _yield_compute_results(step_context, inputs, compute_fn, compute_context):
File "/media/mathews/LENOVO/orchastack/quant-wp/quantenv/lib/python3.12/site-packages/dagster/_core/execution/plan/compute.py", line 159, in _yield_compute_results
for event in iterate_with_context(
File "/media/mathews/LENOVO/orchastack/quant-wp/quantenv/lib/python3.12/site-packages/dagster/_utils/init.py", line 480, in iterate_with_context
next_output = next(iterator)
^^^^^^^^^^^^^^
File "/media/mathews/LENOVO/orchastack/quant-wp/quantenv/lib/python3.12/site-packages/dagster/_core/execution/plan/compute_generator.py", line 127, in _coerce_op_compute_fn_to_iterator
result = invoke_compute_fn(
^^^^^^^^^^^^^^^^^^
File "/media/mathews/LENOVO/orchastack/quant-wp/quantenv/lib/python3.12/site-packages/dagster/_core/execution/plan/compute_generator.py", line 115, in invoke_compute_fn
return fn(context, **args_to_pass) if context_arg_provided else fn(**args_to_pass)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/mathews/LENOVO/orchastack/quant-ml-wp/orcha-quant-pipeline/orcha_quant_pipeline/ds_jobs/akshare_data.py", line 220, in op_bar_data
result = job_download_akshare_bar_data.execute_in_process(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/mathews/LENOVO/orchastack/quant-wp/quantenv/lib/python3.12/site-packages/dagster/_core/definitions/job_definition.py", line 745, in execute_in_process
return core_execute_in_process(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/mathews/LENOVO/orchastack/quant-wp/quantenv/lib/python3.12/site-packages/dagster/_core/execution/execute_in_process.py", line 42, in core_execute_in_process
execution_plan = create_execution_plan(
^^^^^^^^^^^^^^^^^^^^^^
File "/media/mathews/LENOVO/orchastack/quant-wp/quantenv/lib/python3.12/site-packages/dagster/_core/execution/api.py", line 702, in create_execution_plan
resolved_run_config = ResolvedRunConfig.build(job_def, run_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/mathews/LENOVO/orchastack/quant-wp/quantenv/lib/python3.12/site-packages/dagster/_core/system_config/objects.py", line 150, in build
run_config = run_config_schema.config_mapping.resolve_from_unvalidated_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/media/mathews/LENOVO/orchastack/quant-wp/quantenv/lib/python3.12/site-packages/dagster/_core/definitions/config.py", line 89, in resolve_from_unvalidated_config
raise DagsterInvalidConfigError(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant