Skip to content

How to import dates in ISO 8601 from JSON using InfluxDBClient3.write_file? #125

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dimhotepus opened this issue Apr 5, 2025 · 1 comment

Comments

@dimhotepus
Copy link

Please direct all support questions to slack or the forums. Thank you.

Let's assume I use Influxdb3 core and have dates in JSON using ISO 8601 format:

fpo = file_parser_options(lines=True)

client.write_file(file='./Device.json',
                      measurement_name='Device',
                      timestamp_column='CreatedAt',
                      tag_columns=['Name', 'Model', 'Location', 'ActiveIPv4Address', 'Mac'],
                      file_parser_options=fpo,
                      write_precision='ms')

In JSON:

{"CreatedAt": "2023-06-20T17:50:09.702Z", other fields... }

But client unable to import the dates, it tries to import them as float:

    Traceback (most recent call last):
  File "tslib.pyx", line 316, in pandas._libs.tslib.array_with_unit_to_datetime
ValueError: could not convert string to float: '2023-06-20T17:50:09.702Z'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/opc/mongodb-influx-db-python/./mongodb-to-influxdb.py", line 74, in <module>
    client.write_file(file='./Device.fixup.json',
  File "/home/opc/.local/lib/python3.12/site-packages/influxdb_client_3/__init__.py", line 182, in write_file
    raise e
  File "/home/opc/.local/lib/python3.12/site-packages/influxdb_client_3/__init__.py", line 180, in write_file
    self._process_dataframe(df, measurement_name, tag_columns or [], timestamp_column, database=database, **kwargs)
  File "/home/opc/.local/lib/python3.12/site-packages/influxdb_client_3/__init__.py", line 203, in _process_dataframe
    self._write_api.write(bucket=database, record=df,
  File "/home/opc/.local/lib/python3.12/site-packages/influxdb_client_3/write_client/client/write_api.py", line 366, in write
    return self._write_batching(bucket, org, record,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/opc/.local/lib/python3.12/site-packages/influxdb_client_3/write_client/client/write_api.py", line 473, in _write_batching
    serializer = DataframeSerializer(data, self._point_settings, precision, self._write_options.batch_size,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/opc/.local/lib/python3.12/site-packages/influxdb_client_3/write_client/client/write/dataframe_serializer.py", line 111, in __init__
    data_frame_timestamp = pd.to_datetime(data_frame_timestamp, unit=precision)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/opc/.local/lib/python3.12/site-packages/pandas/core/tools/datetimes.py", line 1067, in to_datetime
    values = convert_listlike(arg._values, format)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/opc/.local/lib/python3.12/site-packages/pandas/core/tools/datetimes.py", line 407, in _convert_listlike_datetimes
    return _to_datetime_with_unit(arg, unit, name, utc, errors)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/opc/.local/lib/python3.12/site-packages/pandas/core/tools/datetimes.py", line 526, in _to_datetime_with_unit
    arr, tz_parsed = tslib.array_with_unit_to_datetime(arg, unit, errors=errors)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "tslib.pyx", line 344, in pandas._libs.tslib.array_with_unit_to_datetime
  File "tslib.pyx", line 318, in pandas._libs.tslib.array_with_unit_to_datetime
ValueError: non convertible value 2023-06-20T17:50:09.702Z with the unit 'ms', at position 0
@dimhotepus
Copy link
Author

Oops, missed support note.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant