How to convert timestamp[ms, tz=UTC] object to supported type

[Hasham04]

I am reading a parquet file and one of the date-time columns is of type timestamp[ms, tz=UTC]. I have tried converting this to

df['time'].astype('datetime64[ms]')
df['time'].astype('timestamp')

but I always get the error raise NotImplementedError(f'Cannot convert {arrow_type}') NotImplementedError: Cannot convert timestamp[ms, tz=UTC]. Any suggestions on how to convert this to a supported type.

I can print the data frame but the second I try to interact with this column like doing df.dtypes I get this error.

I want to convert this column to a sting ideally so that I can concat it with another string column.

thanks for your help.

[Hasham04]

In array_types.py I added map_arrow_to_numpy[pa.timestamp(unit,tz="UTC")] = np.dtype(f"datetime64[{unit}]") to include pa.timestamp with a timezone. Once i added this it started working, I was able to convert to 'datetime64[ms]' and interact with the data.

import numpy as np
import pyarrow as pa
import vaex.utils

supported_arrow_array_types = (pa.Array, pa.ChunkedArray)
supported_array_types = (np.ndarray, ) + supported_arrow_array_types
string_types = [pa.string(), pa.large_string()]
_type_names_int = ["int8", "int16", "int32", "int64", "uint8", "uint16", "uint32", "uint64"]
_type_names = ["float64", "float32"] + _type_names_int
map_arrow_to_numpy = {getattr(pa, name)(): np.dtype(name) for name in _type_names}
map_arrow_to_numpy[pa.bool_()] = np.dtype("?")
for unit in 's ms us ns'.split():
	map_arrow_to_numpy[pa.timestamp(unit)] = np.dtype(f"datetime64[{unit}]")
	map_arrow_to_numpy[pa.timestamp(unit,tz="UTC")] = np.dtype(f"datetime64[{unit}]") # added this.

this is a temporary fix. When I deploy I won't be able to change the code in vaex. is there any other way I can convert? Help is appreciated.

[maartenbreddels]

Probably fixed in #1920 which will be released soon (v4.9)

[Hasham04]

Hi Maarten , I think the issue I was having was that vaex doesn't support a timestamp with timezone information tz=UTC. How does the pr you mentioned solve this issue. Please guide.

[maartenbreddels]

Indeed, we don't support timezones, but I though this was not the issue right?

[Hasham04]

I think I might have been unclear @maartenbreddels. The issue is that I can't convert timestamp column with timezone information to a supported type. Maybe convert it to datetime64 and/or dropping the timezone information. I was hoping there might be a way to do this that I am not aware of. Thanks for your response. Looking forward to your talk today!!

[maartenbreddels]

I hope v4.9 will allow you to do what you want, it's much better with datetime conversion.

Hope to hear from you tonight! :)

[KIC]

vaex 4.16 still has the same issue. Why would you not want to support timezoned timestamps? I mean this is an obvious use case :-(

PS even .to_pandas_df() looses the timezone :.-(