Pyogrio and Use Arrow - Read GDB Layer - "Categorical categories cannot be null"

[zachariahBinx]

Introduction
I came across an interesting issue when using GeoPandas to read gdb layers into a geodataframe while setting use_arrow=True and the engine to pyogrio (defualt). Found a stack post about setting "use_arrow" to True to speed up reading and writing, so wanted to try it out.

Problem
If a gdb has a domain with the below properties and the FIRST Code is set to 1 <= x <= 99, pandas returns the following error: ValueError: Categorical categories cannot be null. If the Code is set to any other number (positive or negative), reading the data passes. If the first Code is set to 150 and the second Code entry is set to 50, passes. Only when the first Code is set 1 <= x <= 99 is when it fails.

Properties

Domain Name = Domain_One
Decription = Some Description Text
Field Type = Long
Domain Type = Coded value Domain
Split Policy = Default
Merge Policy = Default

Solutions

1. Rearrange domain codes
2. Setting geopandas engine to "fiona"

Environment Set Up

uv venv --python 3.11
.venv/scripts/activate
uv init
uv add geopandas
uv add pyarrow
uv add fiona

pyproject.toml

[project]
name = "pyogrio-arrow"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.11"
dependencies = [
    "fiona>=1.10.1",
    "geopandas>=1.0.1",
    "pyarrow>=19.0.1",
]

Code

import os
import geopandas
geopandas.options.io_engine = "pyogrio"
os.environ["PYOGRIO_USE_ARROW"] = '1'

def main():
    file_path = r"C:\Users\MY_USER\TEST_DATA\PYOGRIO_USE_ARROW\Test.gdb"
    gdf = geopandas.read_file(file_path, layer="Test_Layer")

if __name__ == "__main__":
    main()

GDB

ArcPro 3.1.2

Created a gdb with "Test" polygon feature class with 4 geometries. Domain with "Long" type and a Code set to 50. Field "Field" domain is set to "Domain_one".

Error Message

Traceback (most recent call last):
  File "C:\Users\MY_USER\PYOGRIO_ARROW\main.py", line 12, in <module>
    main()
  File "C:\Users\MY_USER\PYOGRIO_ARROW\main.py", line 8, in main
    gdf = geopandas.read_file(file_path, layer="Test_Layer")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\MY_USER\PYOGRIO_ARROW\.venv\Lib\site-packages\geopandas\io\file.py", line 294, in _read_file
    return _read_file_pyogrio(
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\MY_USER\PYOGRIO_ARROW\.venv\Lib\site-packages\geopandas\io\file.py", line 547, in _read_file_pyogrio
    return pyogrio.read_dataframe(path_or_bytes, bbox=bbox, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\MY_USER\PYOGRIO_ARROW\.venv\Lib\site-packages\pyogrio\geopandas.py", line 292, in read_dataframe
    df = table.to_pandas(**kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow\\array.pxi", line 889, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow\\table.pxi", line 5132, in pyarrow.lib.Table._to_pandas
  File "C:\Users\MY_USER\PYOGRIO_ARROW\.venv\Lib\site-packages\pyarrow\pandas_compat.py", line 824, in table_to_dataframe
    blocks = [
             ^
  File "C:\Users\MY_USER\PYOGRIO_ARROW\.venv\Lib\site-packages\pyarrow\pandas_compat.py", line 825, in <listcomp>
    _reconstruct_block(item, column_names, ext_columns_dtypes)
  File "C:\Users\MY_USER\PYOGRIO_ARROW\.venv\Lib\site-packages\pyarrow\pandas_compat.py", line 739, in _reconstruct_block
    arr = _pandas_api.categorical_type.from_codes(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\MY_USER\PYOGRIO_ARROW\.venv\Lib\site-packages\pandas\core\arrays\categorical.py", line 745, in from_codes
    dtype = CategoricalDtype._from_values_or_dtype(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\MY_USER\PYOGRIO_ARROW\.venv\Lib\site-packages\pandas\core\dtypes\dtypes.py", line 338, in _from_values_or_dtype
    dtype = CategoricalDtype(categories, ordered)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\MY_USER\PYOGRIO_ARROW\.venv\Lib\site-packages\pandas\core\dtypes\dtypes.py", line 221, in __init__
    self._finalize(categories, ordered, fastpath=False)
  File "C:\Users\MY_USER\PYOGRIO_ARROW\.venv\Lib\site-packages\pandas\core\dtypes\dtypes.py", line 378, in _finalize
    categories = self.validate_categories(categories, fastpath=fastpath)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\MY_USER\PYOGRIO_ARROW\.venv\Lib\site-packages\pandas\core\dtypes\dtypes.py", line 576, in validate_categories
    raise ValueError("Categorical categories cannot be null")
ValueError: Categorical categories cannot be null

[theroggy]

Can you share the GDB triggering this error?

[jorisvandenbossche]

In addition to sharing the files, some other notes:

• This error happens in the step converting the Arrow table to a pandas DataFrame. You can use pyogrio directly to read the Arrow table, so you can show the schema and table:

import pyogrio
import pyarrow as pa
meta, table = pyogrio.read_arrow(file_path, layer=...)

• You mention the workaround to use fiona, but you can also use the default pyogrio engine in geopandas without specifying to use arrow

• My assumption is that the field domain support in GDAL (https://gdal.org/en/stable/drivers/vector/filegdb.html#field-domains) will create Arrow data with a dictionary type, but in such a way apparently that is not supported by pandas

[zachariahBinx]

@theroggy
PYOGRIO_USE_ARROW.zip

@jorisvandenbossche
Thank you for your insite. I will continue to play/test/dig around. Because there are work arounds, not a big issue. Wanted to flag it for others to be aware.

[jorisvandenbossche]

Thanks for providing the file. As I assumed, GDAL creates a dictionary typed column for the field domain:

In [1]: import pyogrio

In [2]: meta, table = pyogrio.read_arrow("../Downloads/PYOGRIO_USE_ARROW/Test.gdb/", layer="Test_Layer")

In [3]: meta
Out[3]: 
{'crs': 'EPSG:6339',
 'encoding': 'UTF-8',
 'fields': array(['Shape_Length', 'Shape_Area', 'Field'], dtype=object),
 'geometry_type': 'MultiPolygon',
 'geometry_name': 'Shape',
 'fid_column': 'OBJECTID'}

In [4]: table.schema
Out[4]: 
Shape_Length: double
  -- field metadata --
  GDAL:OGR:default: 'FILEGEODATABASE_SHAPE_LENGTH'
Shape_Area: double
  -- field metadata --
  GDAL:OGR:default: 'FILEGEODATABASE_SHAPE_AREA'
Field: dictionary<values=string, indices=int32, ordered=0>
  -- field metadata --
  GDAL:OGR:domain_name: 'Domain_One'
Shape: binary
  -- field metadata --
  ARROW:extension:name: 'geoarrow.wkb'
  ARROW:extension:metadata: '{"crs":{
  "$schema": "https://proj.org/sche' + 2702

This dictionary field has all missing values, and also the dictionary itself consists of many missing values:

In [5]: dict_array = table["Field"].chunk(0)

In [6]: dict_array.type
Out[6]: DictionaryType(dictionary<values=string, indices=int32, ordered=0>)

In [7]: dict_array.dictionary
Out[7]: 
<pyarrow.lib.StringArray object at 0x7efcee98f580>
[
  null,
  null,
  null,
  null,
  null,
  null,
  null,
  null,
  null,
  null,
  ...
  null,
  null,
  null,
  null,
  null,
  null,
  null,
  null,
  null,
  "Text"
]

In [8]: dict_array.indices
Out[8]: 
<pyarrow.lib.Int32Array object at 0x7efcee98fe20>
[
  null,
  null,
  null,
  null
]

I am not familiar enough with this file format to know if this is the sensible conversion, or if GDAL should be producing different data.

But given this field with a dictionary with null values, this then fails to convert to pandas:

In [9]: dict_array.to_pandas()
...
ValueError: Categorical categories cannot be null

This is a limitation of the pandas Categorical that it can only have nulls in the indices, and not in the categories ("dictionary" in arrow). But given this limitation, I think pyarrow could be removing those nulls (and convert them in nulls in the indices) before converting to pandas, to ensure the conversion still works.

[zachariahBinx]

Thanks for that information. I played around with that code while changing the Domain value. I noticed that when the value is between that problematic range I posted (1-99), table["Field"].chunk(0) returns all null values. While any other values outside of that range returns at least one entry of "Text".

I think I understand why the error is occuring when it has a mix of categories (mainly the null). But it is still unclear why the domain is not found when set between those values.