Missing numpy dependency when running with PyArrow >= 18

[kien-truong]

dlt version

1.7

Describe the problem

Because PyArrow >= 18 moves numpy to optional runtime dependencies, when running pipelines with SQL sources and pyarrow backend, it'll fail with an import error

ModuleNotFoundError: No module named 'numpy'

Expected behavior

Pipelines using pyarrow backend works with pyarrow >= 18

Steps to reproduce

• Install dlt with pyarrow >= 18
• Run a dlt pipeline using SQL source and pyarrow backend

Operating system

Linux

Runtime environment

Local

Python version

3.12

dlt data source

Any SQL source

dlt destination

No response

Other deployment details

No response

Additional information

No response

[rudolfix]

@kien-truong thanks for pointing this. numpy should be optional and dlt should not raise here. could you paste the full stack trace:

---

note on how to fix this:

1. make sure sql_database works without numpy (arrow backend). there's must be a top level import somewhere after refactor!
2. add numpy to arrow extras explicitly

[kien-truong]

My stacktrace are mangled, but the exception comes from this line

dlt/dlt/common/libs/pyarrow.py

Lines 726 to 733 in 7b3c51e

	def transpose_rows_to_columns(
	rows: TDataItems, column_names: Iterable[str]
	) -> dict[str, Any]: # dict[str, np.ndarray]
	"""Transpose rows (data items) into columns (numpy arrays). Returns a dictionary of {column_name: column_data}
	Uses pandas if available. Otherwise, use numpy, which is slower
	"""
	import numpy as np