You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/website/docs/dlt-ecosystem/verified-sources/sql_database/configuration.md
+56Lines changed: 56 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -18,6 +18,10 @@ import Header from '../_source-info-header.md';
18
18
19
19
Read more about sources and resources here: [General usage: source](../../../general-usage/source.md) and [General usage: resource](../../../general-usage/resource.md).
20
20
21
+
:::note NOTE
22
+
To see complete list of source arguments for `sql_database`[refer to the this section](#arguments-for-sql_database-source).
23
+
:::
24
+
21
25
### Example usage:
22
26
23
27
:::tip
@@ -408,3 +412,55 @@ print(info)
408
412
```
409
413
With the dataset above and a local PostgreSQL instance, the `ConnectorX` backend is 2x faster than the `PyArrow` backend.
410
414
415
+
## Arguments for `sql_database` source
416
+
The following arguments can be used with the `sql_database` source:
417
+
418
+
`credentials` (Union[ConnectionStringCredentials, Engine, str]): Database credentials or an `sqlalchemy.Engine` instance.
419
+
420
+
`schema` (Optional[str]): Name of the database schema to load (if different from default).
421
+
422
+
`metadata` (Optional[MetaData]): Optional `sqlalchemy.MetaData` instance. `schema` argument is ignored when this is used.
423
+
424
+
`table_names` (Optional[List[str]]): A list of table names to load. By default, all tables in the schema are loaded.
425
+
426
+
`chunk_size` (int): Number of rows yielded in one batch. SQL Alchemy will create additional internal rows buffer twice the chunk size.
427
+
428
+
`backend` (TableBackend): Type of backend to generate table data. One of: "sqlalchemy", "pyarrow", "pandas" and "connectorx".
429
+
430
+
- "sqlalchemy" yields batches as lists of Python dictionaries, "pyarrow" and "connectorx" yield batches as arrow tables, "pandas" yields panda frames.
431
+
432
+
- "sqlalchemy" is the default and does not require additional dependencies,
433
+
434
+
- "pyarrow" creates stable destination schemas with correct data types,
435
+
436
+
- "connectorx" is typically the fastest but ignores the "chunk_size" so you must deal with large tables yourself.
437
+
438
+
`detect_precision_hints` (bool): Deprecated. Use `reflection_level`. Set column precision and scale hints for supported data types in the target schema based on the columns in the source tables. This is disabled by default.
439
+
440
+
`reflection_level`: (ReflectionLevel): Specifies how much information should be reflected from the source database schema.
441
+
442
+
- "minimal": Only table names, nullability and primary keys are reflected. Data types are inferred from the data. This is the default option.
443
+
444
+
- "full": Data types will be reflected on top of "minimal". `dlt` will coerce the data into reflected types if necessary.
445
+
446
+
- "full_with_precision": Sets precision and scale on supported data types (ie. decimal, text, binary). Creates big and regular integer types.
447
+
448
+
`defer_table_reflect` (bool): Will connect and reflect table schema only when yielding data. Requires table_names to be explicitly passed.
449
+
Enable this option when running on Airflow. Available on dlt 0.4.4 and later.
450
+
451
+
`table_adapter_callback`: (Callable): Receives each reflected table. May be used to modify the list of columns that will be selected.
452
+
453
+
`backend_kwargs` (**kwargs): kwargs passed to table backend ie. "conn" is used to pass specialized connection string to connectorx.
454
+
455
+
`include_views` (bool): Reflect views as well as tables. Note view names included in `table_names` are always included regardless of this setting. This is set to false by default.
456
+
457
+
`type_adapter_callback`(Optional[Callable]): Callable to override type inference when reflecting columns.
458
+
Argument is a single sqlalchemy data type (`TypeEngine` instance) and it should return another sqlalchemy data type, or `None` (type will be inferred from data)
459
+
460
+
`query_adapter_callback`(Optional[Callable[Select, Table], Select]): Callable to override the SELECT query used to fetch data from the table. The callback receives the sqlalchemy `Select` and corresponding `Table`, 'Incremental` and `Engine` objects and should return the modified `Select` or `Text`.
461
+
462
+
`resolve_foreign_keys` (bool): Translate foreign keys in the same schema to `references` table hints.
463
+
May incur additional database calls as all referenced tables are reflected.
464
+
465
+
`engine_adapter_callback` (Callable[[Engine], Engine]): Callback to configure, modify and Engine instance that will be used to open a connection ie. to set transaction isolation level.
0 commit comments