dlt_fundamentals_dec_2024

Welcome to the self-paced dlt Fundamentals Course!

In this course you will learn the fundamentals of dlt alongside some of the most important topics in the world of Pythonic data engineering.

Here’s what we’ll be covering:

Lesson 1: Quick Start

Discover what dlt is, run your first pipeline with toy data, and explore it like a pro using DuckDB, sql_client, and dlt datasets!

Lesson 2: dlt Resources and Sources

Learn to run pipelines with diverse data sources (dataframes, databases, and REST APIs), master dlt.resource, dlt.source, and dlt.transformer, and create your first REST API pipeline!

Lesson 3: Pagination & Authentication & dlt Configuration

Since it is never a good idea to publicly put your API keys into your code, different environments have different methods to set and access these secret keys. dlt is no different. Master pagination and authentication for REST APIs, explore dlt's RESTClient and manage secrets and configs.

Lesson 4: Using dlt’s pre-built Sources and Destinations

Now that you took a data source and loaded it into a duckdb destination, it is time to look into what other possibilities dlt offers. In this notebook we will take a look at pre-built verified sources and destinations and how to use them.

Lesson 5: Write disposition and incremental loading

Learn to control data behavior with dlt write dispositions (Append, Replace, Merge), master incremental loading, and efficiently update and deduplicate your datasets.

Lesson 6: How dlt works

Discover the magic behind dlt! Learn its three main steps — Extract, Normalize, Load — along with default behaviors and supported file formats.

Lesson 7: Inspecting & Adjusting Schema

dlt creates and manages the schema automatically, but what if you want to control it yourself? Explore the schema and customize it to your needs easily with dlt!

Lesson 8: Understanding Pipeline State & Metadata

After learning about pipelines and how to move data from one place to another, we now turn to information about the pipeline itself — the metadata that can be accessed and edited through dlt. This notebook explores dlt states, what they collect, and where this extra information is stored. It also expands on what the load info and trace in dlt can do.

That’s it! 🎉

Homework & Certification

You finished the dlt Fundamentals course! As a summary of everything we covered in this course, there’s a homework quiz for you to go through. Hopefully it’ll refresh your memory on everything we covered.

Here you go: https://dlthub.learnworlds.com/course/dlt-fundamentals

Help

If you have any questions or need further assistance, don’t hesitate to reach out—we’re here to help.

Join the Community.