You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In this quickstart, you'll use the SQLMesh command line interface (CLI) to get up and running with SQLMesh's scaffold generator. This example project will run locally on your computer using [DuckDB](https://duckdb.org/) as an embedded SQL engine.
3
+
In this quickstart, you'll use the SQLMesh command line interface (CLI) to get up and running with SQLMesh's scaffold generator.
4
+
5
+
It will create an example project that runs locally on your computer using [DuckDB](https://duckdb.org/) as an embedded SQL engine.
4
6
5
7
Before beginning, ensure that you meet all the [prerequisites](../prerequisites.md) for using SQLMesh.
6
8
@@ -39,36 +41,173 @@ mkdir sqlmesh-example
39
41
cd sqlmesh-example
40
42
```
41
43
42
-
If using a python virtual environment, ensure it's activated first by running the `source .env/bin/activate` command from the folder used during [installation](../installation.md).
44
+
If using a Python virtual environment, ensure it's activated first by running the `source .env/bin/activate` command from the folder used during [installation](../installation.md).
45
+
46
+
### 1.1 Initialize the project
47
+
48
+
SQLMesh includes a scaffold generator to initialize a new SQLMesh project.
49
+
50
+
The scaffold generator will ask you some questions and create a SQLMesh configuration file based on your responses.
43
51
44
-
Create a SQLMesh scaffold with the following command, specifying a default SQL dialect for your models. The dialect should correspond to the dialect most of your models are written in; it can be overridden for specific models in the model's `MODEL` specification. All SQL dialects [supported by the SQLGlot library](https://github.com/tobymao/sqlglot/blob/main/sqlglot/dialects/dialect.py) are allowed.
52
+
Depending on your answers, it will also create multiple files for the SQLmesh example project used in this quickstart.
45
53
46
-
In this example, we specify the `duckdb` dialect:
54
+
Start the scaffold generator by executing the `sqlmesh init` command:
47
55
48
56
```bash
49
-
sqlmesh init duckdb
57
+
sqlmesh init
50
58
```
51
59
52
-
The scaffold will include a SQLMesh configuration file for the example project.
60
+
??? info "Skip the questions"
61
+
62
+
If you don't want to use the interactive scaffold generator, you can initialize your project with arguments to the [`sqlmesh init` command](../reference/cli.md#init).
63
+
64
+
The only required argument is `engine`, which specifies the SQL engine your project will use. Specify one of the engine `type`s in the [list of supported engines](../integrations/overview.md#execution-engines).
65
+
66
+
In this example, we specify the `duckdb` engine:
67
+
68
+
```bash
69
+
sqlmesh init duckdb
70
+
```
71
+
72
+
The scaffold will include a SQLMesh configuration file and example project directories and files. You're now ready to continue the quickstart [below](#2-create-a-prod-environment).
73
+
74
+
#### Project type
75
+
76
+
The first question asks about the type of project you want to create. Enter the number corresponding to the type of project you want to create and press `Enter`.
77
+
78
+
```bash
79
+
──────────────────────────────
80
+
Welcome to SQLMesh!
81
+
──────────────────────────────
82
+
83
+
What type of project do you want to set up?
84
+
85
+
[1] DEFAULT - Create SQLMesh example project models and files
86
+
[2] dbt - You have an existing dbt project and want to run it with SQLMesh
87
+
[3] EMPTY - Create a SQLMesh configuration file and project directories only
88
+
89
+
Enter a number: 1
90
+
```
91
+
92
+
For this quickstart, choose the `DEFAULT` option `1` so the example project files are included in the project directories.
93
+
94
+
#### SQL engine
95
+
96
+
The second question asks which SQL engine your project will use. SQLMesh will include that engine's connection settings in the configuration file, which you will fill in later to connect your project to the engine.
97
+
98
+
For this quickstart, choose the `DuckDB` option `1` so we can run the example project with the built-in DuckDB engine that doesn't need additional configuration.
99
+
100
+
```bash
101
+
Choose your SQL engine:
102
+
103
+
[1] DuckDB
104
+
[2] Snowflake
105
+
[3] Databricks
106
+
[4] BigQuery
107
+
[5] MotherDuck
108
+
[6] ClickHouse
109
+
[7] Redshift
110
+
[8] Spark
111
+
[9] Trino
112
+
[10] Azure SQL
113
+
[11] MSSQL
114
+
[12] Postgres
115
+
[13] GCP Postgres
116
+
[14] MySQL
117
+
[15] Athena
118
+
[16] RisingWave
119
+
120
+
Enter a number: 1
121
+
```
122
+
123
+
#### CLI mode
124
+
125
+
SQLMesh's core commands have multiple options that alter their behavior. Some of those options streamline the SQLMesh `plan` workflow and CLI output.
126
+
127
+
If you prefer a streamlined workflow, choose the `FLOW` CLI mode to automatically include those options in your project configuration file. If you prefer to see all the output SQLMesh provides, choose `DEFAULT` mode.
128
+
129
+
```bash
130
+
Choose your SQLMesh CLI experience:
131
+
132
+
[1] DEFAULT - See and control every detail
133
+
[2] FLOW - Automatically run changes and show summary output
134
+
135
+
Enter a number: 1
136
+
```
137
+
138
+
#### Ready to go
139
+
140
+
Your project is now ready to go, and SQLMesh displays a message with some good next steps.
141
+
142
+
If you chose the DuckDB engine, you're ready to move forward and run the example project with DuckDB.
143
+
144
+
If you chose a different engine, add your engine's connection information to the `config.yaml` file before you run any additional SQLMesh commands.
145
+
146
+
```bash
147
+
Your SQLMesh project is ready!
148
+
149
+
Next steps:
150
+
• Update your gateway connection settings (e.g., username/password) in the project configuration file:
??? info "Learn more about the project's configuration"
55
165
SQLMesh project-level configuration parameters are specified in the `config.yaml` file in the project directory.
56
166
57
-
This example project uses the embedded DuckDB SQL engine, so its configuration specifies `duckdb` as the local gateway's connection and the `local` gateway as the default.
167
+
This example project uses the embedded DuckDB SQL engine, so its configuration specifies `duckdb` as the gateway's connection type. All available configuration settings are included in the file, with optional settings set to their default value and commented out.
58
168
59
-
The command to run the scaffold generator **requires** a default SQL dialect for your models, which it places in the config `model_defaults` `dialect` key. In this example, we specified the `duckdb` SQL dialect as the default:
169
+
SQLMesh requires a default model SQL dialect. SQLMesh automatically specifies the SQL dialect for your project's SQL engine, which it places in the config `model_defaults` `dialect` key. In this example, we specified the DuckDB engine, so `duckdb` is the default SQL dialect:
60
170
61
171
```yaml linenums="1"
172
+
# --- Gateway Connection ---
62
173
gateways:
63
-
local:
174
+
duckdb:
64
175
connection:
65
-
type: duckdb
66
-
database: ./db.db
67
-
68
-
default_gateway: local
176
+
# For more information on configuring the connection to your execution engine, visit:
Learn more about SQLMesh project configuration [here](../reference/configuration.md).
@@ -133,13 +272,19 @@ SQLMesh's key actions are creating and applying *plans* to *environments*. At th
133
272
134
273
SQLMesh's key actions are creating and applying *plans* to *environments*.
135
274
136
-
A [SQLMesh environment](../concepts/environments.md) is an isolated namespace containing models and the data they generated. The most important environment is `prod` ("production"), which consists of the databases behind the applications your business uses to operate each day. Environments other than `prod` provide a place where you can test and preview changes to model code before they go live and affect business operations.
275
+
A [SQLMesh environment](../concepts/environments.md) is an isolated namespace containing models and the data they generated.
276
+
277
+
The most important environment is `prod` ("production"), which consists of the databases behind the applications your business uses to operate each day. Environments other than `prod` provide a place where you can test and preview changes to model code before they go live and affect business operations.
278
+
279
+
A [SQLMesh plan](../concepts/plans.md) contains a comparison of one environment to another and the set of changes needed to bring them into alignment.
137
280
138
-
A [SQLMesh plan](../concepts/plans.md) contains a comparison of one environment to another and the set of changes needed to bring them into alignment. For example, if a new SQL model was added, tested, and run in the `dev` environment, it would need to be added and run in the `prod` environment to bring them into alignment. SQLMesh identifies all such changes and classifies them as either breaking or non-breaking.
281
+
For example, if a new SQL model was added, tested, and run in the `dev` environment, it would need to be added and run in the `prod` environment to bring them into alignment. SQLMesh identifies all such changes and classifies them as either breaking or non-breaking.
139
282
140
-
Breaking changes are those that invalidate data already existing in an environment. For example, if a `WHERE` clause was added to a model in the `dev` environment, existing data created by that model in the `prod` environment are now invalid because they may contain rows that would be filtered out by the new `WHERE` clause. Other changes, like adding a new column to a model in `dev`, are non-breaking because all the existing data in `prod` are still valid to use - only new data must be added to align the environments.
283
+
Breaking changes are those that invalidate data already existing in an environment. For example, if a `WHERE` clause was added to a model in the `dev` environment, existing data created by that model in the `prod` environment are now invalid because they may contain rows that would be filtered out by the new `WHERE` clause.
141
284
142
-
After SQLMesh creates a plan, it summarizes the breaking and non-breaking changes so you can understand what will happen if you apply the plan. It will prompt you to "backfill" data to apply the plan - in this context, backfill is a generic term for updating or adding to a table's data (including an initial load or full refresh).
285
+
Other changes, like adding a new column to a model in `dev`, are non-breaking because all the existing data in `prod` are still valid to use - only new data must be added to align the environments.
286
+
287
+
After SQLMesh creates a plan, it summarizes the breaking and non-breaking changes so you can understand what will happen if you apply the plan. It will prompt you to "backfill" data to apply the plan. (In this context, backfill is a generic term for updating or adding to a table's data, including an initial load or full refresh.)
143
288
144
289
The first SQLMesh plan must execute every model to populate the production environment. Running `sqlmesh plan` will generate the plan and the following output:
0 commit comments