Post-Europa docs.dagger.io

[gerhard]

Having spent a significant amount of my last six weeks on #1327, I thought that it would help to share my learnings, current state and the steps that I see us taking next.

While this is the continuity that I imagine, now is a good time to share your perspectives and drive the improvements that you would like to see in https://docs.dagger.io as we open up Dagger to the world.

Learnings worth sharing

It served me well to have a clear user in mind and focus on her need to get started with Dagger Europa. It enabled me to keep content cohesive, single-threaded and write less of it. Less content meant a clearer view of what is missing, both while I was writing it and after the fact.

I genuinely believe that what we leave out is more important than what we put in, and I would love us to continue with this approach.

Some past learnings from our users have been lost in the new docs. The absence of the CUE page was the first one to be noticed. A few of our users went out of their way to mention how useful this page was in pre-Europa, 0.1.0. A CUE primer which is focused on Dagger is an absolute must, and we have a few follow-up PRs that address this (more info below).

The other learning that was missed is to show users how to build a Dagger config from scratch. While our pre-Europa Getting Started section had this, I considered that it was more important to show users how the pieces fit together locally, and how to transition that to a CI environment. I still think that showing the high-level mechanics, and then diving into the from scratch config, is the right approach.

The last learning that helped me was to think about content via the following categories:

1. Getting Started
2. Core Concepts
3. Use Cases
4. Reference

While the current version of Getting Started is missing a few pages, Core Concepts is missing dagger project and how to create packages, Use Cases are missing completely and we have a single beginning of a page for Reference, what we have currently is a step in the right direction. If this was a car, it may not have all the doors, windows, seats, etc. but it works.

In my mind, the current version of the docs is our "John":

The original John from the Grand Tour Mongolia Special

What is the current state?

Having less content that flows well and is focused on solving user needs was my key focus for the 0.2.0 docs. I would very much like us to continue on this trajectory with the new doc pages that we add. If you give 10 minutes to Beth Aitman, a technical writer for Google, she will expand on why this is important:

👆🏻 If you like this and would like to read / listen / watch more, you will find more references here #1327 (comment) 👆🏻

With this principle in mind, I would like to list the things which I am aware we are missing:

• Showing a user how to write the todoapp Dagger config in the Getting Started section (AFAIK @talentedmrjones is already working on this)
• A more to-the-point CUE doc and improved Dagger CUE API doc → Include What is CUE? in Europa docs sidebar, Core Concepts #1758
• A page on how to create a package; this is my last task for the Europa docs issue → Europa documentation website #1327 (comment) (help will be appreciated)
• A page on how to make the builds fast; this is blocked on the BuilKit caching that @TomChv owns Support cache export to github action #1020
• Use Cases which is a team effort as it is the content that we are expected to write after a pilot goes live

I would like to write more about Use Cases and pilots as these two are related. I expect this to be the bulk of my effort over the next few weeks. Since each of us holds specific context for pilots (I own the changelog.com and rakwode.dev context), I expect us to come together and work on this as a team, with each of us contributing their context. This is not just about docs content, but about helping others use Dagger Europa to solve their specific CI/CD issues. When they are successful, I would expect you to summarise the work that you have done with them in a doc page. The only example that we have so far is not complete, but it is sufficient to show the end-result that we are aiming for: https://docs.dagger.io/1211/go-docker-swarm/

While I know that the above list is incomplete, my intention is to share enough information so that you all can get involved and continue the https://docs.dagger.io momentum in the way that you feel is valuable. Remember, we favour intent-lead actions.

My last point would be to encourage you to be aware of tangents. While exploring unknowns and contributing other content are great ideas in principle, each of us should be aware of:

1. Keeping docs consistent and to-the-point
2. What actually matters to our users - Getting Started and solving specific issues via Core Concepts
3. Show the world what Dagger helped others achieve - Use Cases

What happens next?

Share your thoughts and, if you feel courageous (today is a great day for that!), perhaps share what you intend to do via a comment below. The goal that we are working towards is to share and then implement the changes that we would like to see in https://docs.dagger over the next two weeks, before we launch.

I have added the PRs that I have in flight to the Delivery plan below, and will continue doing so as I become aware of other PRs.

If you open a PR which is related to this issue, remember to add Part of #1804 in the description. This will automatically link it to this issue, which will greatly help with visibility.

If you want, you can CC @gerhard and/or @shykes on PRs which are related to this. Since we have the most https://docs.dagger.io context, we are most likely to know what needs to happen for those PRs to get merged.

Delivery plan

• Add What is Cue? page to Europa Docs sidebar #1799
• Add missing definitions to Dagger CUE API docs page #1800
• ...

[helderco]

Maybe we could have a common pitfalls page. Examples would be recommending disjunctions over if conditions and templates over for loops. We can leave in draft and add to it until it's mature enough to publish.

Also, sometimes I feel docs are too basic and need more examples (in general). Maybe a section for more in-depth or advanced concepts? React has a Main Concepts section and an Advanced Guides and I've found those helpful. You can go to the basics if you're starting out, but if you're further along and need more, you have that as well. I'm saying later on, after the basics are solid.

Where would you put a page on How to debug? This is something I jotted down when I was starting out. You pick up a few "tricks" along the way that I think are useful to someone starting out.

To finish, I can contribute the Create your own package page since I'm reusing a private package for a while now. I can also help with the Swarm use case.

[gerhard]

All excellent ideas @helderco!

This is the https://docs.dagger.io that I imagine us all working towards over the coming weeks:

DOCS STRUCTURE TODAY                        DOCS STRUCTURE TOMORROW
--------------------                        -----------------------

⬅️ Dagger 0.1                                ⬅️ Dagger 0.1
Migrate from Dagger 0.1                     Migrate from Dagger 0.1

Getting Started                             Getting Started
|- CI/CD in your local dev                  |- CI/CD in your local dev
|- From local dev to CI environment         |- From local dev to CI environment ✨ new CI envs
                                            |- 🆕 todoapp the hard way ✨ build the config line-by-line and go deep
                                            |- 🆕 helloworld the easy way ✨ this is the first thing that some community members do → https://github.com/dagger/dagger/pull/1817

Core Concepts                               Core Concepts
|- It all starts with a plan                |- It all starts with a plan
|- Interacting with the client              |- Interacting with the client
|- How to use secrets                       |- How to use secrets
|- Building container images                |- Building container images
                                            |- 🆕 Make your builds fast ✨ BuildKit caching
                                            |- 🆕 Create your own package
|- What is CUE?                             |- What is CUE?
|- Dagger CUE API                           |- Dagger CUE API

                                            🆕 Advanced Concepts
                                            |- 🆕 Dagger architecture
                                            |- 🆕 How to debug?
                                            |- 🆕 Remote Docker on Linux ✨ fastest Docker experience
                                            |- 🆕 Share remote Docker on Linux with CI ✨ free shared cache, Tailscale integration (exactly what we did for changelog.com in https://github.com/thechangelog/changelog.com/pull/395)
                                            |- 🆕 Multi-platform builds

                                            🆕 Use Cases
                                            |- 🆕 Dagger with Dagger → https://github.com/dagger/dagger/issues/1522
                                            |- 🆕 Next.js on Netlify ✨ dagger.io → https://github.com/dagger/dagger.io/pull/117
                                            |- 🆕 Docusaurus on Netlify ✨ docs.dagger.io
                                            |- 🆕 .NET on Vercel ✨ nabeel.dev
                                            |- 🆕 Svelte on Vercel with Pulumi ✨ rawkode.dev
                                            |- 🆕 Symfony on Docker Swarm ✨ huma-hum.fr
                                            |- 🆕 Go on Docker Swarm ✨ particubes.com migrated to Europa
                                            |- 🆕 Go on Docker Hub → https://github.com/dagger/dagger/pull/1805
                                            |- 🆕 Phoenix on Kubernetes → https://github.com/dagger/dagger/issues/1336
                                            |- 🆕 Phoenix on fly.io → https://github.com/thechangelog/changelog.com/pull/407

🙋‍♂️ @dagger/team @maert @nabsul @aduermael @rawkode ☝🏻

[helderco]

What about Manage packages using the package manager? It's the next logical step to Create your own package (Develop a new CUE package for Dagger).

I'll write both.

Note: Using the package manager can be used standalone if you're reusing packages from others, but there's no sense in creating a package for yourself without using the package manager to install/update. I've found before (in Discord) confusion on this. Makes me wonder if it would be best to show the package manager first (we need a simple dagger repo for this, where you can just follow the doc examples and it would work), and then how to create one for yourself. I think this would be clearer.

Feature idea: We could have a dagger project edit <repo> to automatically pull and symlink. This is what we do in python.

[gerhard]

@talentedmrjones I think that it will help to have the above context for when we talk docs next 🙂

[RemakingEden]

Hi everyone, I've been looking at Dagger over the weekend and I'm excited by its possibilities! I think I've started to get to grips with how to build, run tests and deploy with it.

One thing issue I'm running into is knowing how to do one other part of my CI flow. Using pre-built tools to check the code, for example how to use this tool? https://github.com/zricethezav/gitleaks. On Github I would use the prebuilt action for this and on Gitlab I would simply run a step in the container and then execute the command gitleaks detect.

Is it expected that I would install Brew and install this tool manually or is there a way to run this container and bring the repo in then execute?

The reason I'm asking on this discussion is that I couldn't find any explanation in the docs. If someone can point me to this I'll happily be on my way. If not and others see it as a valuable use case I'm happy to try writing some docs on it.

[helderco]

Yes, you can create an action that does this. If you're willing to contribute docs on that, please do! We appreciate contributions 🙂

Just a few pointers. There's two ways to go about it:

1. Run the whole plan once (dagger do deploy), but you need to make sure that the gitleaks action is a dependency of something in order to run before it. We don't have explicit dependencies yet, so if you need actions do depend on this one's success, and there's no clear field to use as a dependency, you can make a fake one (e.g., create an env var in actions that depend on this, to gitleak's success status, even if you don't use this env var).
2. Do this pre-check in your CI (i.e., dagger do gitleaks), as a separate step from the main one (dagger do deploy).

Either case, the action is defined in dagger, it's just how you wire it to the rest.

[RemakingEden]

Thanks for the reply @helderco. Thats good advice, I would probably add it as a dependency of test I imagine. I'm going to try and mock one of current workflows that I have on a project.

Is it expected that I would install Brew and install this tool manually or is there a way to run this container and bring the repo in then execute?

As for the docker vs install locally question. Any recommendation on what would be best?

[helderco]

We definitely recommend running everything in containers so you don't depend on the local host. It's more portable this way.

[RemakingEden]

Ive managed to get an example of a pipeline using some tools to very basically check the code for linting/security issues and run some integration tests. https://github.com/RemakingEden/Dagger-Node-Github-Example/blob/main/eden.cue.

Where the "Go on Docker Swarm" use case lost me a little is its focus on the build and can be a little complex.

I wonder if a use case that has:

• A build step npm install
• Some tests (int and unit)
• Some containerised tests (Gitleaks, SonarScanner, audit-ci)
• A publish step (Pushes to Docker hub)

Obviously this may be over simplified for devs doing complex deployment but for myself this would have been a great scaffolding to build on. What do you think of this use case? Do you think it would be benificial?

[shykes]

The more use cases the better! The one you describe seems very useful.

[RemakingEden]

Great i'll get on that now then, trying to use the same style as the "Go on Docker Swarm" use case. Thanks.