Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update troubleshooting guides with debug service usage #41396

Merged
merged 5 commits into from
May 21, 2024
Merged

Update troubleshooting guides with debug service usage #41396

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1799c1e
refactor(docs): update docs to include debug service usage
gabrielcorado May 9, 2024
9499b95
docs: code review suggestions
gabrielcorado May 15, 2024
feed43e
docs: update debug service intro definitions
gabrielcorado May 20, 2024
f1a84da
Apply suggestions from code review
gabrielcorado May 21, 2024
3feed70
docs(diagnostics): move teleport-debug-config section up in the guide
gabrielcorado May 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/pages/includes/config-reference/instance-wide.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,3 +130,7 @@ teleport:
format:
output: text
extra_fields: [level, timestamp, component, caller]

# These settings apply to the Debug service:
debug_service:
enabled: true
gabrielcorado marked this conversation as resolved.
Show resolved Hide resolved
4 changes: 4 additions & 0 deletions docs/pages/includes/diagnostics/debug-service-intro.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Teleport includes an internal service called "Debug". This service contains
gabrielcorado marked this conversation as resolved.
Show resolved Hide resolved
commands that can help you troubleshoot Teleport issues. The service is used
through teleport debug commands, which must be executed on the same
machine/container your Teleport instance runs on.
greedy52 marked this conversation as resolved.
Show resolved Hide resolved
6 changes: 6 additions & 0 deletions docs/pages/includes/diagnostics/teleport-debug-config.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
<Admonition type="tip" title="Specify your Teleport configuration path">
If your Teleport configuration is not placed on the default path
(`/var/lib/teleport`), you must to specify its location to the CLI command
using the `-c/--config` flag.
</Admonition>

25 changes: 25 additions & 0 deletions docs/pages/management/admin/troubleshooting.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,29 @@ services such as the Application Service and Database Service.

## Step 1/3. Enable verbose logging

<Tabs>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this Tabs component, I don't think there's a way for the user to tell which tab to choose without looking at both tabs. I would either:

  • Explain when a user would choose a particular tab before the Tabs component
  • Include both possibilities as separate H3 sections and provide context before the first H3.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a short description of both methods.

<TabItem label="With Debug service">
(!docs/pages/includes/diagnostics/debug-service-intro.mdx!)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I gather, the Debug Service does the same thing as enabling verbose logging. Is that true? If not, these instructions should make it clear what starts happening when the Debug Service runs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this context, yes. I've updated both places that have a debug service intro so they can be more focused on the use case.


Change your instance log level without a restart using
`teleport debug set-log-level`:

```bash
$ teleport debug set-log-level DEBUG
Changed log level from "INFO" to "DEBUG".

# If you're unsure what is the current level you can retrieve it.
$ teleport debug get-log-level
Current log level "DEBUG"
```

After troubleshooting, remember to turn the log level back to avoid generating
unnecessary logs.

(!docs/pages/includes/diagnostics/teleport-debug-config.mdx!)
</TabItem>

<TabItem label="Updating configuration">
To diagnose problems, you can configure the `teleport` process to run with
verbose logging enabled by passing it the `-d` flag. `teleport` will write logs
to stderr.
Expand All @@ -33,6 +56,8 @@ teleport:
Restart the `teleport` process to apply the modified log level. Logs will resemble
the following (these logs were printed while joining a server to a cluster, then
terminating the `teleport` process on the server):
</TabItem>
</Tabs>

```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since these are examples of verbose logging, should we associate them with the "Updating configuration" instructions? Otherwise, they seem to be sitting here with no context.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both flows will change the log levels verbosity. I've moved the paragraph from below the logs example, so it gives context and then shows the example.

DEBU [NODE:PROX] Agent connected to proxy: [aee1241f-0f6f-460e-8149-23c38709e46d.tele.example.com aee1241f-0f6f-460e-8149-23c38709e46d teleport-proxy-us-west-2-6db8db844c-ftmg9.tele.example.com teleport-proxy-us-west-2-6db8db844c-ftmg9 localhost 127.0.0.1 ::1 tele.example.com 100.92.90.42 remote.kube.proxy.teleport.cluster.local]. leaseID:4 target:tele.example.com:11106 reversetunnel/agent.go:414
Expand Down
43 changes: 37 additions & 6 deletions docs/pages/management/diagnostics/profiles.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,20 +7,51 @@ Teleport leverages Go's diagnostic capabilities to collect and export
profiling data. Profiles can help identify the cause of spikes in CPU,
the source of memory leaks, or the reason for a deadlock.

## Enable profiling
## Using `teleport debug profile`

(!docs/pages/includes/diagnostics/debug-service-intro.mdx!)

`teleport debug profile` collects a list of pprof profiles. It outputs a
compressed tarball (`.tar.gz`) to STDOUT. You decompress it using `tar` or
direct the result to a file.

By default, it collects `goroutine`, `heap` and `profile` profiles.

```bash
gabrielcorado marked this conversation as resolved.
Show resolved Hide resolved
# Collect default profiles and save to a file.
$ teleport debug profile > pprof.tar.gz
$ tar xvf pprof.tar.gz

# Collect default profiles and decompress it.
$ teleport debug profile | tar xzv -C ./

# Collect "trace" and "mutex" profiles and save to a file.
$ teleport debug profile trace,mutex > pprof.tar.gz
```
greedy52 marked this conversation as resolved.
Show resolved Hide resolved

If you're running Teleport on a Kubernetes cluster you can directly collect
profiles to a local directory without an interactive session:

```bash
$ kubectl -n teleport my-pod -- teleport debug profile > pprof.tar.gz
gabrielcorado marked this conversation as resolved.
Show resolved Hide resolved
```

(!docs/pages/includes/diagnostics/teleport-debug-config.mdx!)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

teleport-debug-config.mdx can be placed up a little? maybe before kube instruction or before visualization section?


## Using diagnosis endpoints
gabrielcorado marked this conversation as resolved.
Show resolved Hide resolved

The profiling endpoint is only enabled if the `--debug` flag is supplied.

(!docs/pages/includes/diagnostics/diag-addr-prereqs-tabs.mdx flags="--debug" !)

## Collecting profiles
### Collecting profiles

Go's standard profiling endpoints are served at `http://127.0.0.1:3000/debug/pprof/`.
Retrieving a profile requires sending a request to the endpoint corresponding
to the desired profile type. When debugging an issue it is helpful to collect
a series of profiles over a period of time.

### CPU
#### CPU
CPU profile shows execution statistics gathered over a user specified period:

``` code
Expand All @@ -31,7 +62,7 @@ $ curl -o cpu.profile http://127.0.0.1:3000/debug/pprof/profile?seconds=30
$ go tool pprof -http : cpu.profile
```

### Goroutine
#### Goroutine

Goroutine profiles show the stack traces for all running goroutines in the system:

Expand All @@ -43,7 +74,7 @@ $ curl -o goroutine.profile http://127.0.0.1:3000/debug/pprof/goroutine
$ go tool pprof -http : goroutine.profile
```

### Heap
#### Heap

Heap profiles show allocated objects in the system:

Expand All @@ -55,7 +86,7 @@ $ curl -o heap.profile http://127.0.0.1:3000/debug/pprof/heap
$ go tool pprof -http : heap.profile
```

### Trace
#### Trace

Trace profiles capture scheduling, system calls, garbage collections, heap size, and other events that are collected by the Go runtime
over a user specified period of time:
Expand Down
3 changes: 3 additions & 0 deletions docs/pages/reference/cli/teleport.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@ The primary commands for the `teleport` CLI are as follows:
| `teleport start` | Starts the `teleport` process in the foreground using the current shell session, including any services configured by the [configuration YAML file](../config.mdx). |
| `teleport status` | Prints the status of the current active Teleport SSH session. |
| `teleport version` | Prints the current release version of the Teleport binary installed on your system. |
| `teleport debug set-log-level` | Changes instance log level. |
| `teleport debug get-log-level` | Fetches instance current log level. |
| `teleport debug debug profile` | Export the application profiles (pprof format). |

<Notice type="tip">
For more information on subcommands when working with the `teleport` cli, use the `--help` option or `teleport <subcommand> --help`.
Expand Down
Loading