Reorg jetstream into deep dives folder, add autogeneration for sideba…

…r. Small FAQ cleanup. (#538)

docs/jetstream/add-new-platform.md → docs/deep-dives/jetstream/adding-a-platform.md

@@ -1,4 +1,9 @@
-# Adding a Platform
+---
+id: adding-a-platform
+sidebar_position: 10
+title: Adding a Platform
+slug: adding-a-platform 
+---
 
 Jetstream runs analyses for experiments launched on several different platforms, such as Fenix or Firefox Desktop. When adding a new platform in Experimenter, the new platform also needs to be configured in Jetstream to enable automated analyses for launched experiments.
 

docs/jetstream/configuration.md → docs/deep-dives/jetstream/configuration.md

@@ -1,4 +1,6 @@
----
+--- 
+sidebar_position: 5
+slug: configuration
 id: configuration
 title: Configuring Jetstream
 ---
@@ -17,7 +19,7 @@ which should match the experiment slug, like `my-experiment-slug.toml`.
 This works for both Normandy and Nimbus slugs. 
 
 [partybal]: https://protosaur.dev/partybal/
-[outcome]: jetstream/outcomes.md
+[outcome]: ./outcomes
 [metric-hub]: https://github.com/mozilla/metric-hub/tree/main/jetstream
 
 ## Landing configurations
@@ -215,7 +217,7 @@ depends_on = ["moos", "clicks"]
 You should also add some sections to describe how your new metrics should be summarized for reporting.
 You can do this by adding a statistics section to the metric for the statistic you want.
 
-See [Statistics](jetstream/statistics.md) for a list of supported statistics and details about implementing your own.
+See [Statistics](./statistics) for a list of supported statistics and details about implementing your own.
 
 This looks like:
 
@@ -459,7 +461,7 @@ Results for exposure based metrics are currently not visualized in Experimenter.
 
 ## Testing configurations
 
-For more information on how to test configurations see [Testing Jetstream Configs](jetstream/testing.md)
+For more information on how to test configurations see [Testing Jetstream Configs](./testing)
 
 
-[Nimbus exposure event]: ../mobile-feature-api#recording-exposure-events
+[Nimbus exposure event]: /mobile-feature-api#recording-exposure-events

docs/jetstream/data-products.md → docs/deep-dives/jetstream/data-products.md

@@ -1,6 +1,9 @@
 ---
 id: data-products
 title: Jetstream Data Products
+sidebar_position: 6
+sidebar_label: Data products
+slug: data-products
 ---
 
 Jetstream writes analysis results and enrollments information to BigQuery. Statistics data and metadata gets exported to GCS to make it accessible to the Experimenter console.

docs/jetstream/metrics.md → docs/deep-dives/jetstream/metrics.md

@@ -1,6 +1,8 @@
 ---
 id: metrics
 title: Metrics
+sidebar_position: 2
+slug: metrics
 ---
 
 A *metric* describes an aggregation of user activity,
@@ -12,9 +14,9 @@ So is "a user's average page load time."
 
 Metrics are applied over a window of time,
 called an analysis window.
-The [Jetstream overview](jetstream.md) has more details about analysis windows.
+The [Jetstream overview](./overview) has more details about analysis windows.
 
-:::note What's in a name?
+:::tip What's in a name?
 
 [Glean](https://mozilla.github.io/glean/book/index.html)
 also has a concept called
@@ -25,7 +27,7 @@ Jetstream metrics represent aggregations of the recorded values of Glean metrics
 
 :::
 
-[Statistics](statistics.md) summarize the distribution of metrics
+[Statistics](./statistics) summarize the distribution of metrics
 within and between branches of an experiment.
 
 ## How do I add a metric to my experiment?
@@ -34,11 +36,11 @@ A small set of critical guardrail metrics (called Core Firefox Metrics in Nimbus
 specific to each platform is run by default for each experiment.
 These are defined for each platform in the metric-hub repository in [jetstream/defaults/](https://github.com/mozilla/metric-hub/tree/main/jetstream/defaults/). Look at the file for your platform. At the top you will see the metrics collected and the timeframe they are available: daily, weekly, or overall (at the end of the experiment).  For help understanding any aspect of guardrail metrics on your platform - link to the file and ask in #ask-experimenter.
 
-[Outcomes](outcomes.md) are collections of metrics that relate to each other.  You can associate an Outcome with your experiment in the Experiment Console - Metrics section.
+[Outcomes](./outcomes) are collections of metrics that relate to each other.  You can associate an Outcome with your experiment in the Experiment Console - Metrics section.
 Data scientists can extend and define outcomes in the outcomes path of the
 [`metric-hub`](https://github.com/mozilla/metric-hub/tree/main/jetstream/outcomes) repository.  See what [Outcomes are available](https://mozilla.github.io/metric-hub/outcomes/fenix/default-browser/).
 
-If the metrics you need are not covered by Guardrail/Core metrics or existing Outcomes - you can add a metric to your experiment by working with a data scientist to write a [custom configuration](configuration.md) for your experiment.
+If the metrics you need are not covered by Guardrail/Core metrics or existing Outcomes - you can add a metric to your experiment by working with a data scientist to write a [custom configuration](./configuration) for your experiment.
 
 
 

docs/jetstream/operations.md → docs/deep-dives/jetstream/operations.md

@@ -1,9 +1,12 @@
 ---
 id: operations
 title: Jetstream Architecture and Operations
+slug: operations
+sidebar_position: 7
+sidebar_label: Architecture and Operations
 ---
 
-**[Jetstream]** is part of the Cirrus ecosystem and depends on some external services.
+[Jetstream](./overview) is part of the Cirrus ecosystem and depends on some external services.
 
 ![Cirrus overview](/img/jetstream/cirrus.png)
 *High-level overview of Cirrus*
@@ -111,7 +114,7 @@ This date will again be based on the last updated timestamp of the enrollments t
 
 When making changes to experiment-specific configs, jetstream will automatically rerun the affected experiments which will result in the enrollments table getting updated and the most recent configs in metric-hub being used.
 
-More information on how to use the most recent tooling and metric versions can be found [here](jetstream.md/#how-to-use-the-latest-tooling-and-metric-definitions).
+More information on how to use the most recent tooling and metric versions can be found [here](./overview#how-to-use-the-latest-tooling-and-metric-definitions).
 
 [jetstream]: https://github.com/mozilla/jetstream
 [jetstream error dashboard]: https://mozilla.cloud.looker.com/dashboards/246

docs/jetstream/outcomes.md → docs/deep-dives/jetstream/outcomes.md

@@ -1,6 +1,8 @@
 ---
 id: outcomes
 title: Outcomes
+slug: outcomes
+sidebar_position: 4
 ---
 
 An Outcome is a reusable configuration snippet
@@ -50,7 +52,7 @@ If it passes, a data scientist can merge the Outcome definition without addition
 
 Experimenter will need to be re-deployed to pick up a new Outcome. Please ask in #nimbus-project if you're in a hurry!
 
-[Configuring Jetstream]: jetstream/configuration.md
+[Configuring Jetstream]: ./configuration
 
 ## Parameterizing Outcomes
 
@@ -94,5 +96,5 @@ and then copy-paste the metrics into an Outcome for later use.
 ## What happens if an Outcome changes?
 
 Changing an Outcome does not re-run any experiments.
-A commit hash associated with the version of each Outcome is captured in the [experiment metadata](jetstream/data-products.md) published to GCS,
+A commit hash associated with the version of each Outcome is captured in the [experiment metadata](./data-products) published to GCS,
 so it's possible to understand which version of an Outcome was associated with an experiment analysis.

docs/jetstream/jetstream.md → docs/deep-dives/jetstream/overview.md

@@ -1,7 +1,8 @@
 ---
-id: jetstream
-title: Jetstream overview
-slug: /jetstream/jetstream
+id: overview 
+title: Overview
+slug: overview
+sidebar_position: 1
 ---
 
 **[Jetstream]** is an analysis framework for experiments.
@@ -81,7 +82,7 @@ Any telemetry collection can be used as an exposure event, though events are oft
 Many Nimbus features will send a [Nimbus exposure event] automatically when the feature configuration is consulted;
 these are `normandy#expose` events on desktop and `nimbus_events.exposure` events in Glean.
 
-[Nimbus exposure event]: ../mobile-feature-api#recording-exposure-events
+[Nimbus exposure event]: /mobile-feature-api#recording-exposure-events
 
 ## Analysis steps
 
@@ -126,11 +127,11 @@ are available in BigQuery.
 [Technical documentation][jetstream-dtmo]
 is available in the Mozilla data docs.
 
-[experiment configurations]: configuration.md
+[experiment configurations]: ./configuration
 [jetstream]: https://github.com/mozilla/jetstream
 [metric-hub]: https://github.com/mozilla/metric-hub/tree/main/jetstream
 [jetstream-dtmo]: https://docs.telemetry.mozilla.org/datasets/jetstream.html
-[metrics]: metrics.md
-[outcome]: outcomes.md
+[metrics]: ./metrics
+[outcome]: ./outcomes
 [mozanalysis]: https://github.com/mozilla/mozanalysis
-[Jetstream Operations]: operations.md/#tooling-and-metric-versioning
+[Jetstream Operations]: ./operations#tooling-and-metric-versioning

docs/jetstream/statistics.md → docs/deep-dives/jetstream/statistics.md

@@ -1,6 +1,8 @@
 ---
 id: statistics
 title: Statistics
+slug: statistics
+sidebar_position: 3
 ---
 
 Statistics reduce observations of many clients

docs/jetstream/testing.md → docs/deep-dives/jetstream/testing.md

@@ -1,6 +1,8 @@
 ---
 id: testing
 title: Testing Jetstream Configs
+slug: testing
+sidebar_position: 9
 ---
 
 # Testing Jetstream Configs

docs/jetstream/troubleshooting.md → docs/deep-dives/jetstream/troubleshooting.md

@@ -1,6 +1,9 @@
 ---
 id: troubleshooting
 title: Troubleshooting Jetstream
+slug: troubleshooting
+sidebar_label: Troubleshooting
+sidebar_position: 8
 ---
 
 ### How can I see what Jetstream is doing? 

docs/faq/desktop/exposure.mdx

@@ -5,7 +5,7 @@ slug: exposure
 ---
 
 :::tip
-See the [deep dive on growable collections](/cookbook/fml/growable-collections#exposure-events) and [Jetstream enrollment vs. exposure](/jetstream/jetstream/#enrollment-vs-exposure) for more information
+See the [deep dive on growable collections](/cookbook/fml/growable-collections#exposure-events) and [Jetstream enrollment vs. exposure](/deep-dives/jetstream/overview#enrollment-vs-exposure) for more information
 :::
 
 ### Question

docs/faq/feature-restart/feature-restart.mdx

@@ -0,0 +1,18 @@
+---
+id: feature-restart 
+title: Feature restart
+slug: /faq/feature-restart
+---
+
+### Question
+Is feature restart supported?
+
+### Answer
+
+Yes! Feature restart is supported by Nimbus. This is how it works when a feature requires a restart:
+* **Session 1:** Client enrolls, applies feature configuration, but the feature itself won't implement the changes until the next restart
+* **Session 2:** Client is enrolled, feature configuration is applied, feature now shows changes
+
+:::tip Important
+Please share that this experiment needs a restart in your data science Jira ticket and that there is custom work needed at analysis to exlude the pre-restart data.
+:::

docs/faq/general-faq/general-faq.mdx

@@ -2,6 +2,7 @@
 id: general-faq
 title: General FAQ 
 slug: /faq/general-faq
+sidebar_position: 1
 ---
 
 ### Question
@@ -10,4 +11,25 @@ What's the difference between "Experimenter" and "Nimbus"?
 
 ### Answer
 
-Nimbus is the whole experimentation system (handles enrolling/unenrolling users, sending branch configurations, etc).  Nimbus was the code name of the experimentation project.  If you hear Jetstream, that is the whole analysis system (interfaces with data, runs analysis, formats results).  Experimenter is the customer interface to both Nimbus and Jetstream - that you access at experimenter.services.mozilla.com/.  In Experimenter users can configure experiments, manage them (launch, monitor, end), and can see the results.
+Nimbus is the code name of the experimentation project, so Nimbus refers to **the whole experimentation system** (handles enrolling/unenrolling users, sending branch configurations, etc). Experimenter is the **customer interface to both Nimbus and Jetstream** that you access at http://experimenter.services.mozilla.com/. In Experimenter, users can configure experiments, manage them (launch, monitor, end), and can see the results. 
+
+---
+
+### Question
+
+What is Jetstream?
+
+### Answer
+
+[Jetstream](/deep-dives/jetstream/overview) is the whole **analysis system** (interfaces with data, runs analysis, formats results).
+
+---
+
+### Question
+
+What is Normandy?
+
+### Answer
+
+Normandy (https://experimenter.services.mozilla.com/legacy/) is the legacy experimentation platform for Firefox. It is being deprecated in favor of Nimbus.
+

docs/faq/metrics/_category_.yml

@@ -0,0 +1,3 @@
+label: "Metrics"
+collapsible: true
+collapsed: true

docs/faq/metric-availability/non-guardrail-outcome.mdx → docs/faq/metrics/non-guardrail-outcome.mdx

@@ -1,16 +1,19 @@
 ---
 id: feature-metrics
-title: Feature metrics aren't there?
-slug: /metric-availability/feature-metrics
+title: Feature metrics
+slug: /feature-metrics
 ---
 
+### Question
+What do I do if feature metrics aren't there?
+
 :::tip
 Watch [this video](https://mozilla.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=7f677cde-3b61-4cd5-8119-af7b013c7579) to know how metrics work with experiments.  The video covers the steps below. These details are provided here for ease of reference and links.
 :::
 
 After completing your experiment brief and defining your learning goals - attend an office hours to discuss how you could measure the changes.  Based on that discussion - go through the steps below until the metrics needed for this experiment are covered.  You do not need to complete all the steps - just enough to answer your questions.
 
-**Step 1** - Check  [here](/jetstream/metrics/#how-do-i-add-a-metric-to-my-experiment) to learn if the existing Guardrail/Core metrics or Outcomes will answer your questions.
+**Step 1** - Check  [here](/deep-dives/jetstream/metrics#how-do-i-add-a-metric-to-my-experiment) to learn if the existing Guardrail/Core metrics or Outcomes will answer your questions.
 
 **Step 2** - Did an experiment run previously had these metrics you need?  
 * Look at previous experiments in your feature area - do those metrics meet your needs?  Re-using metrics is a much quicker than writing custom metrics from scratch.   

docs/getting-started/data-scientists/auto-sizing-cli.md

@@ -12,7 +12,7 @@ The sizing CLI relies on a local TOML file to configure the job. This TOML file
 
 ### TOML file layout
 
-The TOML configuration file must contain a `metrics`, `data_sources`, `segments`, and `segments.data_sources` section, each containing the definitions for those of interest for the experiments. The definition of each of these follows the same patterns as [Jetstream], and details on how to define your own inside of the TOML file can be found [here](https://experimenter.info/jetstream/configuration#defining-metrics).
+The TOML configuration file must contain a `metrics`, `data_sources`, `segments`, and `segments.data_sources` section, each containing the definitions for those of interest for the experiments. The definition of each of these follows the same patterns as [Jetstream], and details on how to define your own inside of the TOML file can be found [here](/deep-dives/jetstream/configuration#defining-metrics).
 
 The TOML file can also contain references to metrics, segments, data sources, and segment data sources that are already contained in [metric-hub]. To reference these pre-defined objects, an `import_from_metric_hub` list can be included in the TOML file. For instance, to import the `active_hours` metric for Firefox Desktop, the following is included in the TOML config file:
 ```
@@ -52,7 +52,7 @@ The results JSON will include an entry for each combination of power and effect
 ```
 
 
-[Jetstream]: jetstream/jetstream.md
+[Jetstream]: /deep-dives/jetstream/overview
 [metric-hub]: https://github.com/mozilla/metric-hub
 [mozanalysis]: https://github.com/mozilla/mozanalysis
-[auto-sizing]: https://github.com/mozilla/auto-sizing
+[auto-sizing]: https://github.com/mozilla/auto-sizing

docs/getting-started/data-scientists/experiment-sizing.md

@@ -2,6 +2,7 @@
 id: experiment-sizing
 title: Sizing Experiments Using Mozanalysis
 slug: /experiment-sizing
+sidebar_label: Sizing using Mozanalysis
 ---
 
 This page gives an overview of how Mozanalysis can be used to do experiment sizing. **[Mozanalysis]** is a library used to standardize experiment analysis at Mozilla. Mozanalysis contains the statistical tools used by [Jetstream] to produce results from experiments, as well as tools to pull historical data from BigQuery to calculate necessary sample sizes to carry out experiments. The results of the tool are the sample size and percent of the total target population required per branch of an experiment with a balanced design to achieve a given power.
@@ -38,7 +39,7 @@ The time series data returned will contain a row for each client for each time s
 
 Selecting clients for analysis is accomplished by using the [`Segment`](https://github.com/mozilla/mozanalysis/tree/main/src/mozanalysis/segments) objects in Mozanalysis; users can either reuse segments that currently exist in Mozanalysis or define one at runtime. Segments consist of a data source, either a table or view in BigQuery, and a SELECT expression to filter that data source; this SELECT expression must include a SQL aggregate function.
 
-Similarly, Mozanalysis experiment sizing reuses `Metric` objects from [Jetstream](jetstream/metrics.md), and users can reuse metrics that are currently implemented in [Mozanalysis](https://github.com/mozilla/mozanalysis/tree/main/src/mozanalysis/metrics) or in [Jetstream configs](https://github.com/mozilla/metric-hub/tree/main/jetstream), or users can define their own at runtime.
+Similarly, Mozanalysis experiment sizing reuses [`Metric` objects](/deep-dives/jetstream/metrics) from Jetstream, and users can reuse metrics that are currently implemented in [Mozanalysis](https://github.com/mozilla/mozanalysis/tree/main/src/mozanalysis/metrics) or in [Jetstream configs](https://github.com/mozilla/metric-hub/tree/main/jetstream), or users can define their own at runtime.
 
 Metrics and target segments are passed to Mozanalysis experiment sizing as lists of `Segment` or `Metric` objects; users may include multiple of each in the analysis. When multiple `Segment` objects are used, Mozanalysis identifies clients that satisfy the conditions of **all** targets in the list. If users would like to run analyses where `Segment`s should be joined with OR rather than AND, multiple experiment sizing tasks should be completed, for each condition in the OR statement, and aggregate the returned results from each separate study.
 
@@ -60,5 +61,5 @@ Mozanalysis contains functions that take the results from pulling historical dat
 4. Replicating sizing for Waldo, which uses continuous enrollment [(link)](https://colab.research.google.com/drive/1_R4zBUnucRPmHwIVLlPTYInDZwTbCn-F?usp=sharing)
 5. Pulling time series historical data and sizing with empirical sample size calculation [(link)](https://colab.research.google.com/drive/1-XT2DMfGSqiCS18yGPIGCs_YWg75qZzn?usp=sharing)
 
-[Jetstream]: jetstream/jetstream.md
+[Jetstream]: /deep-dives/jetstream/overview
 [mozanalysis]: https://github.com/mozilla/mozanalysis