Merge pull request #50 from bsc-dom/telemetry-doc

Telemetry doc

Author: alexbarcelo

docs/index.rst

@@ -10,8 +10,6 @@ dataClay
    main-concepts
    alien-objects
    advanced-usage
-   telemetry
-   hpc-tracing
    examples/index
 
 .. toctree::
@@ -25,12 +23,22 @@ dataClay
    deployment/hpc-manual-deployment
    deployment/compile-redis
 
+.. toctree::
+   :hidden:
+   :caption: Telemetry
+
+   telemetry/configuration
+   telemetry/offline
+   telemetry/real-time
+   telemetry/prometheus
+   telemetry/hpc-tracing
+
 .. toctree::
    :hidden:
    :caption: Release Notes
 
-   releasenotes/3-x
    releasenotes/4-x
+   releasenotes/3-x
 
 .. toctree::
    :hidden:

docs/telemetry.rst

@@ -0,0 +1,41 @@
+Telemetry Configuration
+=======================
+
+dataClay is instrumented with `OpenTelemetry <https://opentelemetry.io/>`_ to allow observability of
+distributed traces, metrics, and logs. You can configure tracing to export telemetry data either in real-time or for post-mortem analysis. Visualizations can be performed in Grafana.
+
+Configuration
+-------------
+
+To activate tracing in dataClay, the following environment variables need to be set:
+
+- **`DATACLAY_TRACING`**: Set to `true` to enable tracing.
+- **`DATACLAY_TRACING_EXPORTER`**: Export traces to the OpenTelemetry Collector (`otlp`) or print traces to the console (`console`). The default is `otlp`.
+- **`DATACLAY_TRACING_HOST`**: Host of the OpenTelemetry Collector (default: `localhost`).
+- **`DATACLAY_TRACING_PORT`**: Port of the OpenTelemetry Collector (default: `4317`).
+- **`DATACLAY_SERVICE_NAME`**: The service name, which identifies dataClay components in trace data.
+
+Metrics
+-------
+
+.. list-table::
+   :header-rows: 1
+
+   * - Metric
+     - Description
+     - Service
+   * - dataclay_inmemory_objects
+     - Number of objects in memory
+     - backend, client
+   * - dataclay_loaded_objects
+     - Number of loaded objects
+     - backend
+   * - dataclay_stored_objects
+     - Number of stored objects
+     - backend
+   * - dataclay_inmemory_misses_total
+     - Number of inmemory misses
+     - backend, client
+   * - dataclay_inmemory_hits_total
+     - Number of inmemory hits
+     - backend, client

docs/telemetry/configuration.rst

@@ -1,12 +1,9 @@
-===========
 HPC Tracing
 ===========
 
-How to generate paraver traces in MN5
-=====================================
+How to generate paraver traces in MN5 using COMPSs
+--------------------------------------------------
 
-Using COMPSs
-------------
 In order to get the traces we will create a script.
 
 - First we have to import the COMPSs and DataClay modules in order to be able to use them, as well as defining which python version we will be using:
@@ -57,7 +54,8 @@ In order to generate the paraver files, we will call another COMPSs script, "com
 If we run this script in the same directory where we found the traces ($HOME/.COMPSs/[SLURM_JOB_ID]/trace/), the paraver files will appear.
 
 How to inspect the traces in Paraver
-====================================
+------------------------------------
+
 To be able to see these files we will have to open them using the following commands:
 
 .. code-block:: bash

docs/hpc-tracing.rst docs/telemetry/hpc-tracing.rstdocs/hpc-tracing.rst renamed to docs/telemetry/hpc-tracing.rst

@@ -0,0 +1,62 @@
+
+Offline Telemetry Example
+=========================
+
+This example demonstrates exporting OpenTelemetry traces to a JSON file for post-mortem analysis in Grafana.
+
+1. **Activate tracing** by setting environment variables as described in the `telemetry configuration <https://dataclay.bsc.es/docs/telemetry/configuration>`_.
+2. **Generate traces**:
+
+   - Navigate to the `json-exporter` folder in the `offline telemetry example JSON exporter <https://github.com/bsc-dom/dataclay/tree/telemetry-doc/examples/telemetry/offline/json-exporter>`_.
+   - Start dataClay and OpenTelemetry Collector services:
+     
+     .. code-block:: bash
+
+        docker compose up
+     
+   - Run the dataClay client:
+     
+     .. code-block:: bash
+
+        python3 client.py
+     
+   - Traces are exported to the `traces` folder. You can visualize the JSON traces in Grafana.
+
+3. **Visualize in Grafana**:
+
+   - Navigate to the `json-post-mortem` folder in the `offline telemetry example post-mortem <https://github.com/bsc-dom/dataclay/tree/telemetry-doc/examples/telemetry/offline/json-post-mortem>`_.
+   - Start the OpenTelemetry Collector, Tempo, and Grafana services:
+     
+     .. code-block:: bash
+
+        docker compose up
+     
+   - Open Grafana at <http://localhost:3000> (default username/password: `admin`/`admin`).
+   - In the `Explore` section, select `Tempo` as the data source and use the `Trace ID` field to query traces.
+
+4. **Alternative Trace Export**:
+
+   - Run the OpenTelemetry Collector manually:
+     
+     .. code-block:: bash
+
+        docker run \
+        -v ./config/otel-collector.yaml:/etc/otel-collector.yaml \
+        otel/opentelemetry-collector-contrib \
+        "--config=/etc/otel-collector.yaml"
+
+5. **Copy Traces from MareNostrum 5**:
+
+   - To analyze traces from MareNostrum 5, copy them locally:
+     
+     .. code-block:: bash
+
+        scp transfer1.bsc.es:~/.dataclay/otel-traces.json ./traces/otel-traces.json
+
+6. **Troubleshooting**:
+
+   - If permission issues arise for the `/traces` folder, adjust permissions:
+     
+     .. code-block:: bash
+
+        sudo chmod -R 777 traces

docs/telemetry/offline.rst

@@ -0,0 +1,78 @@
+Prometheus
+==========
+
+
+Metrics
+-------
+
+.. list-table::
+   :header-rows: 1
+
+   * - Metric
+     - Description
+     - Service
+   * - dataclay_inmemory_objects
+     - Number of objects in memory
+     - backend, client
+   * - dataclay_loaded_objects
+     - Number of loaded objects
+     - backend
+   * - dataclay_stored_objects
+     - Number of stored objects
+     - backend
+   * - dataclay_inmemory_misses_total
+     - Number of inmemory misses
+     - backend, client
+   * - dataclay_inmemory_hits_total
+     - Number of inmemory hits
+     - backend, client
+
+
+Deploy dataClay with Prometheus
+-------------------------------
+
+Run dataClay with Prometheus:
+
+.. note::
+    This example is available in `GitHub <https://github.com/bsc-dom/dataclay/tree/main/examples/telemetry/prometheus>`__.
+
+.. code-block:: bash
+
+    docker compose up -d
+
+The ``metadata-service`` and ``backends`` will post their metrics to the ``8000`` port.
+Prometheus is configured to scrape this port to pull the metrics.
+
+Access Prometheus at `http://localhost:9090 <http://localhost:9090>`_. You can query the metrics defined above.
+
+
+Deploy dataClay with Prometheus Pushgateway
+-------------------------------------------
+
+Run dataClay with Prometheus Pushgateway:
+
+.. note::
+    This example is available in `GitHub <https://github.com/bsc-dom/dataclay/tree/main/examples/telemetry/prometheus-pushgateway>`__.
+
+.. code-block:: bash
+    
+    docker compose up -d
+
+
+The ``metadata-service`` and ``backends`` will push their metrics to the ``pushgateway`` at the ``9091`` port.
+
+The ``client.py`` can also push metrics using the ``pushgateway``:
+
+.. code-block:: bash
+
+    export DATACLAY_METRICS=true
+    export DATACLAY_METRICS_EXPORTER=pushgateway
+    export DATACLAY_METRICS_HOST=localhost  # the default
+    export DATACLAY_METRICS_PORT=9091
+    python3 client.py
+
+
+Access the Pushgateway at `http://localhost:9091 <http://localhost:9091>`_ and Prometheus at `http://localhost:9090 <http://localhost:9090>`_.
+
+.. note::
+    When using ``pushgateway``, a new Python thread will run to push the metrics every 10 seconds (default).