Merge branch 'main' into logger-config

.flake8

@@ -0,0 +1,4 @@
+[flake8]
+max-line-length = 100
+extend-ignore = E203,E701
+exclude = */proto/*_pb2*.py

.github/workflows/docker-publish.yml

@@ -20,7 +20,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ['3.9', '3.10', '3.11', '3.12']
+        python-version: ['3.9', '3.10', '3.11', '3.12', '3.13']
 
     permissions:
       contents: read

.github/workflows/tests.yml

@@ -5,24 +5,25 @@ on:
   - pull_request
 
 jobs:
-  build:
+  generate-jobs:
+    runs-on: ubuntu-latest
+    outputs:
+      session: ${{ steps.set-matrix.outputs.session }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: wntrblm/nox@main
+      - id: set-matrix
+        shell: bash
+        run: echo session=$(nox --json -l -s tests | jq -c '[.[].session]') | tee --append $GITHUB_OUTPUT
+  checks:
+    name: Session ${{ matrix.session }}
+    needs: [generate-jobs]
     runs-on: ubuntu-latest
     strategy:
+      fail-fast: false
       matrix:
-        python-version: ["3.9", "3.10", "3.11", "3.12"]
-
+        session: ${{ fromJson(needs.generate-jobs.outputs.session) }}
     steps:
       - uses: actions/checkout@v4
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v5
-        with:
-          python-version: ${{ matrix.python-version }}
-      - name: Install dependencies
-        run: |
-          python -m pip install --upgrade pip
-          python -m pip install tox tox-gh-actions
-          # TODO: when calling pytest, it build a new dev image, but it should build the image
-          # using the specific python version. Before it was working cause the compose was not
-          # building the image, rather the image was built here with the {{ matrix.python-version}}
-      - name: Test with tox
-        run: tox
+      - uses: wntrblm/nox@main
+      - run: nox -s "${{ matrix.session }}"

.gitignore

@@ -15,6 +15,8 @@ tests/mock/client.properties
 /setup.py.orig
 /.project
 .tox
+.nox
+.mypy_cache
 *__pycache__*
 /.idea/
 /venv*/

docs/contributing.rst

@@ -1,7 +1,8 @@
 Contributing
 ============
 
-dataClay is a `BSC <https://www.bsc.es/research-and-development/software-and-apps/software-list/dataclay>`_
+The dataClay distributed data store is a 
+`BSC <https://www.bsc.es/research-and-development/software-and-apps/software-list/dataclay>`_
 project under the `BSD License <https://github.com/bsc-dom/dataclay/blob/main/LICENSE.txt>`_
 and we happily accept contributions.
 
@@ -16,7 +17,7 @@ If you wish to add a new feature or fix a bug:
 #. Write a test which shows that the bug was fixed or that the feature works
    as expected.
 #. Format your changes with `Black <https://black.readthedocs.io/en/stable/>`_ using the
-   command `tox -e format` and lint your changes using the command `tox -e lint`.
+   command `nox -s format` and lint your changes using the command `nox -s lint`.
 #. Send a pull request and follow up with the maintainer until it gets merged and published.
 
 .. #. Add a `changelog entry
@@ -25,14 +26,14 @@ If you wish to add a new feature or fix a bug:
 Setting up your development environment
 ---------------------------------------
 
-To set up your development environment, you will need `tox`_ installed on your machine:
+To set up your development environment, you will need `nox`_ installed on your machine:
 
 .. code-block:: console
 
-   $ python -m pip install --user --upgrade tox
+   $ python -m pip install --user --upgrade nox
 
 You wll also need to have `docker engine <https://docs.docker.com/engine/install/ubuntu/>`_ installed 
-for `tox`_ to use `pytest-docker <https://pypi.org/project/pytest-docker/>`_.
+for `nox`_ to use `pytest-docker <https://pypi.org/project/pytest-docker/>`_.
 
 Install dataClay in editable mode with the ``dev`` extra requirement:
 
@@ -44,11 +45,11 @@ Running the tests
 -----------------
 
 When running the test suite, we use external dependencies, multiple interpreters, and code coverage analysis. 
-Our `tox.ini <https://github.com/bsc-dom/dataclay/blob/main/tox.ini>`_ file handles much of this for you:
+Our `noxfile.py <https://github.com/bsc-dom/dataclay/blob/main/noxfile.py>`_ file handles much of this for you:
 
 .. code-block:: console
 
-   $ tox
+   $ nox
   
 
-.. _tox: https://tox.wiki/en/stable/
+.. _nox: https://nox.thea.codes/en/stable/

docs/index.rst

@@ -10,8 +10,6 @@ dataClay
    main-concepts
    alien-objects
    advanced-usage
-   telemetry
-   hpc-tracing
    logging
    examples/index
 
@@ -26,12 +24,22 @@ dataClay
    deployment/hpc-manual-deployment
    deployment/compile-redis
 
+.. toctree::
+   :hidden:
+   :caption: Telemetry
+
+   telemetry/configuration
+   telemetry/offline
+   telemetry/real-time
+   telemetry/prometheus
+   telemetry/hpc-tracing
+
 .. toctree::
    :hidden:
    :caption: Release Notes
 
-   releasenotes/3-x
    releasenotes/4-x
+   releasenotes/3-x
 
 .. toctree::
    :hidden:

docs/telemetry/configuration.rst

@@ -0,0 +1,41 @@
+Telemetry Configuration
+=======================
+
+dataClay is instrumented with `OpenTelemetry <https://opentelemetry.io/>`_ to allow observability of
+distributed traces, metrics, and logs. You can configure tracing to export telemetry data either in real-time or for post-mortem analysis. Visualizations can be performed in Grafana.
+
+Configuration
+-------------
+
+To activate tracing in dataClay, the following environment variables need to be set:
+
+- **`DATACLAY_TRACING`**: Set to `true` to enable tracing.
+- **`DATACLAY_TRACING_EXPORTER`**: Export traces to the OpenTelemetry Collector (`otlp`) or print traces to the console (`console`). The default is `otlp`.
+- **`DATACLAY_TRACING_HOST`**: Host of the OpenTelemetry Collector (default: `localhost`).
+- **`DATACLAY_TRACING_PORT`**: Port of the OpenTelemetry Collector (default: `4317`).
+- **`DATACLAY_SERVICE_NAME`**: The service name, which identifies dataClay components in trace data.
+
+Metrics
+-------
+
+.. list-table::
+   :header-rows: 1
+
+   * - Metric
+     - Description
+     - Service
+   * - dataclay_inmemory_objects
+     - Number of objects in memory
+     - backend, client
+   * - dataclay_loaded_objects
+     - Number of loaded objects
+     - backend
+   * - dataclay_stored_objects
+     - Number of stored objects
+     - backend
+   * - dataclay_inmemory_misses_total
+     - Number of inmemory misses
+     - backend, client
+   * - dataclay_inmemory_hits_total
+     - Number of inmemory hits
+     - backend, client

docs/hpc-tracing.rst → docs/telemetry/hpc-tracing.rst

@@ -1,12 +1,9 @@
-===========
 HPC Tracing
 ===========
 
-How to generate paraver traces in MN5
-=====================================
+How to generate paraver traces in MN5 using COMPSs
+--------------------------------------------------
 
-Using COMPSs
-------------
 In order to get the traces we will create a script.
 
 - First we have to import the COMPSs and DataClay modules in order to be able to use them, as well as defining which python version we will be using:
@@ -57,7 +54,8 @@ In order to generate the paraver files, we will call another COMPSs script, "com
 If we run this script in the same directory where we found the traces ($HOME/.COMPSs/[SLURM_JOB_ID]/trace/), the paraver files will appear.
 
 How to inspect the traces in Paraver
-====================================
+------------------------------------
+
 To be able to see these files we will have to open them using the following commands:
 
 .. code-block:: bash

docs/telemetry/offline.rst

@@ -0,0 +1,62 @@
+
+Offline Telemetry Example
+=========================
+
+This example demonstrates exporting OpenTelemetry traces to a JSON file for post-mortem analysis in Grafana.
+
+1. **Activate tracing** by setting environment variables as described in the `telemetry configuration <https://dataclay.bsc.es/docs/telemetry/configuration>`_.
+2. **Generate traces**:
+
+   - Navigate to the `json-exporter` folder in the `offline telemetry example JSON exporter <https://github.com/bsc-dom/dataclay/tree/telemetry-doc/examples/telemetry/offline/json-exporter>`_.
+   - Start dataClay and OpenTelemetry Collector services:
+
+     .. code-block:: bash
+
+        docker compose up
+     
+   - Run the dataClay client:
+
+     .. code-block:: bash
+
+        python3 client.py
+     
+   - Traces are exported to the `traces` folder. You can visualize the JSON traces in Grafana.
+
+3. **Visualize in Grafana**:
+
+   - Navigate to the `json-post-mortem` folder in the `offline telemetry example post-mortem <https://github.com/bsc-dom/dataclay/tree/telemetry-doc/examples/telemetry/offline/json-post-mortem>`_.
+   - Start the OpenTelemetry Collector, Tempo, and Grafana services:
+
+     .. code-block:: bash
+
+        docker compose up
+     
+   - Open Grafana at <http://localhost:3000> (default username/password: `admin`/`admin`).
+   - In the `Explore` section, select `Tempo` as the data source and use the `Trace ID` field to query traces.
+
+4. **Alternative Trace Export**:
+
+   - Run the OpenTelemetry Collector manually:
+
+     .. code-block:: bash
+
+        docker run \
+        -v ./config/otel-collector.yaml:/etc/otel-collector.yaml \
+        otel/opentelemetry-collector-contrib \
+        "--config=/etc/otel-collector.yaml"
+
+5. **Copy Traces from MareNostrum 5**:
+
+   - To analyze traces from MareNostrum 5, copy them locally:
+
+     .. code-block:: bash
+
+        scp transfer1.bsc.es:~/.dataclay/otel-traces.json ./traces/otel-traces.json
+
+6. **Troubleshooting**:
+
+   - If permission issues arise for the `/traces` folder, adjust permissions:
+
+     .. code-block:: bash
+
+        sudo chmod -R 777 traces

docs/telemetry/prometheus.rst

@@ -0,0 +1,78 @@
+Prometheus
+==========
+
+
+Metrics
+-------
+
+.. list-table::
+   :header-rows: 1
+
+   * - Metric
+     - Description
+     - Service
+   * - dataclay_inmemory_objects
+     - Number of objects in memory
+     - backend, client
+   * - dataclay_loaded_objects
+     - Number of loaded objects
+     - backend
+   * - dataclay_stored_objects
+     - Number of stored objects
+     - backend
+   * - dataclay_inmemory_misses_total
+     - Number of inmemory misses
+     - backend, client
+   * - dataclay_inmemory_hits_total
+     - Number of inmemory hits
+     - backend, client
+
+
+Deploy dataClay with Prometheus
+-------------------------------
+
+Run dataClay with Prometheus:
+
+.. note::
+    This example is available in `GitHub <https://github.com/bsc-dom/dataclay/tree/main/examples/telemetry/prometheus>`__.
+
+.. code-block:: bash
+
+    docker compose up -d
+
+The ``metadata-service`` and ``backends`` will post their metrics to the ``8000`` port.
+Prometheus is configured to scrape this port to pull the metrics.
+
+Access Prometheus at `http://localhost:9090 <http://localhost:9090>`_. You can query the metrics defined above.
+
+
+Deploy dataClay with Prometheus Pushgateway
+-------------------------------------------
+
+Run dataClay with Prometheus Pushgateway:
+
+.. note::
+    This example is available in `GitHub <https://github.com/bsc-dom/dataclay/tree/main/examples/telemetry/prometheus-pushgateway>`__.
+
+.. code-block:: bash
+    
+    docker compose up -d
+
+
+The ``metadata-service`` and ``backends`` will push their metrics to the ``pushgateway`` at the ``9091`` port.
+
+The ``client.py`` can also push metrics using the ``pushgateway``:
+
+.. code-block:: bash
+
+    export DATACLAY_METRICS=true
+    export DATACLAY_METRICS_EXPORTER=pushgateway
+    export DATACLAY_METRICS_HOST=localhost  # the default
+    export DATACLAY_METRICS_PORT=9091
+    python3 client.py
+
+
+Access the Pushgateway at `http://localhost:9091 <http://localhost:9091>`_ and Prometheus at `http://localhost:9090 <http://localhost:9090>`_.
+
+.. note::
+    When using ``pushgateway``, a new Python thread will run to push the metrics every 10 seconds (default).