Add link checking workflow and script (and fix some broken links) (#3438)

* Add link checking workflow and script

- Introduced a GitHub Actions workflow (`check-links.yml`) to automate the checking of absolute and relative links in markdown files upon push and pull request events.
- Added a new script (`check_relative_links.py`) to verify that relative links in markdown files point to existing files in the repository, enhancing documentation integrity.
- The script scans markdown files, extracts relative links, and checks their validity without making HTTP requests, providing detailed output for any broken links found.

This addition aims to improve the reliability of documentation links and streamline the review process for changes in markdown files.

* Fixing static pro links

* Fix more pro links

* Update internal links in Kubernetes orchestrator documentation

- Changed links for the remote artifact store and remote container registry sections to point to the correct README files, ensuring accurate navigation within the documentation.

This update aims to enhance the clarity and usability of the Kubernetes orchestrator guide.

* Update documentation for CloudpickleMaterializer warning

- Clarified the warning regarding the use of the built-in CloudpickleMaterializer, emphasising the potential security risks associated with uploading arbitrary objects.
- Improved the link to the custom Materializer documentation for better guidance on creating robust and efficient formats.

This update aims to enhance user awareness of security implications and provide clearer instructions for safer practices in ZenML step output handling.

* Update internal links in API documentation

- Modified links in the OSS and Pro API sections to remove unnecessary README.md suffixes, ensuring cleaner and more direct navigation to the documentation.

This update aims to enhance the clarity and usability of the API documentation.

* ignore mailto links

* Don't check local urls

* Fix SDK docs links

* Fix SDK docs links

* Update CI checker script

* Enhance GitHub Actions workflow for link checking

- Updated the check-links.yml workflow to ignore changes in the 'src' and 'tests' directories, as well as Python files, during push and pull request events on the main branch. This aims to streamline the link checking process by focusing on relevant changes only.

This update improves the efficiency of the CI workflow by reducing unnecessary checks.

* Handle backslashes in docs better

* Fix links broken in projects renaming

* Fix broken links

* Link checker in CI should actually fail now

* Update documentation links for clarity and accuracy

- Changed the reference from the "starter guide" to the "production guide" in the migration guide.
- Updated the service connector link to point to the correct section on infrastructure deployment in the remote storage documentation.
- Minor formatting adjustment in the workspaces documentation for consistency.

These changes ensure that users have access to the most relevant and accurate resources.

* Refine paths-ignore syntax in CI workflow

- Updated the paths-ignore entries in the check-links.yml workflow to use double quotes for consistency.
- Ensured that the Python version is specified with double quotes as well.

These changes enhance the readability and maintainability of the workflow configuration.

Author: strickvl

.github/workflows/check-links.yml

@@ -0,0 +1,89 @@
+---
+name: Check Links
+on:
+  workflow_dispatch:
+  push:
+    branches: [main]
+    paths-ignore: [src/**, tests/**, "**.py"]
+  pull_request:
+    types: [opened, synchronize, reopened]
+    paths-ignore: [src/**, tests/**, "**.py"]
+concurrency:
+  # New commit on branch cancels running workflows of the same branch
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+jobs:
+  check-absolute-links:
+    if: github.event.pull_request.draft == false
+    runs-on: ubuntu-latest
+    outputs:
+      exit_code: ${{ steps.check-absolute.outputs.exit_code }}
+    steps:
+      - name: Checkout Repository
+        uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.11"
+      - name: Install dependencies
+        run: pip install requests
+      - name: Check Absolute Links
+        id: check-absolute
+        run: |
+          # Only check absolute links (http/https), not relative links
+          python docs/link_checker.py --dir docs/book --substring "http" --validate-links --timeout 15 --ci-mode
+          echo "exit_code=$?" >> $GITHUB_OUTPUT
+        # Continue on error so both checks can run to completion
+        continue-on-error: true
+  check-relative-links:
+    if: github.event.pull_request.draft == false
+    runs-on: ubuntu-latest
+    outputs:
+      exit_code: ${{ steps.check-relative.outputs.exit_code }}
+    steps:
+      - name: Checkout Repository
+        uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.11"
+      - name: Check Relative Links
+        id: check-relative
+        run: |
+          # Check if relative links resolve within the repository
+          python scripts/check_relative_links.py --dir docs/book
+          echo "exit_code=$?" >> $GITHUB_OUTPUT
+        # Continue on error so both checks can run to completion
+        continue-on-error: true
+  summary:
+    needs: [check-absolute-links, check-relative-links]
+    if: always() && github.event.pull_request.draft == false
+    runs-on: ubuntu-latest
+    steps:
+      - name: Create Summary
+        run: |-
+          echo "# Documentation Link Check Results" >> $GITHUB_STEP_SUMMARY
+
+          # Check for failures in absolute links job
+          if [[ "${{ needs.check-absolute-links.outputs.exit_code }}" != "0" ]]; then
+            echo "❌ **Absolute links check failed**" >> $GITHUB_STEP_SUMMARY
+            echo "There are broken absolute links in the documentation. Please check the job logs for details." >> $GITHUB_STEP_SUMMARY
+            FAILED=true
+          else
+            echo "✅ **Absolute links check passed**" >> $GITHUB_STEP_SUMMARY
+          fi
+
+          # Check for failures in relative links job
+          if [[ "${{ needs.check-relative-links.outputs.exit_code }}" != "0" ]]; then
+            echo "❌ **Relative links check failed**" >> $GITHUB_STEP_SUMMARY
+            echo "There are broken relative links in the documentation. Please check the job logs for details." >> $GITHUB_STEP_SUMMARY
+            FAILED=true
+          else
+            echo "✅ **Relative links check passed**" >> $GITHUB_STEP_SUMMARY
+          fi
+
+          # Exit with failure if any checks failed
+          if [[ "$FAILED" == "true" ]]; then
+            echo "::error::One or more link checks failed. Please fix the broken links."
+            exit 1
+          fi

README.md

@@ -266,7 +266,7 @@ And finally, here are some other examples and use cases for inspiration:
 
 1. [E2E Batch Inference](examples/e2e/): Feature engineering, training, and inference pipelines for tabular machine learning.
 2. [Basic NLP with BERT](examples/e2e_nlp/): Feature engineering, training, and inference focused on NLP.
-3. [LLM RAG Pipeline with Langchain and OpenAI](https://github.com/zenml-io/zenml-projects/tree/main/llm-agents): Using Langchain to create a simple RAG pipeline.
+3. [LLM RAG Pipeline with Langchain and OpenAI](https://github.com/zenml-io/zenml-projects/tree/main/zenml-support-agent): Using Langchain to create a simple RAG pipeline.
 4. [Huggingface Model to Sagemaker Endpoint](https://github.com/zenml-io/zenml-projects/tree/main/huggingface-sagemaker): Automated MLOps on Amazon Sagemaker and HuggingFace
 5. [LLMops](https://github.com/zenml-io/zenml-projects/tree/main/llm-complete-guide): Complete guide to do LLM with ZenML
 
@@ -341,8 +341,8 @@ our GitHub repo.
 ## 📚 LLM-focused Learning Resources
 
 1. [LL Complete Guide - Full RAG Pipeline](https://github.com/zenml-io/zenml-projects/tree/main/llm-complete-guide) - Document ingestion, embedding management, and query serving
-2. [LLM Fine-Tuning Pipeline](https://github.com/zenml-io/zenml-projects/tree/main/llm-finetuning) - From data prep to deployed model
-3. [LLM Agents Example](https://github.com/zenml-io/zenml-projects/tree/main/llm-agents) - Track conversation quality and tool usage
+2. [LLM Fine-Tuning Pipeline](https://github.com/zenml-io/zenml-projects/tree/main/zencoder) - From data prep to deployed model
+3. [LLM Agents Example](https://github.com/zenml-io/zenml-projects/tree/main/zenml-support-agent) - Track conversation quality and tool usage
 
 ## 🤖 AI-Friendly Documentation with llms.txt
 

docs/book/api-docs/toc.md

@@ -4,7 +4,7 @@
 
 ## OSS API
 
-* [OSS API](oss-api-docs/README.md)
+* [OSS API](oss-api-docs)
     * [Artifacts](oss-api-docs/v1/artifacts.md)
     * [Artifact versions](oss-api-docs/v1/artifact-versions/README.md)
         * [Batch](oss-api-docs/v1/artifact-versions/batch.md)
@@ -55,7 +55,7 @@
 
 ## Pro API
 
-* [Pro API](pro-api-docs/README.md)
+* [Pro API](pro-api-docs)
   * [API reference](pro-api-docs/api-reference/README.md)
     * [Tenants](pro-api-docs/api-reference/tenants/README.md)
       * [Deploy](pro-api-docs/api-reference/tenants/deploy.md)
@@ -105,4 +105,4 @@
       * [Resource members](pro-api-docs/api-reference/rbac/resource-members.md)
     * [Server](pro-api-docs/api-reference/server/README.md)
       * [Info](pro-api-docs/api-reference/server/info.md)
-  * [Pro API Specification](https://cloudapi.zenml.io/openapi.json) 
+  * [Pro API Specification](https://cloudapi.zenml.io/openapi.json) 

docs/book/component-guide/alerters/discord.md

@@ -9,10 +9,10 @@ directly from within your ZenML pipelines.
 
 The `discord` integration contains the following two standard steps:
 
-* [discord\_alerter\_post\_step](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-discord/#zenml.integrations.discord.steps.discord_alerter_post_step.discord_alerter_post_step)\
+* [discord\_alerter\_post\_step](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-discord.html#zenml.integrations.discord)\
   takes a string message, posts it to a Discord channel, and returns whether the\
   operation was successful.
-* [discord\_alerter\_ask\_step](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-discord/#zenml.integrations.discord.steps.discord_alerter_ask_step.discord_alerter_ask_step)\
+* [discord\_alerter\_ask\_step](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-discord.html#zenml.integrations.discord)\
   also posts a message to a Discord channel, but waits for user feedback, and\
   only returns `True` if a user explicitly approved the operation from within\
   Discord (e.g., by sending "approve" / "reject" to the bot in response).
@@ -98,8 +98,8 @@ permissions:
 ### How to Use the Discord Alerter
 
 After you have a `DiscordAlerter` configured in your stack, you can directly import\
-the [discord\_alerter\_post\_step](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-discord/#zenml.integrations.discord.steps.discord_alerter_post_step.discord_alerter_post_step)\
-and [discord\_alerter\_ask\_step](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-discord/#zenml.integrations.discord.steps.discord_alerter_ask_step.discord_alerter_ask_step)\
+the [discord\_alerter\_post\_step](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-discord.html#zenml.integrations.discord)\
+and [discord\_alerter\_ask\_step](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-discord.html#zenml.integrations.discord)\
 steps and use them in your pipelines.
 
 Since these steps expect a string message as input (which needs to be the output of another step), you typically also\
@@ -131,7 +131,7 @@ if __name__ == "__main__":
 ```
 
 For more information and a full list of configurable attributes of the Discord alerter, check out\
-the [SDK Docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-discord/#zenml.integrations.discord.alerters.discord_alerter.DiscordAlerter)\
+the [SDK Docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-discord.html#zenml.integrations.discord)\
 .
 
 <figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>

docs/book/component-guide/alerters/slack.md

@@ -231,6 +231,6 @@ if __name__ == "__main__":
 ```
 
 For more information and a full list of configurable attributes of the Slack 
-alerter, check out the [SDK Docs](https://sdkdocs.zenml.io/latest/integration\_code\_docs/integrations-slack/#zenml.integrations.slack.alerters.slack\_alerter.SlackAlerter) .
+alerter, check out the [SDK Docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-slack.html#zenml.integrations.slack) .
 
 <figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>

docs/book/component-guide/artifact-stores/README.md

@@ -70,8 +70,8 @@ All ZenML Artifact Stores implement [the same IO API](custom.md) that resembles
 
 Accessing the low-level Artifact Store API can be done through the following Python modules:
 
-* `zenml.io.fileio` provides low-level utilities for manipulating Artifact Store objects (e.g. `open`, `copy`, `rename` , `remove`, `mkdir`). These functions work seamlessly across Artifact Stores types. They have the same signature as the [Artifact Store abstraction methods](https://sdkdocs.zenml.io/latest/core_code_docs/core-artifact_stores/#zenml.artifact_stores.base_artifact_store.BaseArtifactStore) ( in fact, they are one and the same under the hood).
-* [zenml.utils.io\_utils](https://sdkdocs.zenml.io/latest/core_code_docs/core-utils/#zenml.utils.io_utils) includes some higher-level helper utilities that make it easier to find and transfer objects between the Artifact Store and the local filesystem or memory.
+* `zenml.io.fileio` provides low-level utilities for manipulating Artifact Store objects (e.g. `open`, `copy`, `rename` , `remove`, `mkdir`). These functions work seamlessly across Artifact Stores types. They have the same signature as the [Artifact Store abstraction methods](https://sdkdocs.zenml.io/latest/core_code_docs/core-artifact_stores.html#zenml.artifact_stores.base_artifact_store) ( in fact, they are one and the same under the hood).
+* [zenml.utils.io\_utils](https://sdkdocs.zenml.io/latest/core_code_docs/core-utils.html#zenml.utils.io_utils) includes some higher-level helper utilities that make it easier to find and transfer objects between the Artifact Store and the local filesystem or memory.
 
 {% hint style="info" %}
 When calling the Artifact Store API, you should always use URIs that are relative to the Artifact Store root path, otherwise, you risk using an unsupported protocol or storing objects outside the store. You can use the `Repository` singleton to retrieve the root path of the active Artifact Store and then use it as a base path for artifact URIs, e.g.:

docs/book/component-guide/artifact-stores/azure.md

@@ -218,7 +218,7 @@ zenml stack register custom_stack -a az_store ... --set
 {% endtab %}
 {% endtabs %}
 
-For more, up-to-date information on the Azure Artifact Store implementation and its configuration, you can have a look at [the SDK docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-azure/#zenml.integrations.azure.artifact_stores) .
+For more, up-to-date information on the Azure Artifact Store implementation and its configuration, you can have a look at [the SDK docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-azure.html#zenml.integrations.azure) .
 
 ### How do you use it?
 

docs/book/component-guide/artifact-stores/custom.md

@@ -12,7 +12,7 @@ ZenML comes equipped with [Artifact Store implementations](./#artifact-store-fla
 
 ### Base Abstraction
 
-The Artifact Store establishes one of the main components in every ZenML stack. Now, let us take a deeper dive into the fundamentals behind its abstraction, namely [the `BaseArtifactStore` class](https://sdkdocs.zenml.io/latest/core_code_docs/core-artifact_stores/#zenml.artifact_stores.base_artifact_store.BaseArtifactStore):
+The Artifact Store establishes one of the main components in every ZenML stack. Now, let us take a deeper dive into the fundamentals behind its abstraction, namely [the `BaseArtifactStore` class](https://sdkdocs.zenml.io/latest/core_code_docs/core-artifact_stores.html#zenml.artifact_stores.base_artifact_store):
 
 1. As ZenML only supports filesystem-based artifact stores, it features a configuration parameter called `path`, which will indicate the root path of the artifact store. When registering an artifact store, users will have to define this parameter.
 2. Moreover, there is another variable in the config class called `SUPPORTED_SCHEMES`. This is a class variable that needs to be defined in every subclass of the base artifact store configuration. It indicates the supported file path schemes for the corresponding implementation. For instance, for the Azure artifact store, this set will be defined as `{"abfs://", "az://"}`.
@@ -126,7 +126,7 @@ class BaseArtifactStoreFlavor(Flavor):
 ```
 
 {% hint style="info" %}
-This is a slimmed-down version of the base implementation which aims to highlight the abstraction layer. In order to see the full implementation and get the complete docstrings, please check the [SDK docs](https://sdkdocs.zenml.io/latest/core_code_docs/core-artifact_stores/#zenml.artifact_stores.base_artifact_store.BaseArtifactStore) .
+This is a slimmed-down version of the base implementation which aims to highlight the abstraction layer. In order to see the full implementation and get the complete docstrings, please check the [SDK docs](https://sdkdocs.zenml.io/latest/core_code_docs/core-artifact_stores.html#zenml.artifact_stores.base_artifact_store) .
 {% endhint %}
 
 **The effect on the `zenml.io.fileio`**
@@ -139,7 +139,7 @@ This means that when you utilize a method such as `fileio.open(...)` with a file
 
 If you want to implement your own custom Artifact Store, you can follow the following steps:
 
-1. Create a class that inherits from [the `BaseArtifactStore` class](https://sdkdocs.zenml.io/latest/core_code_docs/core-artifact_stores/#zenml.artifact_stores.base_artifact_store.BaseArtifactStore) and implements the abstract methods.
+1. Create a class that inherits from [the `BaseArtifactStore` class](https://sdkdocs.zenml.io/latest/core_code_docs/core-artifact_stores.html#zenml.artifact_stores.base_artifact_store) and implements the abstract methods.
 2. Create a class that inherits from [the `BaseArtifactStoreConfig` class](custom.md) and fill in the `SUPPORTED_SCHEMES` based on your file system.
 3. Bring both of these classes together by inheriting from [the `BaseArtifactStoreFlavor` class](custom.md).
 

docs/book/component-guide/artifact-stores/gcp.md

@@ -190,7 +190,7 @@ zenml stack register custom_stack -a gs_store ... --set
 {% endtab %}
 {% endtabs %}
 
-For more, up-to-date information on the GCS Artifact Store implementation and its configuration, you can have a look at [the SDK docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-gcp/#zenml.integrations.gcp.artifact_stores.gcp_artifact_store) .
+For more, up-to-date information on the GCS Artifact Store implementation and its configuration, you can have a look at [the SDK docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-gcp.html#zenml.integrations.gcp) .
 
 ### How do you use it?
 

docs/book/component-guide/artifact-stores/local.md

@@ -75,7 +75,7 @@ zenml stack register custom_stack -o default -a custom_local --set
 Same as all other Artifact Store flavors, the local Artifact Store does take in a `path` configuration parameter that can be set during registration to point to a custom path on your machine. However, it is highly recommended that you rely on the default `path` value, otherwise, it may lead to unexpected results. Other local stack components depend on the convention used for the default path to be able to access the local Artifact Store.
 {% endhint %}
 
-For more, up-to-date information on the local Artifact Store implementation and its configuration, you can have a look at [the SDK docs](https://sdkdocs.zenml.io/latest/core_code_docs/core-artifact_stores/#zenml.artifact_stores.local_artifact_store) .
+For more, up-to-date information on the local Artifact Store implementation and its configuration, you can have a look at [the SDK docs](https://sdkdocs.zenml.io/latest/core_code_docs/core-artifact_stores.html#zenml.artifact_stores.local_artifact_store) .
 
 ### How do you use it?