Add llms.txt to Jenkins to disallow LLMs as we do it with robots.txt

[StefanSpieker]

Add llms.txt to Jenkins to disallow LLMs as we do it with robots.txt
Some Background can be found at llmstxt.org

Testing done

tried curl:

Proposed changelog entries

• also deliver a llms.txt file for LLMs

Proposed changelog category

/label rfe

Proposed upgrade guidelines

N/A

Submitter checklist

• The Jira issue, if it exists, is well-described.
• The changelog entries and upgrade guidelines are appropriate for the audience affected by the change (users or developers, depending on the change) and are in the imperative mood (see examples). Fill in the Proposed upgrade guidelines section only if there are breaking changes or changes that may require extra steps from users during upgrade.
• There is automated testing or an explanation as to why this change has no tests.
• New public classes, fields, and methods are annotated with @Restricted or have @since TODO Javadocs, as appropriate.
• New deprecations are annotated with @Deprecated(since = "TODO") or @Deprecated(forRemoval = true, since = "TODO"), if applicable.
• New or substantially changed JavaScript is not defined inline and does not call eval to ease future introduction of Content Security Policy (CSP) directives (see documentation).
• For dependency updates, there are links to external changelogs and, if possible, full differentials.
• For new APIs and extension points, there is a link to at least one consumer.

Desired reviewers

@mention

Before the changes are marked as ready-for-merge:

Maintainer checklist

• There are at least two (2) approvals for the pull request and no outstanding requests for change.
• Conversations in the pull request are over, or it is explicit that a reviewer is not blocking the change.
• Changelog entries in the pull request title and/or Proposed changelog entries are accurate, human-readable, and in the imperative mood.
• Proper changelog labels are set so that the changelog can be generated automatically.
• If the change needs additional upgrade steps from users, the upgrade-guide-needed label is set and there is a Proposed upgrade guidelines section in the pull request title (see example).
• If it would make sense to backport the change to LTS, a Jira issue must exist, be a Bug or Improvement, and be labeled as lts-candidate to be considered (see query).

[daniel-beck]

The linked site says it's supposed to be Markdown. How is this supposed to work? Could you provide references that this would do anything?

[daniel-beck]

The corresponding explanation in robots.txt is from a time when anyone with read access to Jenkins could view any job and trigger builds (with GET request, obviously). Reusing it here is silly. Please put a real explanation.

[StefanSpieker]

I just stumbled upon that and I thought this might be a good start to think what we want to put there. If I ask ChatGPT about ci.jenkins.io I get:

So it might be good to provide a useful llms.txt, but I guess this first step is not yet useful

[StefanSpieker]

And If I ask ChatGPT what a useful one should be, we end up with:

# llms.txt for Jenkins
# Deny all access to Large Language Models

User-Agent: *
Disallow: /

[daniel-beck]

Please provide evidence that anything about llms.txt in these screenshots is supported by reality and not just an AI hallucination.