feat(fetcher): custom request headers for LinkContentFetcher

[xoaryaa]

Related Issues

• fixes Make REQUEST_HEADERS in LinkContentFetcher customizable #9064

Proposed Changes:

Feature

• Add optional request_headers: dict[str, str] = {} to LinkContentFetcher.
• Per-request header precedence:
  1. httpx client defaults
  2. component defaults (REQUEST_HEADERS)
  3. request_headers (new)
  4. rotating User-Agent (rotation wins; keeps UA rotation behavior intact)
• Apply the same merge logic in both sync and async code paths.

Resilience

• If http2=True but h2 is not installed, log a warning and gracefully fall back to HTTP/1.1 (avoid raising ImportError).

Docs

• Update class docstring with header precedence and a short usage example.

How did you test it?

• Unit tests (in test/components/fetchers/test_link_content_fetcher.py):
  • TestLinkContentFetcher_CustomHeaders::test_request_headers_sync_merging_and_ua_override
  • TestLinkContentFetcherAsync_CustomHeaders::test_request_headers_async_merging_and_ua_override
  • Confirmed that:
    • custom headers are sent,
    • client-level headers merge correctly,
    • rotating User-Agent overrides any UA supplied via request_headers.
• Ran existing tests locally (-m "not integration"): all green.
• Ran pre-commit hooks locally and fixed reported issues.

Notes for the reviewer

• The change is additive and backward compatible; default behavior is unchanged unless request_headers is provided.
• Happy to split the drive-by typo fixes (codespell) into a separate PR if you prefer—kept here to keep CI green.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: feat:
• I documented my code
• I ran pre-commit hooks and fixed any issue

[coveralls]

Pull Request Test Coverage Report for Build 17424233833

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 23 unchanged lines in 1 file lost coverage.
• Overall coverage increased (+0.02%) to 92.093%

---

Files with Coverage Reduction	New Missed Lines	%
components/fetchers/link_content.py	23	86.44%

Totals
Change from base Build 17402313102:	0.02%
Covered Lines:	12939
Relevant Lines:	14050

---

💛 - Coveralls

[sjrl]

hey @xoaryaa thanks for opening a new PR! Could you remove all changes made to non-code files?

[sjrl]

A few other notes:

• It would be great if you could add a unit test where we specify custom headers
• Add a release note for your PR. Instructions on how to do so can be found here https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#release-notes

[xoaryaa]

Okay sure. I’ll do that.

…

On Wed, Sep 3, 2025 at 17:33 Sebastian Husch Lee ***@***.***> wrote: *sjrl* left a comment (deepset-ai/haystack#9756) <#9756 (comment)> A few other notes: - It would be great if you could add a unit test where we specify custom headers - Add a release note for your PR. Instructions on how to do so can be found here https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#release-notes — Reply to this email directly, view it on GitHub <#9756 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3HSYNEASDU2BVK7ME6OECD3Q3KIXAVCNFSM6AAAAACFNNQTQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTENBYHE2TSOJWGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[xoaryaa]

Opened a new clean pr

[sjrl]

Closing and moving discussion to the new PR #9760