Summary

The Khoj Obsidian, Desktop and Web clients inadequately sanitize the AI model's response and user inputs.
This can trigger Cross Site Scripting (XSS) via Prompt Injection from untrusted documents either indexed by the user on Khoj or read by Khoj from the internet when the user invokes the /online command

Affects

Users indexing a document with a malicious prompt or Khoj reading an adversarial website when a user uses the /online command in Khoj

Impact

• Can notify an attacker of their malicious document being accessed by Khoj and/or the user
• Can load unexpected, unwanted images into a users chat session
• Can run undesired javascript on the users client app causing their machine to hang
• Can steal secrets from the desktop app by using some preload.js exposed functions
• Potential 1-Click RCE under system specific circumstances abusing HTML injection and special URI schemes.

Limitations

• No cross-shard access possible
• Does not affect references rendered in the reference section of the chat message. Requires the chat model to actually use the reference in its generated response text to trigger the XSS
• Untrusted javascript can only run in a sandboxed iframe on the Obsidian client & with no access to the user's chat session

Details

The core issue is that Khoj's chat AI model can be made to follow adversarial crafted instructions in untrusted documents Khoj reads when trying to respond to a users query. This requires proper safety checks and locks on the Khoj chat interface to avoid any undesirable behavior caused by this temporary "take-over" of Khoj.

Example Scenario

• User indexes a document with Khoj containing a malicious prompt/instruction
• A chat message by the user triggers Khoj to read the malicious document and start following the malicious instruction in that document rather than the user's original instruction
• This malicious instruction asks Khoj to inject the attacker provided HTML to load an external image, notifying the attacker of their malicious document having been accessed

Proof of Concept

To trigger it via user input:
Input: <img src=x onerror=alert(1)//> in your message to Khoj via the Obsidian chat modal -> results in triggering js in the chat UI.

To trigger it via AI response:
Input the above payload (the img one), and ask the AI to repeat what you just said

Fix

• Sanitized rendered chat messages using DOMPurify
• Added Content Security Policy (CSP) domain and content restrictions in the Obsidian, Desktop and Web chat clients

Future Improvements

• Create and use finetuned chat LLM models to cleanly separate Data from Instructions. This should address prompt injection root-cause better