Web Search: Playwright, spatial parsing, markdown

[Saghen]

Context

The existing web search implementation naively scrapes and chunks content for passing to the LLM. A brief explanation of how the relevant components work:

• Scraping: Fetch the page's HTML, statically parse it with selectors
  • document.querySelectorAll('p, table, pre, ul, ol')
• Chunking: Concatenate the text of all elements with spaces, chunk the resulting text so that the length of each chunk is less than the maximum embedding length
• Embedding: Get sentence similarity for each chunk, pass top 8 chunks to the LLM

Solution

• Scraping: Load the page into Playwright and perform spatial parsing
  • The spatial parser uses a clustering technique based on the position to find the primary content. So i.e., you might end up with a cluster for the header, footer, primary content and sidebar. Heuristics, such as text density, find the critical cluster which should contain the primary content
  • Metadata scraping: title, description, site name, author, updated at, created at
• Conversion to markdown/chunking:
  • Convert the resulting list of HTML elements into a tree like h1 [h2 [p p blockquote] h2 [h3 [...] ] ]
  • Convert the HTML elements into their markdown equivalents
• Chunking: Treat each markdown element as a chunk. Split elements where element.text.length > embeddingMaxLength based on sentence boundaries
• Embedding:
  • Get sentence similiarty for each markdown element
  • Get top chunks and their parent heading (based on the tree from conversion to markdown) until embedding distance increases beyond a threshold, or a character limit is hit

Dynamically includes anywhere from 3000 chars -> 8000 chars based on embedding distance. May result in longer search queries when using local CPU embedding

Spatial parsing implementation written by @Aaditya-Sahay

[mishig25]

for the ones who are testing, make sure to run

npm ci
npx playwright install

to get the necessary deps before running

[Aaditya-Sahay]

@Saghen Since we are using playwright as a library, we should add @playwright/browser-chromium to our list of dependencies so it automatically installs for people . See here

[Saghen]

For nix users, pin the playwright version to the latest version in nixpkgs (currently 1.40.0) via npm i [email protected]. Then launch a nix shell with the following config:

{ pkgs ? import <nixpkgs> { } }:
pkgs.mkShell {
  nativeBuildInputs = with pkgs; [ playwright-driver.browsers ];

  shellHook = ''
    export PLAYWRIGHT_BROWSERS_PATH=${pkgs.playwright-driver.browsers}
    export PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=true
  '';
}

[gary149]

Let's go with JS disabled by default :)

[mishig25]

lgtm! Great work 🔥