prompt adjustments

app/Domains/Prompts/SearchPrompt.php

@@ -0,0 +1,36 @@
+<?php
+
+namespace App\Domains\Prompts;
+
+use Illuminate\Support\Facades\Log;
+
+class SearchPrompt
+{
+    public static function prompt(string $originalPrompt): string
+    {
+
+
+        Log::info("[LaraChain] - SearchPrompt");
+        return <<<PROMPT
+The user is asking to search the web but I want you to review the query and clean it keeping it as
+a string.
+
+Here are some examples of how I want you to return the data:
+### RETURN FORMAT
+in: Search the web for PHP news and Laravle news
+out: php news OR laravel news -filetype:pdf -intitle:pdf
+
+in: current data on the llm industry
+out: llm industry news OR llm industry updates -filetype:pdf -intitle:pdf
+
+in: latest news on the laravel framework
+out: laravel framework news OR laravel framework updates -filetype:pdf -intitle:pdf
+### END RETURN FORMAT
+
+
+### START USER QUERY
+$originalPrompt
+### END USER QUERY
+PROMPT;
+    }
+}

app/Domains/Prompts/SummarizeDocumentPrompt.php

@@ -2,34 +2,25 @@
 
 namespace App\Domains\Prompts;
 
+use Illuminate\Support\Facades\Log;
+
 class SummarizeDocumentPrompt
 {
     public static function prompt(string $documentContent): string
     {
+        Log::info("[LaraChain] - SummarizeDocumentPrompt");
+
         return <<<PROMPT
-  # **Role, Task, Format (R.T.F)**
-  **Role**: You are a summarization agent tasked with distilling content from a larger document into a brief, clear summary.
-  **Task**: Generate a concise 4-10 line summary of the provided content, focusing on capturing the essence of the text.
-  **Format**: Provide the summary in plain text, avoiding any additional text or formatting beyond the essential summary.
-  
-  # **Context, Action, Result, Example (C.A.R.E)**
-  **Context**: The provided content is part of a larger document, and the user requires a summary to understand the main points or themes quickly.
-  **Action**: Read the content carefully and distill it into a 4-10 line summary that encapsulates the key information or message.
-  **Result**: A succinct summary that allows the user to grasp the core content at a glance, useful for viewing in a summary view alongside other related pages.
-  **Example**: If the document discusses the impact of climate change on marine life, the summary might read: "Examines the effects of warming oceans on marine biodiversity, highlighting the risk to coral reefs."
-  
-  # **Before, After, Bridge (B.A.B)**
-  **Before**: The user has a segment of a larger document and needs a quick understanding of its content.
-  **After**: The user has a brief summary that effectively conveys the main points of the content, suitable for integration with summaries of other related pages.
-  **Bridge**: By condensing the provided text into a concise summary, you bridge the gap between detailed content and a digestible overview.
-  
-  # **Task, Action, Goal (T.A.G)**
-  **Task**: Summarize the given content into 4-10 lines.
-  **Action**: Identify the central ideas or messages in the text and express them concisely.
-  **Goal**: Deliver a clear and brief summary that captures the essence of the content, aiding in quick comprehension and comparison with related document sections.
-  
-  ---
-  
+# **Task, Action, Goal (T.A.G)**
+  **Task**: Synthesize and summarize the retrieved data into a coherent output.
+
+  **Action**:
+
+  - Analyze the data retrieved by the RAG system to identify key themes and information.
+  - Integrate these insights into a concise and informative summary or response.
+
+  **Goal**: Produce a distilled and accurate synthesis of the retrieved information, aiding in quick comprehension and effective decision-making based on comprehensive data input.
+
   **Content to Summarize**:
 $documentContent
 

app/Domains/Prompts/SummarizePrompt.php

@@ -2,34 +2,19 @@
 
 namespace App\Domains\Prompts;
 
+use Illuminate\Support\Facades\Log;
+
 class SummarizePrompt
 {
     public static function prompt(string $originalPrompt, string $context): string
     {
+        Log::info("[LaraChain] - SummarizePrompt");
+
         return <<<PROMPT
   # **Role, Task, Format (R.T.F)**
-  **Role**: You are the core Agent of the Retrieval Augmented Generation system (RAG). Your primary role is to respond to user queries accurately by interpreting and synthesizing relevant information from provided documents.
-  **Task**: Prioritize the user’s query to guide your response, using the context from the documents to support and inform your answer.
-  **Format**: Provide responses in Markdown format that directly address the user’s question, supplemented by relevant information extracted from the context.
-  
-  # **Context, Action, Result, Example (C.A.R.E)**
-  **Context**: The text provided is a scientific article discussing Lyme borreliosis in Europe.
-  **Action**: Identify the user’s specific query and use key points from the article to construct a response that directly addresses this query, providing additional insights where relevant.
-  **Result**: A tailored response that directly answers the user's question, supported by accurate and pertinent information from the context.
-  **Example**: If the user asks about the effectiveness of the vaccine, focus your response on vaccine outcomes and supporting data mentioned in the article.
-  
-  # **Before, After, Bridge (B.A.B)**
-  **Before**: The user has a question that may require background information or specific details from a larger document.
-  **After**: The user receives a concise, informative answer that directly addresses their question, using the context to enhance the response.
-  **Bridge**: By analyzing the user's query and linking it with relevant information from the document, you bridge the gap between the user's need for specific information and the comprehensive details available in the context.
-  
-  # **Task, Action, Goal (T.A.G)**
-  **Task**: Directly respond to the user’s query.
-  **Action**: Use the document's context to inform and support your response, ensuring it is relevant and comprehensive.
-  **Goal**: Deliver an answer that satisfies the user's inquiry and provides them with a deeper understanding of the topic based on the provided document.
-  
-  ---
-
+  **Role**: As the core Agent of the Retrieval Augmented Generation system (RAG), your primary role is to respond accurately to user queries by interpreting and synthesizing relevant information from provided documents.
+  **Task**: Prioritize and respond to the user’s query using the context from the documents to support and inform your answer. The response should be direct and precise, addressing the specific aspects of the query based on the document’s content.
+  **Format**: Deliver the response in a concise, clear Markdown format that directly addresses the user’s questions, supplemented by pertinent information extracted from the context.
 
 **The User's Query**:
 $originalPrompt

app/Domains/Prompts/VerificationPrompt.php

@@ -2,41 +2,28 @@
 
 namespace App\Domains\Prompts;
 
+use Illuminate\Support\Facades\Log;
+
 class VerificationPrompt
 {
     public static function prompt(string $llmResponse, string $context): string
     {
+        Log::info("[LaraChain] - VerificationPrompt");
+
         return <<<PROMPT
   # **Role, Task, Format (R.T.F)**
   **Role**: You are a Verification Agent tasked with ensuring the accuracy and relevance of responses given to user queries.
-  **Task**: Review the initial response to ensure it directly addresses the user's query and correctly uses the document's context to support the answer.
-  **Format**: Provide feedback in a concise format, noting any discrepancies or areas for enhancement.
-  
-  # **Context, Action, Result, Example (C.A.R.E)**
-  **Context**: The initial response was crafted to answer a specific query using the context from a scientific article.
-  **Action**: Verify that the response adequately addresses the user's question and that the information from the context is accurately and effectively integrated.
-  **Result**: A confirmation that the response is accurate and fulfills the user's informational needs, or a correction if discrepancies are found.
-  **Example**: Ensure that the facts used to support the response are correctly interpreted from the article and that the user’s query is the central focus.
-  
-  # **Before, After, Bridge (B.A.B)**
-  **Before**: There may be concerns about the accuracy or relevance of the initial response.
-  **After**: The user receives a verified answer that is both accurate and highly relevant to their query.
-  **Bridge**: By critically reviewing the initial response and making necessary corrections, you ensure the integrity and usefulness of the information provided to the user.
-  
-  # **Task, Action, Goal (T.A.G)**
-  **Task**: Confirm the relevance and accuracy of the initial response.
-  **Action**: Scrutinize the response against the user's query and the document's context.
-  **Goal**: Provide assurance or necessary corrections to ensure the response adequately addresses the user's query with accurate support from the context.
-  
-  ---
-  
+  **Task**: Review the initial response to verify it directly addresses the user's query using the document's context accurately and relevantly.
+  **Format**: Provide feedback in a concise, clear format, noting any discrepancies, inaccuracies, or areas for enhancement.
+
+
   **Initial Response for Verification**:
   $llmResponse
 
-  
+
   **Context of data used in above response**:
   $context
-          
+
   PROMPT;
     }
 }

app/Domains/Sources/WebSearchSource.php

@@ -2,6 +2,8 @@
 
 namespace App\Domains\Sources;
 
+use App\Domains\Collections\CollectionStatusEnum;
+use App\Domains\Prompts\SearchPrompt;
 use App\Domains\Sources\WebSearch\Response\SearchResponseDto;
 use App\Domains\Sources\WebSearch\WebSearchFacade;
 use App\Jobs\GetWebContentJob;
@@ -22,29 +24,7 @@ public function handle(Source $source): void
             $limit = data_get($meta_data, 'limit', 5);
             $driver = data_get($meta_data, 'driver', 'mock');
 
-            $prompt = <<<PROMPT
-            The user is asking to search the web but I want you to review the query and clean it keeping it as 
-            a string.
-
-            Here are some examples of how I want you to return the data:
-            ### RETURN FORMAT
-            in: Search the web for PHP news and Laravle news
-            out: php news OR laravel news -filetype:pdf -intitle:pdf
-
-            in: current data on the llm industry
-            out: llm industry news OR llm industry updates -filetype:pdf -intitle:pdf
-
-            in: latest news on the laravel framework
-            out: laravel framework news OR laravel framework updates -filetype:pdf -intitle:pdf
-            ### END RETURN FORMAT
-
-            
-            ### START USER QUERY
-            $search
-            ### END USER QUERY
-
-         
-PROMPT;
+            $prompt = SearchPrompt::prompt($search);
 
             Log::info('[LaraChain] Asking LLM to optimize search query');
 
@@ -86,6 +66,12 @@ public function handle(Source $source): void
             $source->last_run = now();
             $source->save();
 
+            notify_collection_ui(
+                collection: $source->collection,
+                status: CollectionStatusEnum::PENDING,
+                message: "Search complete getting results from each page"
+            );
+
         } catch (\Exception $e) {
             Log::error('[LaraChain] - Error running WebSearchSource', [
                 'error' => $e->getMessage(),

app/Jobs/GetWebContentJob.php

@@ -73,7 +73,15 @@ public function handle(): void
                 'file_path' => md5($this->webResponseDto->url).'.pdf',
             ]);
 
-            $batch = Bus::batch([
+            /**
+             * @NOTE
+             * making them PDF for now
+             * I ran into "noise" issues
+             * of just a lot of script tags and stuff
+             * there is some code in the getPage for html
+             * that might be worth it later
+             */
+            Bus::batch([
                 new ParsePdfFileJob($document),
             ])
                 ->name('Process PDF Document - '.$document->id)
@@ -118,7 +126,7 @@ public function handle(): void
                     ]
                 );
 
-                Log::info('[Larachain] adding to new batch');
+                Log::info('[LaraChain] adding to new batch');
 
                 $this->batch()->add([
                     new VectorlizeDataJob($DocumentChunk),