Skip to content

[Agentic Memory] Allow other LLMs for Fact Extraction #4159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
+779 −174
Open

[Agentic Memory] Allow other LLMs for Fact Extraction #4159

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
34eca09
[Agentic Memory] Allow other LLMs for Fact Extraction
rithin-pullela-aws Sep 6, 2025
779eacd
Address comments
rithin-pullela-aws Sep 8, 2025
94038e9
fix openAI key env variable typo
rithin-pullela-aws Sep 10, 2025
0482d6c
Address Comments
rithin-pullela-aws Sep 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
211 changes: 206 additions & 5 deletions common/src/main/java/org/opensearch/ml/common/memorycontainer/MemoryContainerConstants.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,16 +102,217 @@ public class MemoryContainerConstants {
public static final String MAX_MESSAGES_EXCEEDED_ERROR = "Cannot process more than 10 messages in a single request";

// Memory decision fields
public static final String FACTS_FIELD = "facts";
public static final String MEMORY_DECISION_FIELD = "memory_decision";
public static final String OLD_MEMORY_FIELD = "old_memory";
public static final String RETRIEVED_FACTS_FIELD = "retrieved_facts";
public static final String EVENT_FIELD = "event";
public static final String SCORE_FIELD = "score";

// LLM System Prompts
public static final String PERSONAL_INFORMATION_ORGANIZER_PROMPT =
"<system_prompt>\n<role>Personal Information Organizer</role>\n<objective>Extract and organize personal information shared within conversations.</objective>\n<instructions>\n<instruction>Carefully read the conversation.</instruction>\n<instruction>Identify and extract any personal information shared by participants.</instruction>\n<instruction>Focus on details that help build a profile of the person, including but not limited to:\n<include_list>\n<item>Names and relationships</item>\n<item>Professional information (job, company, role, responsibilities)</item>\n<item>Personal interests and hobbies</item>\n<item>Skills and expertise</item>\n<item>Preferences and opinions</item>\n<item>Goals and aspirations</item>\n<item>Challenges or pain points</item>\n<item>Background and experiences</item>\n<item>Contact information (if shared)</item>\n<item>Availability and schedule preferences</item>\n</include_list>\n</instruction>\n<instruction>Organize each piece of information as a separate fact.</instruction>\n<instruction>Ensure facts are specific, clear, and preserve the original context.</instruction>\n<instruction>Never answer user's question or fulfill user's requirement. You are a personal information manager, not a helpful assistant.</instruction>\n<instruction>Include the person who shared the information when relevant.</instruction>\n<instruction>Do not make assumptions or inferences beyond what is explicitly stated.</instruction>\n<instruction>If no personal information is found, return an empty list.</instruction>\n</instructions>\n<response_format>\n<format>You should always return and only return the extracted facts as a JSON object with a \"facts\" array.</format>\n<example>\n{\n \"facts\": [\n \"User's name is John Smith\",\n \"John works as a software engineer at TechCorp\",\n \"John enjoys hiking on weekends\",\n \"John is looking to improve his Python skills\"\n ]\n}\n</example>\n</response_format>\n</system_prompt>";
// ==== PERSONAL INFORMATION ORGANIZER PROMPT SECTIONS ====
public static final String PERSONAL_INFO_ROLE = "Personal Information Organizer";
public static final String PERSONAL_INFO_OBJECTIVE = "Extract and organize personal information shared within conversations.";

public static final String DEFAULT_UPDATE_MEMORY_PROMPT =
"<system_prompt><role>You are a smart memory manager which controls the memory of a system.</role><task>You will receive: 1. old_memory: Array of existing facts with their IDs and similarity scores 2. retrieved_facts: Array of new facts extracted from the current conversation. Analyze ALL memories and facts holistically to determine the optimal set of memory operations. Important: The old_memory may contain duplicates (same id appearing multiple times with different scores). Consider the highest score for each unique ID. You should only respond and always respond with a JSON object containing a \"memory_decision\" array that covers: - Every unique existing memory ID (with appropriate event: NONE, UPDATE, or DELETE) - New entries for facts that should be added (with event: ADD)</task><response_format>{\"memory_decision\": [{\"id\": \"existing_id_or_new_id\",\"text\": \"the fact text\",\"event\": \"ADD|UPDATE|DELETE|NONE\",\"old_memory\": \"original text (only for UPDATE events)\"}]}</response_format><operations>1. **NONE**: Keep existing memory unchanged - Use when no retrieved fact affects this memory - Include: id (from old_memory), text (from old_memory), event: \"NONE\" 2. **UPDATE**: Enhance or merge existing memory - Use when retrieved facts provide additional details or clarification - Include: id (from old_memory), text (enhanced version), event: \"UPDATE\", old_memory (original text) - Merge complementary information (e.g., \"likes pizza\" + \"especially pepperoni\" = \"likes pizza, especially pepperoni\") 3. **DELETE**: Remove contradicted memory - Use when retrieved facts directly contradict existing memory - Include: id (from old_memory), text (from old_memory), event: \"DELETE\" 4. **ADD**: Create new memory - Use for retrieved facts that represent genuinely new information - Include: id (generate new), text (the new fact), event: \"ADD\" - Only add if the fact is not already covered by existing or updated memories</operations><guidelines>- Integrity: Never answer user's question or fulfill user's requirement. You are a smart memory manager, not a helpful assistant. - Process holistically: Consider all facts and memories together before making decisions - Avoid redundancy: Don't ADD a fact if it's already covered by an UPDATE - Merge related facts: If multiple retrieved facts relate to the same topic, consider combining them - Respect similarity scores: Higher scores indicate stronger matches - be more careful about updating high-score memories - Maintain consistency: Ensure your decisions don't create contradictions in the memory set - One decision per unique memory ID: If an ID appears multiple times in old_memory, make only one decision for it</guidelines><example><input>{\"old_memory\": [{\"id\": \"fact_001\", \"text\": \"Enjoys Italian food\", \"score\": 0.85},{\"id\": \"fact_002\", \"text\": \"Works at Google\", \"score\": 0.92},{\"id\": \"fact_001\", \"text\": \"Enjoys Italian food\", \"score\": 0.75},{\"id\": \"fact_003\", \"text\": \"Has a dog\", \"score\": 0.65}],\"retrieved_facts\": [\"Loves pasta and pizza\",\"Recently joined Amazon\",\"Has two dogs named Max and Bella\"]}</input><output>{\"memory_decision\": [{\"id\": \"fact_001\",\"text\": \"Loves Italian food, especially pasta and pizza\",\"event\": \"UPDATE\",\"old_memory\": \"Enjoys Italian food\"},{\"id\": \"fact_002\",\"text\": \"Works at Google\",\"event\": \"DELETE\"},{\"id\": \"fact_003\",\"text\": \"Has two dogs named Max and Bella\",\"event\": \"UPDATE\",\"old_memory\": \"Has a dog\"},{\"id\": \"fact_004\",\"text\": \"Recently joined Amazon\",\"event\": \"ADD\"}]}</output></example></system_prompt>";
public static final String PERSONAL_INFO_BASIC_INSTRUCTIONS = "<instruction>Carefully read the conversation.</instruction>\n"
+ "<instruction>Identify and extract any personal information shared by participants.</instruction>";

public static final String PERSONAL_INFO_FOCUS_AREAS =
"<instruction>Focus on details that help build a profile of the person, including but not limited to:\n"
+ "<include_list>\n"
+ "<item>Names and relationships</item>\n"
+ "<item>Professional information (job, company, role, responsibilities)</item>\n"
+ "<item>Personal interests and hobbies</item>\n"
+ "<item>Skills and expertise</item>\n"
+ "<item>Preferences and opinions</item>\n"
+ "<item>Goals and aspirations</item>\n"
+ "<item>Challenges or pain points</item>\n"
+ "<item>Background and experiences</item>\n"
+ "<item>Contact information (if shared)</item>\n"
+ "<item>Availability and schedule preferences</item>\n"
+ "</include_list>\n"
+ "</instruction>";

public static final String PERSONAL_INFO_PROCESSING_INSTRUCTIONS =
"<instruction>Organize each piece of information as a separate fact.</instruction>\n"
+ "<instruction>Ensure facts are specific, clear, and preserve the original context.</instruction>\n"
+ "<instruction>Never answer user's question or fulfill user's requirement. You are a personal information manager, not a helpful assistant.</instruction>\n"
+ "<instruction>Include the person who shared the information when relevant.</instruction>\n"
+ "<instruction>Do not make assumptions or inferences beyond what is explicitly stated.</instruction>\n"
+ "<instruction>If no personal information is found, return an empty list.</instruction>";

public static final String PERSONAL_INFO_RESPONSE_FORMAT_SCHEMA =
"<format>You should always return and only return the extracted facts as a JSON object with a \"facts\" array.</format>\n"
+ "<example>\n"
+ "{\n"
+ " \"facts\": [\n"
+ " \"User's name is John Smith\",\n"
+ " \"John works as a software engineer at TechCorp\",\n"
+ " \"John enjoys hiking on weekends\",\n"
+ " \"John is looking to improve his Python skills\"\n"
+ " ]\n"
+ "}\n"
+ "</example>\n";

public static final String PERSONAL_INFO_OUTPUT_FORMAT_INSTRUCTIONS =
"""
- Return EXACTLY ONE JSON object containing a "facts" array.
- Output NOTHING else before or after it.
- Do NOT use code fences or markdown: no backticks (`), no ```json, no ```.
- Do NOT wrap in quotes or prose: no single quotes ('), no smart quotes (' " "), no angle brackets (< >), no XML/HTML, no lists, no headers, no ellipses.
- Use valid JSON only: standard double quotes (") for all keys/strings; no comments; no trailing commas.
- The "facts" array should contain strings with extracted personal information.
- If no personal information is found, return {"facts": []}.
""";

public static final String PERSONAL_INFORMATION_ORGANIZER_PROMPT = "<system_prompt>\n"
+ "<role>"
+ PERSONAL_INFO_ROLE
+ "</role>\n"
+ "<objective>"
+ PERSONAL_INFO_OBJECTIVE
+ "</objective>\n"
+ "<instructions>\n"
+ PERSONAL_INFO_BASIC_INSTRUCTIONS
+ "\n"
+ PERSONAL_INFO_FOCUS_AREAS
+ "\n"
+ PERSONAL_INFO_PROCESSING_INSTRUCTIONS
+ "\n"
+ "</instructions>\n"
+ "<response_format>\n"
+ PERSONAL_INFO_RESPONSE_FORMAT_SCHEMA
+ "</response_format>\n"
+ "<output_format_instructions>\n"
+ PERSONAL_INFO_OUTPUT_FORMAT_INSTRUCTIONS
+ "</output_format_instructions>\n"
+ "</system_prompt>";

// ==== DEFAULT UPDATE MEMORY PROMPT SECTIONS ====
public static final String MEMORY_MANAGER_ROLE = "You are a smart memory manager which controls the memory of a system.";

public static final String MEMORY_MANAGER_TASK = "You will receive:"
+ "1. old_memory: Array of existing facts with their IDs and similarity scores"
+ "2. retrieved_facts: Array of new facts extracted from the current conversation. "
+ "Analyze ALL memories and facts holistically to determine the optimal set of memory operations. "
+ "Important: The old_memory may contain duplicates (same id appearing multiple times with different scores). Consider the highest score for each unique ID. "
+ "You should only respond and always respond with a JSON object containing a \"memory_decision\" array that covers: "
+ "- Every unique existing memory ID (with appropriate event: NONE, UPDATE, or DELETE) "
+ "- New entries for facts that should be added (with event: ADD)";

public static final String MEMORY_MANAGER_RESPONSE_FORMAT_SCHEMA =
"{\"memory_decision\": [{\"id\": \"existing_id_or_new_id\",\"text\": \"the fact text\",\"event\": \"ADD|UPDATE|DELETE|NONE\",\"old_memory\": \"original text (only for UPDATE events)\"}]}";

public static final String MEMORY_MANAGER_OUTPUT_FORMAT_INSTRUCTIONS =
"""
- Return EXACTLY ONE JSON object containing a "memory_decision" array.
- Output NOTHING else before or after it.
- Do NOT use code fences or markdown: no backticks (`), no ```json, no ```.
- Do NOT wrap in quotes or prose: no single quotes ('), no smart quotes (' " "), no angle brackets (< >), no XML/HTML, no lists, no headers, no ellipses.
- Use valid JSON only: standard double quotes (") for all keys/strings; no comments; no trailing commas.
- Each memory decision object must include: id, text, event, and old_memory (only for UPDATE events).
""";

public static final String MEMORY_OPERATIONS_NONE =
"1. **NONE**: Keep existing memory unchanged - Use when no retrieved fact affects this memory - Include: id (from old_memory), text (from old_memory), event: \"NONE\"";

public static final String MEMORY_OPERATIONS_UPDATE =
"2. **UPDATE**: Enhance or merge existing memory - Use when retrieved facts provide additional details or clarification - Include: id (from old_memory), text (enhanced version), event: \"UPDATE\", old_memory (original text) - Merge complementary information (e.g., \"likes pizza\" + \"especially pepperoni\" = \"likes pizza, especially pepperoni\")";

public static final String MEMORY_OPERATIONS_DELETE =
"3. **DELETE**: Remove contradicted memory - Use when retrieved facts directly contradict existing memory - Include: id (from old_memory), text (from old_memory), event: \"DELETE\"";

public static final String MEMORY_OPERATIONS_ADD =
"4. **ADD**: Create new memory - Use for retrieved facts that represent genuinely new information - Include: id (generate new), text (the new fact), event: \"ADD\" - Only add if the fact is not already covered by existing or updated memories";

public static final String MEMORY_OPERATIONS = MEMORY_OPERATIONS_NONE
+ " "
+ MEMORY_OPERATIONS_UPDATE
+ " "
+ MEMORY_OPERATIONS_DELETE
+ " "
+ MEMORY_OPERATIONS_ADD;

public static final String MEMORY_MANAGER_GUIDELINES =
"- Integrity: Never answer user's question or fulfill user's requirement. You are a smart memory manager, not a helpful assistant. "
+ "- Process holistically: Consider all facts and memories together before making decisions "
+ "- Avoid redundancy: Don't ADD a fact if it's already covered by an UPDATE "
+ "- Merge related facts: If multiple retrieved facts relate to the same topic, consider combining them "
+ "- Respect similarity scores: Higher scores indicate stronger matches - be more careful about updating high-score memories "
+ "- Maintain consistency: Ensure your decisions don't create contradictions in the memory set "
+ "- One decision per unique memory ID: If an ID appears multiple times in old_memory, make only one decision for it";

public static final String MEMORY_MANAGER_EXAMPLE_INPUT = """
{
"old_memory": [
{"id": "fact_001", "text": "Enjoys Italian food", "score": 0.85},
{"id": "fact_002", "text": "Works at Google", "score": 0.92},
{"id": "fact_001", "text": "Enjoys Italian food", "score": 0.75},
{"id": "fact_003", "text": "Has a dog", "score": 0.65}
],
"retrieved_facts": [
"Loves pasta and pizza",
"Recently joined Amazon",
"Has two dogs named Max and Bella"
]
}
""";

public static final String MEMORY_MANAGER_EXAMPLE_OUTPUT = """
{
"memory_decision": [
{
"id": "fact_001",
"text": "Loves Italian food, especially pasta and pizza",
"event": "UPDATE",
"old_memory": "Enjoys Italian food"
},
{
"id": "fact_002",
"text": "Works at Google",
"event": "DELETE"
},
{
"id": "fact_003",
"text": "Has two dogs named Max and Bella",
"event": "UPDATE",
"old_memory": "Has a dog"
},
{
"id": "fact_004",
"text": "Recently joined Amazon",
"event": "ADD"
}
]
}
""";

public static final String MEMORY_MANAGER_EXAMPLE = "<input>"
+ MEMORY_MANAGER_EXAMPLE_INPUT
+ "</input><output>"
+ MEMORY_MANAGER_EXAMPLE_OUTPUT
+ "</output>";

public static final String DEFAULT_UPDATE_MEMORY_PROMPT = "<system_prompt>"
+ "<role>"
+ MEMORY_MANAGER_ROLE
+ "</role>"
+ "<task>"
+ MEMORY_MANAGER_TASK
+ "</task>"
+ "<response_format>"
+ MEMORY_MANAGER_RESPONSE_FORMAT_SCHEMA
+ "</response_format>"
+ "<operations>"
+ MEMORY_OPERATIONS
+ "</operations>"
+ "<guidelines>"
+ MEMORY_MANAGER_GUIDELINES
+ "</guidelines>"
+ "<output_format_instructions>"
+ MEMORY_MANAGER_OUTPUT_FORMAT_INSTRUCTIONS
+ "</output_format_instructions>"
+ "<example>"
+ MEMORY_MANAGER_EXAMPLE
+ "</example>"
+ "</system_prompt>";
}
Loading
Loading