AUTO Learning Objective Generation: RAG retrieval causes incomplete course coverage due to similarity bias and 200-chunk limit

[kelvin-xu]

🐛 Problem

When generating learning objectives in AUTO mode, the current RAG-based retrieval approach has two compounding issues:

1. Wrong retrieval strategy — RAG retrieves chunks based on similarity to a query. For a generic query like "generate learning objectives", all course content scores similarly high, meaning the 200 chunks returned are not meaningfully filtered — they are essentially arbitrary.

2. Insufficient coverage — The 200-chunk retrieval limit means that for large materials or if we allow instructors to select multiple materials in AUTO mode, a significant portion of content never reaches the LLM. Because of the 200-chunk limit, the remaining chunks are silently excluded and the generated LOs will not reflect the full course.

The result is learning objectives that are both incomplete and biased toward whichever chunks happened to score marginally higher — not a reliable representation of what the course actually covers.

Note: Manual mode is unaffected, as instructors provide their own learning objectives and RAG is well-suited for that targeted retrieval pattern.

---

🎯 Goal

Find an approach that balances accuracy and cost for AUTO learning objective generation:

• Accuracy — Generated LOs must reflect the full course content, not a partial or biased subset
• Cost — The solution must avoid expensive LLM calls on every request, and must not pass raw course content to the LLM unnecessarily

The ideal solution ensures complete course coverage while keeping LLM processing to the minimum required.

---

🔍 Root Cause

RAG is fundamentally misaligned with the requirements of AUTO learning objective generation:

	RAG	LO Generation (AUTO)
Goal	Find relevant chunks	Cover all content
Mindset	"What is most similar to this query?"	"What does this entire course cover?"
Ideal for	Q&A, search, chat	Course auditing, LO generation

The core issue is that there is no meaningful relevance gap for RAG to exploit in this context. The 200-chunk limit then arbitrarily excludes lower-scoring chunks, producing incomplete and unreliable learning objectives.

---

✅ Proposed Solution: Lazy Summarization with Persistent Caching

Bypass RAG entirely for AUTO LO generation. Instead, use a summarization pipeline that runs once per course material and caches results for all subsequent requests.

Flow

UPLOAD (already working)
─────────────────────────────────────────────────────
Extract text + describe images (LiteParse)
│
└──► Chunk → embed → Qdrant

FIRST AUTO LO GENERATION REQUEST
─────────────────────────────────────────────────────
Check DB: summary exists for this material?
│
No
│
▼
Qdrant.scroll(source_id)
└── Fetch ALL chunks by source_id filter (no similarity search)
└── Sort by page + chunk_index to restore document order
│
▼
Batch chunks → Summarize all batches via LLM
└── Run in parallel to reduce latency
└── Combine all batch summaries into one final summary
│
▼
Save single summary entry to DB
│
▼
Generate LOs from summary

SUBSEQUENT AUTO LO GENERATION REQUESTS (same material)
─────────────────────────────────────────────────────
Check DB: summary exists for this source_id?
│
Yes
│
▼
Fetch cached summary from DB
│
▼
Generate LOs ← LLM never reached for summery again, no reprocessing cost

Why This Works

• ✅ Full coverage — scroll retrieves all chunks, not top-K by similarity
• ✅ Cost efficient — summarization runs once per course material, not per request
• ✅ No raw text duplication — Qdrant remains the single source of truth for course content