There are two main AI contextualising methods for adding additional knowledge to a prompt. These are Retrieval Augmented Generation and Long context prompting.
| Retrieval Augmented Generation | Long context prompting | |
|---|---|---|
| Pros | Pre-processed, read once | Simpler infrastructure |
| Gives model less noise | User provides all relevant context | |
| Filters large amounts of content too big for models | ||
| Cons | Complex preparation and search infrastructure | Re-reading tax (reads all context even if not needed) |
| Silent failure (semantic model provides limited information) | Needle in a haystack (model gets lost in detail) | |
| Whole book problem (gaps between chunks provided by semantic model lookup) |
Long context prompting is suited to smaller datasets or few documents.
Retrieval Augmented Generation is suited to large datasets such as found in a business.
Source: Is RAG Still Needed? Choosing the Best Approach for LLMs - YouTube