RAG isn't all you need!
A tale of experience vs intellect
Imagine you are running a law firm and need to hire a paralegal to help with fact-finding and research. You need someone right away as you recently lost someone, and the caseload has been increasing. You have two candidates, the first Larry, has a number of years of experience in the space, most recently having worked at a similar firm in the same area, and is looking to join you to grow. The second Linda, is fresh out of college, where she graduated top of class. She comes with a background in literature and her legal experience comes from reading John Grisham’s legal thrillers. Who do you hire?
A similar story is playing out in the LLM world, where we are going all in on Linda. Is that the right choice? In the case of LLMs, Retrieval-Augmented Generation (RAG) has become the answer to everything (our 42). Everywhere you look, there are solutions leveraging vector databases to find and using LLMs to summarize the results.
RAG has a lot going for it. Looking for the answers from text reduces hallucinations by providing fact-based answers. The retrieval aspect can apply traditional approaches to access control on the content itself. As the model never learns from the private data directly, it can only provide answers from what is shared with it. This limits any security and privacy concerns that seemed like big concerns for LLMs.
With the LLMs having read a lot of the Internet, in this case they are a lot like Linda, with a general knowledge about a lot of areas, including legal, however zero experience actually working on these problems. So while they can do a reasonable job at summarizing text that’s provided to them, this is done with a limited “understanding” of that text, particularly when the content starts to be more domain specific.
A number of the largest models have been trained on data across the web that includes some content from many subjects across the board, however as specialization grows the understanding diminishes rapidly. For more complex documents, fine tuning the model with domain specific data can help. Just like we may expect Larry to do better coming in the door.
One thing to note is that models that are better at general tasks (often larger) learn from the domain a lot more quickly (with fewer examples) than other models. This is no different from what we see in the real world, where Linda perhaps with her sharp skills and ideas across domains may not be the best at the beginning, but may be able to pick up things quickly.
To get another perspective I posed the original question to Gemini and I thought its recommendation was quite on point. The entire answer from Gemini is included at the end.
Gemini’s Recommendation: While Linda has potential, given the pressing need for immediate help, Larry is the most practical choice. However, explore options to incorporate Linda's potential like mentorship or a phased entry into the role.
We shouldn’t lose sight of what RAG enables and what it is limited for. While it may be great for a generic intranet search, a dumb implementation of RAG using the current best open source model is not the best solution for understanding a medicine database.
Can we have the cake and eat it too? Combining the intellect and experience, i.e. fine tuning a strong model on your custom dataset, and then using it as part of a RAG solution, seems like a much better bet.
Gemini’s Answer
In this scenario, given the immediate need for help with fact-finding and research due to a recent staff loss and increasing caseload, Larry would be the stronger candidate to hire. Here's why:
Immediate Contribution: Larry's experience in the legal field and similar practice area allows him to contribute effectively from day one. He'll understand legal research methodologies, relevant databases, and how to navigate the legal system for fact-finding.
Reduced Training Time: With Larry, there's minimal onboarding required, allowing you to integrate him quickly into the workflow and address pressing casework.
Reduced Risk: Larry's proven track record minimizes the risk of errors or delays due to inexperience. You can rely on him to handle tasks efficiently and accurately.
While Linda may be a promising candidate:
Significant Training Time: Her lack of experience in the legal field necessitates extensive training in legal research techniques, relevant resources, and legal terminology. This will place an additional burden on your already stretched resources.
Uncertain Learning Curve: It's hard to predict how quickly Linda can grasp complex legal concepts and procedures. You can't guarantee an immediate positive impact on your workload.
However, considering Linda's potential:
Potential for Growth: She could be a valuable asset in the long run. Her strong academic background suggests a quick learning ability.
Mentorship Opportunity: You could consider a mentorship program to train Linda alongside Larry. This leverages his experience while allowing Linda to learn and contribute.
Alternatives to Consider:
Temporary Paralegal: You could consider hiring a temporary paralegal to bridge the gap while training Linda. This option provides immediate assistance while offering time to develop Linda's skills.
Part-Time Role for Linda: Offer Linda a part-time paralegal role with a focus on research tasks that align with her background (e.g., legal document analysis). This allows her to gain experience while minimizing disruption to your workflow.
Conclusion:
While Linda has potential, given the pressing need for immediate help, Larry is the most practical choice. However, explore options to incorporate Linda's potential like mentorship or a phased entry into the role.