How to construct domain-specific LLM evaluation systems.

[on-hold music] My name is Emil Sedgh. I'm CTO at Rechat, and, uh, with my partner Hamel, uh, we're gonna talk about the product we built, um, the challenges we faced, and, uh, and how our eval framework came to the

rescue, and we'll also show you some results.

Uh, a little bit about us and, uh, how the product that we built came to be. Uh, last year we tried to see if we have, uh, any AI play.

Our application is designed for real estate agents and brokers, and we had a lot of features like contact management, email marketing, social marketing, whatever. Uh, so we realized, we realized that we have a lot of APIs, uh, that we've built internally, and we have a lot of data.

So naturally, we came to the unique and brilliant idea that we need to build an AI agent for our real estate agents.

So, uh, I'm gonna, uh, rewind back a year. Basically last year, um, when we started this, we started with the process of creating a prod- prototype. Uh, we built this prototype using GPT, uh, the original GPT-3.5, uh, and React framework.

It was very, very slow, and, uh, it was making mistakes all the time. Uh, but when it worked, it was a majestic experience. It was beautiful experience. So we thought, "Okay, we got the products in a demo state, but now we have to take it to production," and that's when we started, um, uh, partnering up with Hamel,

uh, to, uh, to basically create a, a production-ready, uh, uh, product. I'm gonna show you some very, very basic examples of how this product work bas- works. Basically, agents ask it to do things for them, like create a, a contacts for me with this information, or send an email to somebody with some instructions, um, find me some

listings, because that's... real estate agents, uh, tend to do, uh, or create a website for me.

Uh, so yeah, we created this, uh, prototype, then we started the improvement of language model phase. Uh, the problem was when we tried to make changes, uh, to see if we can improve it, we didn't really know, uh, if we're improving things or not.

We would make a change, we would invoke it a couple of times. Uh, we would get a feeling that, yeah, it worked a couple of times, uh, but we don't, we didn't really know what the success rate or failure rate was.

Is it gonna work sixty percent of times or eighty percent of times? And it's very difficult to launch a production app when you don't really know how well it's gonna function.

The other problem was we improved the situation. We got a feeling that it's okay, it's improving this situation, but the moment we change the prompts, it was likely that it's gonna break other use cases.

Uh, and we were essentially in the dark. Uh, and that's when we started to partner up with Hamel to guide us to see if we can make this app production ready.

Uh, I'm gonna let him, uh, take it from here.

Thanks, Emil. So what Emil described is he was able to use prompt engineering, implement RAG, agents, so on and so forth, and iterate with just vibe checks really fast to go from zero to one.

And this is a really common approach to building an MVP. It actually works really we- well for building an MVP.

However, in reality, this approach doesn't work for that long at all. It leads to stagnation, and if you don't have a way of measuring progress, you can't really build.

So in this talk, what I'm gonna go over is a systematic approach you can use to improve your AI consistently.

I'm also gonna talk about how to avoid common traps and give you some resources on how to learn more, because you can't learn everything in a fifteen-minute talk.

This diagram is an illustration of the recipe of this systematic approach, um, of creating an evaluation framework. You don't have to fixate too much on the details of this diagram because I'm gonna be walking through it slowly.

But the first thing I wanna talk about is unit tests and assertions.

So a lot of people are familiar with unit tests and assertions if you have been building software. Uh, but for whatever reason, people tend to skip this step.

Um, and it's kind of the foundation for evaluation systems. You don't want to jump straight to LLM-as-a-judge or generic evals. You want to try to write down as many assertions and unit tests as you can about the failure modes that you're, that you're experiencing with your large language model.

And it, it really comes from looking at data. So what you have on the slide here are some simple unit tests and assertions that Rechat wrote based upon failure modes that we observed in the data.

And these are not all of them. There's many of these, but these are just examples of, like, very simple things like testing if agents are working properly, so an email's not being sent, or things like invalid placeholders or other details being repeated when they shouldn't.

The, the details of these specific assertions don't matter. What I'm trying to drive home is this is a very simple thing that people skip, but it's absolutely essential because running these assertions give you immediate feedback and are almost free to run.

And it's really critical to your overall evaluation system if you can have them.

And how do you run the assertions? One very reasonable way is to use CI. You can outgrow CI, and it may not work as you mature, but one theme I want to get across is use what you have when you begin.

Don't jump straight into tools. Another thing that you wanna do with these assertions and unit tests is log the results to a database But when you're starting out, you want to keep it simple and stupid.

Use your existing tools. So in Rechat's case, they were already using Metabase, so we log these results to Metabase, and then use Metabase to, like, visualize and track the results so that we could see if we're making progress on these dumb failure modes over time.

Again, my recommendation is don't buy stuff. Use what you have when you, when you're beginning, and then get into tools later, and I'll talk more about that in a minute.

So we talked a little bit about unit tests and assertions. The next thing I wanna talk about is logging and human review.

So it's important to log your traces. Um, there's a lot of tools that you can use to do this. This is one area where I actually do suggest using a tool right off the bat.

Um, there's a lot of commercial tools and open source tools that are listed on this slide. In Rechat's case, they ended up using LangSmith.

But more importantly than, you know, it's not enough to just log your traces, you have to look at them. Otherwise, there's no point in logging them. And one kind of nuance here is that looking at your data is so important that I actually recommend building your own data viewing and annotation tools in a lot of cases.

And the reason is because your data and application are often very unique. There's a lot of domain-specific stuff in your traces. So in Rechat's case, we found that tools had too much friction for us, so we built our own kind of little application, and you can do this very easily in something like Gradio, Streamlit.

I use Shiny for Python. It really doesn't matter. But we have a lot of domain-specific stuff in this, like, webpage, things that allows us to filter data in ways that are very specific to Rechat, but then also lots of other metadata that's associated with each trace that is Rechat-specific that-- where I don't have to hunt for information

to evaluate a trace. And then there's other things going on here. This is not only a kind of a data viewing app, this is also a data labeling app where it k- it, like, facilitates human review, um, which I'll talk about in a second.

So this is the most important part. If you remember anything from this talk, it is you need to look at your data, and you need to fight as hard as you can to remove all friction in looking at your data, even down to creating your own data viewing apps if you have to.

And it's absolutely critical. If you have any friction in looking at data, people are not going to do it, and it will destroy the whole process, and none of this is gonna work.

So we talked a little bit about unit tests,

logging your traces, and human review. Um, and you might be wondering, okay, like, you have these tests, but what about the test cases? What do we do about that?

Especially when you're starting out, you might not have any users.

So you can use LLMs to systematically generate inputs to your system. So in Rechat's case, we basically use an LLM to cosplay as a real estate agent and ask questions as inputs into this, uh, into Lucy, which is their AI assistant,

for all the different features and the scenarios and the tools to get really good test coverage. So I just wanna point out that using LLMs to synthetically generate inputs is a good way to bootstrap these test cases.

So we talked a little bit about unit tests, logging traces,

um, you know, h- having a human review. And so when you have a very minimal setup like this, this is, like, the very minimal thing, like, a very minimal evaluation system, like bare bones.

And what you want to do when you first kind of construct that is you want to test out the evaluation system. So you want to do something to make progress on your AI, and the easiest way to try to make progress on your AI is to do prompt engineering.

So what you should do is go through this loop as many times as possible. Uh, you know, try to improve your AI with prompt engineering and see if your test coverage is good.

Are you logging your, uh, are you logging your traces correctly? Um, did you remove as much friction as possible from looking at your data? And it-- this will help you debug that, but also give you the satisfaction of, like, making progress on your AI as well.

One thing I wanna point out is the upshot of having an evaluation system is you get other superpowers for almost free. So all of the work in fine-tuning, or most of the work, is data curation.

So we already talked about, like, synthetic data generation and how that interacts with the eval framework. And what you can do is you can use your eval framework to kind of filter out good cases and feed that into your human review, um, like we showed with that application, and you can start to curate data for fine-tuning.

And also for the failed cases, you have this workflow that you can use to work through those and continuously update your fine-tuning data. And what we've seen over time is that the more comprehensive your eval framework is, the, the cost of human review goes down because you're automating more and more, um, of these things and getting more

confidence in your data. So once you have kind of this setup, now you're in a position, like, to know whether or not you're making progress or not. You have a workflow that you can use to quickly make improvements, and you can start getting rid of those dumb failure modes.

But also, now you're set up to move into more advanced things like LLM-as-a-judge, because you can't express everything as an assertion,

um, or a unit test. Now, LLM-as-a-judge is a deep topic that is outside the scope of this talk. But one thing I wanna point out is it's very, very important to align the LLM judge to a human because you need to know whether you can trust the LLM as a judge.

You need a way, a principled way of reasoning about how reliable the LLM as a judge is. So

what I like to do is, again, keep it simple and stupid. I like to use a spreadsheet often. Don't make it complicated. But what I do is have a domain expert label data, uh, you know, label the critique in, in critique data, and keep iterating on that until my LLM as a judge is in alignment with my

human judge, and I have high confidence that the LLM judge is doing what it's supposed to do.

So I'm gonna go through some common mistakes that people make when building LLM-as-evaluation systems. One

is not looking at your data. It's easier said than done, but the-- people don't do the best job of doing this, and one key to unlocking this is to remove all the friction, as I mentioned before.

The second one, and this is just as important, is focusing on tools, not processes.

So if, if you're having a conversation about evals and the first thing you start thinking about is tools, that's a smell that you're not going to be successful in your evaluations.

People like to jump straight to the tools. "Tell me about the tools. What tools should I use?" It's really important to try not to use tools to begin with and try to do some of these things manually with what you already have.

Because if you don't do that, you won't be able to evaluate the tools, and you should-- you have to know what the process is before you jump straight into the tools.

Otherwise, it's gonna-- you're gonna be blindsided. Another common mistake is people using generic evals off the shelf. So

don't wanna reach for generic evals. You want to write evals that are very specific to your domain, things like conciseness score, toxicity score, you know, all these different evals you can get off the shelf with tools.

You don't want to go directly to those. That's also a smell that you are not doing things correctly.

It's not that they're not valuable at all. It's just that you shouldn't rely on them because they can become a crutch. And then finally, the other common mistake is with LLM as a judge and using that too early.

I often find that if I'm looking at the data closely enough, I can al-always find plenty of assertions and failure modes. It's not always the case, but it's often the case.

So don't go to LLM as a judge too early, and also make sure you align LLM as, as a judge with a human.

So I'm gonna flip it back over to Emil, and he's gonna talk about the results of implementing this system.

All right, so after we got to the virtuous, um, cycle that Hamel just displayed, we were mana-- uh, we managed to rapidly increase the success rate of the LLM application.

Uh, without the eval framework, a, a project similar to this seemed completely impossible for us. Uh, one, one thing that I've started to hear a lot is that few-shot prompting is gonna replace fine-tuning or, uh, some notions like that.

Uh, in our case, uh, we never managed to get everything that we wanted by few-shot prompting, even using the newer, uh, and smarter agents. Uh, s- I wish we could.

I, I've seen a lot of, uh, judgment of companies and products being just ChatGPT wrappers. I wish we could just be a ChatGPT wrapper and manage to extract the experience we want for our users.

But we never, uh, had that opportunity because we had some really difficult cases. Uh, one of the things that we wanted our agent to be able to do was to mix, uh, natural language with user interface elements like this inside the output, and this essentially required us to, uh, mix structured output and unstructured output together.

Uh, we never managed to get this working, uh, without fine-tuning, uh, reliably. Uh, another thing was, uh, feedback. So sometimes the user asks in a case like this, "Do this for me," but the agent can't just do that.

It needs, uh, some sort of feedback, more, uh, input from the user. Again, something like this was very difficult for us to execute on, especially given the previous, um, challenge of injecting, uh, user interfaces inside the conversation.

Uh, and third reason that we had to, um, fine-tune was complex commands like this. Uh, I'm gonna show a tiny video that shows how this command was executed. Uh, but basically in this example, um, the user is asking, uh, for a very complex command that requires using like five or six different tools, uh, to be done.

Uh, basically, what we wanted is-- was for it to take that input, break it down into, uh, many different, uh, function calls, and execute it. Uh, so in this case, I'm asking it to find me some listings with some criteria a-and then create a website.

That's what real estate agents sometimes do for their listings that they're responsible for, and also an Instagram post, so they wanna market it. Uh, they want this done only for the most, most expensive listing of these three.

So the, um, the application has found three listings, created a website for that, created and rendered an Instagram, uh, post video for it, uh, and then has prepared an email to Hamel, including all the information about the listings, um, and also including the website that was created and the Instagram story that was created.

Also, um, invited Hammer, uh, Hamel to a dinner and created a follow-up task. Creating something like this for a non-savvy, uh, real estate agent may take a couple of hours to do.

But using the agent, um, they can do it in a minute, and that essentially was not gonna be possible without us using a comprehensive eval framework. Nailed the timing.

Thank you, guys. [upbeat music]