Evaluate via SDK
When you arrive in a new organization, you will see these steps. They tell you how to run your first experiment:
Install Braintrust libraries
First, install the Braintrust SDK (TypeScript, Python and API wrappers in other languages).
or
Create a simple evaluation script
The eval framework allows you to declaratively define evaluations in your code. Inspired by tools like Jest, you can define a set of evaluations in files named _.eval.ts or _.eval.js (Node.js) or eval_*.py (Python).
Create a file named tutorial.eval.ts
or eval_tutorial.py
with the following code.
This script sets up the basic scaffolding of an evaluation:
data
is an array or iterator of data you'll evaluatetask
is a function that takes in an input and returns an outputscores
is an array of scoring functions that will be used to score the tasks's output
(You can also write your own code. Make sure to follow the naming conventions for your language. TypeScript
files should be named *.eval.ts
and Python files should be named eval_*.py
.)
Create an API key
Next, create an API key to authenticate your evaluation script. You can create an API key in the settings page.
Run your evaluation script
Run your evaluation script with the following command:
This will create an experiment in Braintrust. Once the command runs, you'll see a link to your experiment.
View your results
Congrats, you just ran an eval! You should see a dashboard like this when you load your experiment. This view is called the experiment view, and as you use Braintrust, we hope it becomes your trusty companion each time you change your code and want to run an eval.
The experiment view allows you to look at high level metrics for performance, dig into individual examples, and compare your LLM app's performance over time.
Run another experiment
After running your first evaluation, you’ll see that we achieved a 77.8% score. Can you adjust the evaluation to improve this score? Make your changes and re-run the evaluation to track your progress.
Next steps
- Dig into our Evals guide to learn more about how to run evals.
- Look at our cookbook to learn how to evaluate RAG, summarization, text-to-sql, and other popular use cases.
- Learn how to log traces to Braintrust.
- Read about Braintrust's platform and architecture.