Evaluation

Evaluation is a form of testing that helps you validate your LLM’s responses and ensure they meet your quality bar.

Genkit supports third-party evaluation tools through plugins, paired with powerful observability features that provide insight into the runtime state of your LLM-powered applications. Genkit tooling helps you automatically extract data including inputs, outputs, and information from intermediate steps to evaluate the end-to-end quality of LLM responses as well as understand the performance of your system’s building blocks.

Types of evaluation

Genkit supports two types of evaluation:

Inference-based evaluation: This type of evaluation runs against a collection of pre-determined inputs, assessing the corresponding outputs for quality.

This is the most common evaluation type, suitable for most use cases. This approach tests a system’s actual output for each evaluation run.

You can perform the quality assessment manually, by visually inspecting the results. Alternatively, you can automate the assessment by using an evaluation metric.
Raw evaluation: This type of evaluation directly assesses the quality of inputs without any inference. This approach typically is used with automated evaluation using metrics. All required fields for evaluation (e.g., input, context, output and reference) must be present in the input dataset. This is useful when you have data coming from an external source (e.g., collected from your production traces) and you want to have an objective measurement of the quality of the collected data.

For more information, see the Advanced use section of this page.

This section explains how to perform inference-based evaluation using Genkit.

Quick start

Perform these steps to get started quickly with Genkit.

Setup

Use an existing Genkit app or create a new one by following our Get started guide.
Add the following code to define a simple RAG application to evaluate. For this guide, we use a dummy retriever that always returns the same documents.

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/firebase/genkit/go/ai"
    "github.com/firebase/genkit/go/genkit"
    "github.com/firebase/genkit/go/plugins/googlegenai"
)

func main() {
    ctx := context.Background()

    // Initialize Genkit
    g := genkit.Init(ctx,
        genkit.WithPlugins(&googlegenai.GoogleAI{}),
        genkit.WithDefaultModel("googleai/gemini-flash-latest"),
    )

    // Dummy retriever that always returns the same facts. The last parameter is
    // the retriever's config, which this one has no use for.
    dummyRetrieverFunc := func(ctx context.Context, req *ai.RetrieverRequest, _ struct{}) (*ai.RetrieverResponse, error) {
        facts := []string{
            "Dog is man's best friend",
            "Dogs have evolved and were domesticated from wolves",
        }
        // Just return facts as documents.
        var docs []*ai.Document
        for _, fact := range facts {
            docs = append(docs, ai.DocumentFromText(fact, nil))
        }
        return &ai.RetrieverResponse{Documents: docs}, nil
    }
    factsRetriever := genkit.DefineRetrieverAction(g, "dogFacts", nil, dummyRetrieverFunc)

    m := googlegenai.GoogleAIModel(g, "gemini-flash-latest")
    if m == nil {
        log.Fatal("failed to find model")
    }

    // A simple question-answering flow
    genkit.DefineFlow(g, "qaFlow", func(ctx context.Context, query string) (string, error) {
        factDocs, err := genkit.Retrieve(ctx, g, ai.WithRetriever(factsRetriever), ai.WithTextDocs(query))
        if err != nil {
            return "", fmt.Errorf("retrieval failed: %w", err)
        }
        llmResponse, err := genkit.Generate(ctx, g,
            ai.WithModelName("googleai/gemini-flash-latest"),
            ai.WithPrompt("Answer this question with the given context: %s", query),
            ai.WithDocs(factDocs.Documents...),
        )
        if err != nil {
            return "", fmt.Errorf("generation failed: %w", err)
        }
        return llmResponse.Text(), nil
    })
}

You can optionally add evaluation metrics to your application to use while evaluating. This guide uses the EvaluatorRegex metric from the evaluators package.

import (
    "github.com/firebase/genkit/go/plugins/evaluators"
)

func main() {
    // ...

    metrics := []evaluators.MetricConfig{
        {
            MetricType: evaluators.EvaluatorRegex,
        },
    }

    // Initialize Genkit
    g := genkit.Init(ctx,
        genkit.WithPlugins(
            &googlegenai.GoogleAI{},
            &evaluators.GenkitEval{Metrics: metrics}, // Add this plugin
        ),
        genkit.WithDefaultModel("googleai/gemini-flash-latest"),
    )
}

Note: Ensure that the evaluators package is installed in your go project:

go get github.com/firebase/genkit/go/plugins/evaluators

Start your Genkit application.

genkit start -- go run main.go

Create a dataset

Create a dataset to define the examples we want to use for evaluating our flow.

Go to the Dev UI at http://localhost:4000 and click the Datasets button to open the Datasets page.
Click the Create Dataset button to open the create dataset dialog.

a. Provide a datasetId for your new dataset. This guide uses myFactsQaDataset.

b. Select Flow dataset type.

c. Leave the validation target field empty and click Save
Your new dataset page appears, showing an empty dataset. Add examples to it by following these steps:

a. Click the Add example button to open the example editor panel.

b. Only the Input field is required. Enter "Who is man's best friend?" in the Input field, and click Save to add the example has to your dataset.

If you have configured the EvaluatorRegex metric and would like to try it out, you need to specify a Reference string that contains the pattern to match the output against. For the preceding input, set the Reference output text to "(?i)dog", which is a case-insensitive regular- expression pattern to match the word “dog” in the flow output.

c. Repeat steps (a) and (b) a couple of more times to add more examples. This guide adds the following example inputs to the dataset:

"Can I give milk to my cats?"
"From which animals did dogs evolve?"

If you are using the regular-expression evaluator, use the corresponding reference strings:

"(?i)don't know"
"(?i)wolf|wolves"

Note that this is a contrived example and the regular-expression evaluator may not be the right choice to evaluate the responses from qaFlow. However, this guide can be applied to any Genkit Go evaluator of your choice.

By the end of this step, your dataset should have 3 examples in it, with the values mentioned above.

Run evaluation and view results

To start evaluating the flow, click the Run new evaluation button on your dataset page. You can also start a new evaluation from the Evaluations tab.

Select the Flow radio button to evaluate a flow.
Select qaFlow as the target flow to evaluate.
Select myFactsQaDataset as the target dataset to use for evaluation.
If you have installed an evaluator metric using Genkit plugins, you can see these metrics in this page. Select the metrics that you want to use with this evaluation run. This is entirely optional: Omitting this step will still return the results in the evaluation run, but without any associated metrics.

If you have not provided any reference values and are using the EvaluatorRegex metric, your evaluation will fail since this metric needs reference to be set.
Click Run evaluation to start evaluation. Depending on the flow you’re testing, this may take a while. Once the evaluation is complete, a success message appears with a link to view the results. Click the link to go to the Evaluation details page.

You can see the details of your evaluation on this page, including original input, extracted context and metrics (if any).

Core concepts

Terminology

Knowing the following terms can help ensure that you correctly understand the information provided on this page:

Evaluation: An evaluation is a process that assesses system performance. In Genkit, such a system is usually a Genkit primitive, such as a flow, a prompt, or a model. An evaluation can be automated or manual (human evaluation).
Bulk inference Inference is the act of running an input on a flow or model to get the corresponding output. Bulk inference involves performing inference on multiple inputs simultaneously.
Metric An evaluation metric is a criterion on which an inference is scored. Examples include accuracy, faithfulness, maliciousness, whether the output is in English, etc.
Dataset A dataset is a collection of examples to use for inference-based evaluation. A dataset typically consists of Input and optional Reference fields. The Reference field does not affect the inference step of evaluation but it is passed verbatim to any evaluation metrics. In Genkit, you can create a dataset through the Dev UI. There are three types of datasets in Genkit: Flow datasets, Model datasets, and Prompt datasets.

Supported evaluators

Genkit supports several evaluators, some built-in, and others provided externally.

Genkit evaluators

Genkit includes a small number of built-in evaluators to help you get started. The Go constant you configure the plugin with and the name the evaluator is registered under are different strings, and both matter: the constant goes in MetricConfig, the registered name goes to --evaluators and appears in the Dev UI.

Go constant	Registered name	Checks that
`evaluators.EvaluatorDeepEqual`	`genkitEval/deep_equal`	The output is deep-equal to the reference.
`evaluators.EvaluatorRegex`	`genkitEval/regex`	The output matches the regular expression in the reference.
`evaluators.EvaluatorJsonata`	`genkitEval/jsonata`	The output matches the JSONata expression in the reference.

All three require a reference: without one the metric errors with reference was not provided.

genkitEval/regex matches strings only. The reference must be a string regular expression, and the output must be a string too. A non-string output, an object or an array, is not serialized and is not matched: the metric scores false with status FAIL. For structured outputs use deep_equal, or write a custom evaluator that marshals the output first.

The Dev UI’s Evaluate page lists every registered evaluator name, including the custom ones you define.

Custom evaluators

You can extend Genkit to support custom evaluation by defining your own evaluator functions. An evaluator can use an LLM as a judge, perform programmatic (heuristic) checks, or call external APIs to assess the quality of a response.

You define a custom evaluator using the genkit.DefineEvaluatorAction function. The callback function for the evaluator can contain any logic you need. Its last parameter is the evaluator’s config: Genkit infers a JSON schema from that type, validates each request against it, and hands you the deserialized value, so an evaluator that takes no options declares it as struct{}.

The data point you receive

req.Input is one ai.Example, the row under evaluation:

type Example struct {
    TestCaseId string   `json:"testCaseId,omitempty"`
    Input      any      `json:"input"`
    Output     any      `json:"output,omitempty"`
    Context    []any    `json:"context,omitempty"`
    Reference  any      `json:"reference,omitempty"`
    TraceIds   []string `json:"traceIds,omitempty"`
}

Input, Output and Reference are any, holding decoded JSON: a map[string]any for an object, a float64 for any number, a string for a string. Context is []any of the same. Reading a structured value therefore takes a type assertion, or a round trip through json.Marshal into your own struct. Reference is the field the built-in deep_equal and regex evaluators compare against; it is passed through verbatim from the dataset and never touched by inference.

The score you return

Each entry in the Evaluation slice is an ai.Score:

type Score struct {
    Id      string         `json:"id,omitempty"`
    Score   any            `json:"score,omitempty"`
    Status  string         `json:"status,omitempty"` // UNKNOWN, FAIL, or PASS
    Error   string         `json:"error,omitempty"`
    Details map[string]any `json:"details,omitempty"`
}

Score is any, so a number, a boolean, or a string are all valid; the built-in evaluators store a boolean. Status is what the Dev UI keys pass/fail highlighting on, so set it with ai.ScoreStatusPass.String(), ai.ScoreStatusFail.String(), or ai.ScoreStatusUnknown.String() rather than a bare string. Details is free-form and displayed alongside the score, which is where the reasoning behind a judgement belongs. Error carries a per-score failure message when one score failed but the others are still usable. Id labels a score, so one evaluator may return several distinct scores in a single Evaluation slice.

Here’s an example of a custom evaluator that uses an LLM to check for “deliciousness”:

import (
    "context"
    "errors"
    "fmt"
    "strings"

    "github.com/firebase/genkit/go/ai"
    "github.com/firebase/genkit/go/core/api"
    "github.com/firebase/genkit/go/genkit"
)

// NewFoodEvaluator creates a custom evaluator for food.
func NewFoodEvaluator(g *genkit.Genkit) ai.Evaluator {
    return genkit.DefineEvaluatorAction(g, api.NewName("custom", "foodEvaluator"),
        &ai.EvaluatorOptions{
            DisplayName: "Food Evaluator",
            Definition:  "Determines if an output is a delicious food item.",
        },
        func(ctx context.Context, req *ai.EvaluatorCallbackRequest, _ struct{}) (*ai.EvaluatorCallbackResponse, error) {
            if req.Input.Output == nil {
                return nil, errors.New("output is required for food evaluation")
            }
            outputStr, ok := req.Input.Output.(string)
            if !ok {
                return nil, errors.New("output must be a string")
            }

            // You can use an LLM as a judge for more complex evaluations.
            resp, err := genkit.Generate(ctx, g,
                ai.WithModelName("googleai/gemini-flash-latest"),
                ai.WithPrompt(fmt.Sprintf(`Is the following food delicious? Respond with "yes", "no", or "maybe". Food: %s`, outputStr)),
            )
            if err != nil {
                return nil, fmt.Errorf("failed to generate evaluation: %w", err)
            }

            // You can also perform any custom logic in the evaluator.
            // if strings.Contains(outputStr, "marmite") {
            //     handleMarmite()
            // }
            // or...
            // score, err := myApi.Evaluate(ctx, &myApi.Request{
            //      Type: "deliciousness",
            //      Value: outputStr,
            // })

            // Turn the judge's verdict into a numeric score and a status the
            // Dev UI can highlight on, and keep its wording as the reasoning.
            verdict := strings.ToLower(strings.TrimSpace(resp.Text()))
            score, statusName := 0.0, ai.ScoreStatusFail
            switch {
            case strings.HasPrefix(verdict, "yes"):
                score, statusName = 1.0, ai.ScoreStatusPass
            case strings.HasPrefix(verdict, "maybe"):
                score, statusName = 0.5, ai.ScoreStatusUnknown
            }

            return &ai.EvaluatorCallbackResponse{
                TestCaseId: req.Input.TestCaseId,
                Evaluation: []ai.Score{{
                    Id:      "deliciousness",
                    Score:   score,
                    Status:  statusName.String(),
                    Details: map[string]any{"reasoning": resp.Text()},
                }},
            }, nil
        },
    )
}

You can then use this custom evaluator just like any other Genkit evaluator. You can use them with your datasets in the Dev UI or with the CLI in the eval:run or eval:flow commands:

genkit eval:flow myFlow --input myDataset.json --evaluators=custom/foodEvaluator

Advanced use

Along with its basic functionality, Genkit also provides advanced support for certain evaluation use cases.

Evaluation comparison

The Developer UI can show several evaluation runs side by side, so you can see what a prompt or model change did to output quality. One run is designated the Baseline, and every other run in the comparison is judged improved or regressed against it.

Comparison has three prerequisites:

The evaluations must come from a dataset source. Runs from a file are not comparable.
Every evaluation compared must be from the same dataset.
For metric highlighting, they must share at least one metric that produces a number or a boolean score.

To compare runs:

Perform at least two evaluation runs on the same dataset, as described in Run evaluation and view results.
In the Developer UI, go to the Datasets page, select the dataset, and open its Evaluations tab.
Open the run you want as the baseline and click + Comparison. A disabled button means no other comparable run exists for this dataset.
Pick another run from the dropdown in the new column. Up to three runs can be compared at once.

To color-code the difference, choose a metric from the Choose a metric to compare menu. Improvements are green and regressions red. By default lower numeric scores and false boolean values count as improvements; tick the checkbox in the menu to reverse that. Highlighting works on numeric and boolean metrics only, and the metric has to be present in every run being compared.

Evaluation using the CLI

Genkit CLI provides a rich API for performing evaluation. This is especially useful in environments where the Dev UI is not available (e.g. in a CI/CD workflow).

Genkit CLI provides 3 main evaluation commands: eval:flow, eval:extractData, and eval:run.

Evaluation `eval:flow` command

The eval:flow command runs inference-based evaluation on an input dataset. This dataset may be provided either as a JSON file or by referencing an existing dataset in your Genkit runtime.

# Referencing an existing dataset
genkit eval:flow qaFlow --input myFactsQaDataset -- go run main.go

# or, using a dataset from a file
genkit eval:flow qaFlow --input testInputs.json -- go run main.go

Here, testInputs.json should be an array of objects containing an input field and an optional reference field, like below:

[
  {
    "input": "What is the French word for Cheese?"
  },
  {
    "input": "What green vegetable looks like cauliflower?",
    "reference": "Broccoli"
  },
  {
    "input": { "query": "Which animals did dogs evolve from?", "locale": "en" },
    "reference": "(?i)wolf|wolves"
  }
]

input is handed to the flow as its input value verbatim, and it can be any JSON type. The strings in the first two rows are only because the sample qaFlow takes a string; a flow whose input is a struct takes an object, as in the third row. Nothing stringifies it on the way in.

If your flow requires auth, you may specify it using the --context argument:

genkit eval:flow qaFlow --input testInputs.json --context '{"auth": {"email_verified": true}}' -- go run main.go

That JSON object becomes the flow’s runtime context. Read it inside the flow with core.FromContext, which returns a core.ActionContext, an alias for map[string]any:

import "github.com/firebase/genkit/go/core"

genkit.DefineFlow(g, "qaFlow", func(ctx context.Context, query string) (string, error) {
    auth, _ := core.FromContext(ctx)["auth"].(map[string]any)
    if verified, _ := auth["email_verified"].(bool); !verified {
        return "", status.PublicErrorf(status.ErrPermissionDenied, "verified email required")
    }
    // ...
})

In production the same map is populated by genkit.WithContextProviders, so the flow reads context identically whether it was invoked by the CLI or over HTTP.

By default, the eval:flow and eval:run commands use all available metrics for evaluation. To run on a subset of the configured evaluators, use the --evaluators flag and provide a comma-separated list of evaluators by name:

genkit eval:flow qaFlow --input testInputs.json --evaluators=genkitEval/regex,genkitEval/jsonata -- go run main.go

You can view the results of your evaluation run in the Dev UI at localhost:4000/evaluate.

`eval:extractData` and `eval:run` commands

To support raw evaluation, Genkit provides tools to extract data from traces and run evaluation metrics on extracted data. This is useful, for example, if you are using a different framework for evaluation or if you are collecting inferences from a different environment to test locally for output quality.

You can batch run your Genkit flow and extract an evaluation dataset from the resultant traces. A raw evaluation dataset is a collection of inputs for evaluation metrics, without running any prior inference.

Run your flow over your test inputs:

genkit flow:batchRun qaFlow testInputs.json -- go run main.go

Extract the evaluation data. --maxRows defaults to 100; the dataset built earlier in this guide has 3 examples:

genkit eval:extractData qaFlow --maxRows 3 --output factsEvalDataset.json -- go run main.go

The exported data has a format different from the dataset format presented earlier. This is because this data is intended to be used with evaluation metrics directly, without any inference step. Here is the syntax of the extracted data.

Array<{
  "testCaseId": string,
  "input": any,
  "output": any,
  "context": any[],
  "traceIds": string[],
}>;

The data extractor automatically locates retrievers and adds the produced docs to the context array. You can run evaluation metrics on this extracted dataset using the eval:run command.

genkit eval:run factsEvalDataset.json -- go run main.go

Like every other command on this page, eval:extractData and eval:run attach to a running runtime, so pass one after --.

By default, eval:run runs against all configured evaluators, and as with eval:flow, results for eval:run appear in the evaluation page of Developer UI, located at localhost:4000/evaluate.

Evaluation in CI

Both eval:flow and eval:run can write the run to disk instead of leaving it in the Dev UI. --output-format accepts json (the default) or csv:

genkit eval:flow qaFlow --input testInputs.json -o results.json --output-format json -- go run main.go

eval:run takes the same pair of flags, spelled --output (it has no -o short form).

The JSON file is an array of results, one per test case, each carrying the extracted input, output and context plus a metrics array whose entries look like this:

{
  "evaluator": "genkitEval/regex",
  "scoreId": "deliciousness",
  "score": true,
  "status": "PASS",
  "rationale": "...",
  "error": null,
  "traceId": "..."
}

Custom extractors

Genkit’s tooling has default logic for pulling the input, output and context fields out of a trace. When you need something else, override it with an extractor. Extractors apply to the eval:extractData and eval:flow commands.

Extraction works off named steps, so give the step you want to read from a name with genkit.Run:

genkit.DefineFlow(g, "qaFlow", func(ctx context.Context, query string) (string, error) {
    factDocs, err := genkit.Retrieve(ctx, g, ai.WithRetriever(factsRetriever), ai.WithTextDocs(query))
    if err != nil {
        return "", err
    }

    // Keep only the facts worth answering from; isSillyFact is your own
    // predicate. Any step you name here can be referenced by an extractor.
    factDocsModified, err := genkit.Run(ctx, "factModified", func() ([]*ai.Document, error) {
        var kept []*ai.Document
        for _, d := range factDocs.Documents {
            if isSillyFact(d.Content[0].Text) {
                kept = append(kept, d)
            }
        }
        return kept, nil
    })
    if err != nil {
        return "", err
    }

    resp, err := genkit.Generate(ctx, g,
        ai.WithPrompt("Answer this question with the given context: %s", query),
        ai.WithDocs(factDocsModified...),
    )
    if err != nil {
        return "", err
    }
    return resp.Text(), nil
})

Extractors are configured in genkit-tools.conf.js at your project root. That file belongs to the Genkit CLI, not to any one SDK, so it is a JavaScript file even in a Go project and it needs no Node dependencies:

cd /path/to/your/genkit/app
touch genkit-tools.conf.js

module.exports = {
  evaluators: [
    {
      actionRef: '/flow/qaFlow',
      extractors: {
        context: { outputOf: 'factModified' },
      },
    },
  ],
};

Run the evaluation again and context is now the output of the factModified step rather than the retriever’s raw result:

genkit eval:flow qaFlow --input testInputs.json -- go run main.go

The config takes:

evaluators: an array of entries, each scoped to one action by actionRef.
extractors: the overrides for that action. The supported keys are input, output and context. A value can be:
- a string, read as a step name, whose output is extracted;
- { inputOf: 'step' } or { outputOf: 'step' }, to pick one side of a step;
- a function taking the trace and returning any value, for anything more involved. The trace schema is in genkit-tools/common/src/types/trace.ts.

The extracted value keeps the type the step produced. If factModified returns an array of documents, the extracted context is an array of documents.

Synthesizing test data using an LLM

A dataset is easier to build if a model drafts it. This flow reads a PDF and turns each chunk into a candidate question. It uses the same two libraries as the RAG guide: github.com/ledongthuc/pdf to read the file and github.com/tmc/langchaingo/textsplitter to chunk it.

go get github.com/tmc/langchaingo/textsplitter
go get github.com/ledongthuc/pdf

type QuestionSet struct {
    Questions []string `json:"questions"`
}

splitter := textsplitter.NewRecursiveCharacter(
    textsplitter.WithChunkSize(2000),
    textsplitter.WithChunkOverlap(100),
)

genkit.DefineFlow(g, "synthesizeQuestions", func(ctx context.Context, path string) (*QuestionSet, error) {
    // readPDF extracts the file's text; see the RAG guide for its body.
    text, err := genkit.Run(ctx, "extract-text", func() (string, error) {
        return readPDF(path)
    })
    if err != nil {
        return nil, err
    }

    chunks, err := genkit.Run(ctx, "chunk-it", func() ([]string, error) {
        return splitter.SplitText(text)
    })
    if err != nil {
        return nil, err
    }

    out := &QuestionSet{}
    for _, chunk := range chunks {
        resp, err := genkit.Generate(ctx, g,
            ai.WithPrompt("Generate one question about the following text: %s", chunk),
        )
        if err != nil {
            return nil, err
        }
        out.Questions = append(out.Questions, resp.Text())
    }
    return out, nil
})

Run it and write the questions to a file:

genkit flow:run synthesizeQuestions '"my_input.pdf"' --output synthesizedQuestions.json -- go run main.go

Reshape that file into the [{"input": ...}] dataset format before passing it to eval:flow, and read the questions before you trust them: a model drafting its own test set will happily write questions the source text does not answer.

Next steps

Learn about creating flows to build AI workflows that can be evaluated
Explore retrieval-augmented generation (RAG) for building knowledge-based systems that benefit from evaluation
See tool calling for creating AI agents that can be tested with evaluation metrics
Check out the developer tools documentation for more information about the Genkit Developer UI