Local observability and metrics

Genkit records a trace of every flow, action, and step it runs. During development, the Genkit Developer UI shows these traces so you can debug a flow step by step. In production, the Firebase plugin sends traces and metrics to Genkit Monitoring in the Firebase console, or you can export them to any OpenTelemetry backend.

Tracing & metrics

Genkit traces every run without explicit configuration. Under genkit start, Genkit sends its spans straight to the Developer UI’s telemetry server, with no OpenTelemetry SDK on that path, so you can analyze a flow step by step with its inputs, outputs, and timing. The Developer UI shows Genkit spans only. Spans that other libraries create through OpenTelemetry, such as otelhttp, do not appear there.

Everywhere else, Genkit reports its spans through pluggable instrumentation. The default provider creates OpenTelemetry spans that carry genkit:* attributes, on the TracerProvider your application registers. Genkit does not install a TracerProvider itself, so nothing leaves the process until you register one, directly or through a plugin. See Export to any OTel backend.

Metrics travel a separate path. Genkit records action metrics, and the Firebase and Google Cloud plugins add metric families for feature, flow, generate, and tool activity, including genkit/flow/latency and the token counters genkit/ai/generate/input/tokens and genkit/ai/generate/output/tokens. The full table is in Telemetry collection. The Developer UI does not display metrics: they only leave the process once an exporting plugin such as the Firebase plugin is installed. The OpenTelemetry GenAI instrumentation records the spec’s token usage and duration metrics instead.

Log and export events

Genkit Go logs through github.com/firebase/genkit/go/core/logger, a thin layer over the standard library’s log/slog. Log with the package-level functions, passing the context first:

import "github.com/firebase/genkit/go/core/logger"

genkit.DefineFlow(g, "summarize", func(ctx context.Context, input string) (string, error) {
    logger.Info(ctx, "summarizing", "size", len(input))

    resp, err := genkit.Generate(ctx, g, ai.WithPrompt(input))
    if err != nil {
        logger.Warn(ctx, "generation failed, returning the input unchanged", "error", err)
        return input, nil
    }

    logger.Debug(ctx, "summary ready", "outputSize", len(resp.Text()))
    return resp.Text(), nil
})

Passing the context is what ties a record to its surroundings. The context carries the trace span that is active at that moment, so the record is correlated with the step that produced it, and it carries any attributes bound to the context’s logger. The basic-errors sample logs through these helpers as it classifies and recovers from failures.

logger.FromContext(ctx) is still supported and still correlates: it returns the context’s logger bound to that context, so records logged through its plain methods keep the span. Reach for it when you want to bind attributes once and have them follow everything logged downstream:

ctx = logger.WithContext(ctx, logger.FromContext(ctx).With("requestId", requestID))

// Everything logged below carries requestId, with no extra plumbing.
logger.Info(ctx, "handling request")

Plain slog works too, with one caveat: only the *Context methods carry the span. slog.InfoContext(ctx, ...) is correlated, slog.Info(...) is not.

View logs in the Developer UI

Running your app under the CLI is enough to get logs into the Developer UI:

genkit start -- go run .

No code change and no configuration is needed. The CLI sets the environment the runtime looks for, genkit.Init installs the export handler, and records stream to the Developer UI, which lists them against the span that emitted them in the trace viewer. Starting the Developer UI separately with genkit ui:start also works, since the CLI hands the telemetry server’s address to the running app.

The Developer UI receives every record at debug level and above, independent of the console level, so the terminal can stay quiet while the full debug narrative lands in the trace viewer. Genkit’s own records travel the same channel: span start and finish, the resolved generate request, each model turn with its finish reason and token counts, tool batches, and each middleware hook with its duration and whether it short-circuited.

A record logged without a context is still exported, but it carries no span, so it never appears against a step.

Do not call slog.SetDefault after genkit.Init

slog.SetDefault replaces the composed handler wholesale and silently disconnects the Developer UI sink. Use logger.AddHandler to add a destination alongside the ones already installed, or logger.SetDefaultHandler to replace the base console handler while keeping them.

// Mirror every record to a file without disturbing console output or
// Developer UI streaming.
f, err := os.Create("genkit.log")
if err != nil {
    log.Fatal(err)
}
logger.AddHandler(slog.NewJSONHandler(f, &slog.HandlerOptions{Level: slog.LevelDebug}))

Set the console level

The console starts at info. To see Genkit’s per-request detail in the terminal as well, lower the level:

func main() {
    ctx := context.Background()

    // Console only. In dev the Developer UI already receives debug and above,
    // whatever this is set to.
    logger.SetLevel(slog.LevelDebug)

    g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{}))
    // ... define flows, serve ...
    _ = g
}

Order relative to genkit.Init does not matter. SetLevel is not slog.SetDefault: it recomposes Genkit’s console handler with every sink already registered rather than replacing the default outright, and genkit.Init recomposes again when it registers the Developer UI sink. The caution above does not apply to it.

If your application configured its own default slog handler, Genkit leaves it alone and warns instead, so set that handler’s level rather than calling SetLevel.

Environment variables

Variable	Accepted values	Default	Effect
GENKIT_LOG_LEVEL	debug, info, warn, error, case-insensitive, with an optional offset such as error+2	info	Minimum level for the console. It never changes what the Developer UI receives. A value it cannot parse, such as warning, verbose, or a number, is ignored with a warning, and so is the variable as a whole if your application installed its own default slog handler.
GENKIT_OTEL_ENABLE_LOGS	any value that parses as false (false, 0, f) turns export off	unset, so export is on	An opt-out. You do not have to set it to see logs. Unset, true, and anything that does not parse as false all leave export on.
GENKIT_ENV	dev installs the Developer UI log sink	prod	genkit start sets dev. Outside the dev environment, nothing is exported unless the reflection API is on.
GENKIT_REFLECTION_ENABLED	true, false	unset, so it follows GENKIT_ENV	true runs the reflection API and installs the log sink in any environment. false turns the reflection API off, even under dev. See Developer tools.
GENKIT_TELEMETRY_SERVER	a base URL, for example http://localhost:4033	empty	Where records are posted. genkit start sets it. When it is empty, the CLI supplies the address at runtime instead.

Coming from the JS SDK

GENKIT_OTEL_ENABLE_LOGS reads the opposite way in Go: JS treats it as an opt-in and stays silent until you set it, Go treats it as an opt-out and exports by default.

What survives the trip

Attribute values reach the Developer UI as strings, integers, or booleans. Everything else is rendered to text on the way: a float64 arrives as its decimal string, a time.Duration as 412ms, an error as its Error() text, and anything else as JSON. slog groups flatten into dotted keys.

Export never blocks your code. Records are batched and posted in the background, and dropped rather than queued when the buffer fills, with one warning on stderr.

Trace your own work

Add a step and attach metadata

A step is a named span inside a flow. genkit.Run creates one:

hits, err := genkit.Run(ctx, "retrieve", func() (int, error) {
    return len(docs), nil
})

genkit.Run does not pass the step’s context into fn, so anything inside it that traces its own work reports against the enclosing flow instead of the step. Use genkit.RunWithContext when the work inside takes a context:

summary, err := genkit.RunWithContext(ctx, "summarize-chunk",
    func(ctx context.Context) (string, error) {
        // The generate and model spans nest under "summarize-chunk".
        return genkit.GenerateText(ctx, g, ai.WithPrompt("Summarize: %s", chunk))
    })

To attach a measurement to the active span, use tracing.SetSpanMetadata. It works with every instrumentation provider, so the values appear in the Developer UI trace viewer and in your OpenTelemetry backend alike:

import (
    "strconv"

    "github.com/firebase/genkit/go/core/tracing"
)

tracing.SetSpanMetadata(ctx, map[string]string{
    "retrieval.hitRate": strconv.FormatFloat(rate, 'f', 2, 64),
    "retrieval.docs":    strconv.Itoa(len(docs)),
})

Each entry is stored as a genkit:metadata:<key> attribute. Values are strings, and the call is safe from any goroutine. Outside a span it does nothing.

Get the current trace id

Inside a flow, a step, or a tool, tracing.SpanTraceInfo(ctx) returns the ids of the span that is active:

import (
    "github.com/firebase/genkit/go/core/logger"
    "github.com/firebase/genkit/go/core/tracing"
)

info := tracing.SpanTraceInfo(ctx)
logger.Error(ctx, "request failed",
    "traceId", info.TraceID,
    "spanId", info.SpanID,
    "path", tracing.SpanPath(ctx))

Return that trace id to the caller in a response header and an operator can go straight to genkit trace:get <traceId>. tracing.SpanPath(ctx) gives the step’s path within the trace, which is what identifies a step in the Developer UI.

The ids are always set inside a span. Under genkit start they are the Developer UI’s ids. Elsewhere they come from your OpenTelemetry SDK when you have registered one, and Genkit assigns its own when no provider reports any. Outside a span, SpanTraceInfo returns empty ids.

Export to any OTel backend

Genkit creates its spans on the global OpenTelemetry TracerProvider. Register an SDK provider with your exporter, and Genkit’s traces go to it:

import (
    "github.com/firebase/genkit/go/genkit"
    "go.opentelemetry.io/otel"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func main() {
    ctx := context.Background()

    tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(yourExporter))
    otel.SetTracerProvider(tp)
    defer tp.Shutdown(context.Background()) // Flushes buffered spans.

    g := genkit.Init(ctx)
    // ... define flows, serve ...
    _ = g
}

yourExporter is any sdktrace.SpanExporter: OTLP, Jaeger, or one you wrote. Genkit looks up the global provider for each span, so a provider registered after genkit.Init is picked up too. Register it first so no early span is lost. Under genkit start, the Developer UI keeps receiving every trace, because its feed does not go through OpenTelemetry.

To keep Genkit’s spans on a dedicated provider, for example one with its own sampler, without changing the global one, pass it to the default provider:

tracing.SetInstrumentation(&tracing.OTelInstrumentation{TracerProvider: tp})

Outside the dev environment, inline base64 media in the genkit:input, genkit:init, and genkit:output attributes keeps only its data URL header, such as data:image/png;base64,[redacted], so a trace records the media type without a copy of the bytes. Under GENKIT_ENV=dev, nothing is redacted, because the Developer UI renders media from these attributes.

For telemetry that follows the OpenTelemetry GenAI semantic conventions instead of the genkit:* attributes, see OpenTelemetry GenAI semantic conventions. If you are packaging an exporter for other people to install rather than wiring it into one application, do it as a plugin instead; see Writing Genkit plugins.

Choose the instrumentation providers

tracing.SetInstrumentation sets the providers that receive every Genkit span, in order, and replaces the default OpenTelemetry provider. List &tracing.OTelInstrumentation{} to keep it next to your own:

tracing.SetInstrumentation(&tracing.OTelInstrumentation{}, myProvider)

Call it once, at startup. The setting is process-wide, because Genkit actions can run without a Genkit instance (for example, a model taken straight from a plugin) and are traced too. Calling it with no providers restores the default. Under genkit start, the Developer UI’s provider always runs first, whatever you set.

tracing.NewDirectTelemetryInstrumentation(client) is the provider that feeds the Developer UI, built over any tracing.TelemetryClient. Use it to send Genkit spans to a telemetry server of your own without an OpenTelemetry SDK, or to capture them in memory with tracing.NewTestOnlyTelemetryClient() in a test. It saves each span when the span ends.

Write your own provider

A provider implements tracing.Instrumentation. Genkit calls StartSpan on each provider when a span starts, runs the operation once, and then calls End on each returned tracing.Span, so a provider observes a run but cannot skip, repeat, or change it:

import (
    "context"
    "log"

    "github.com/firebase/genkit/go/core/tracing"
)

type logInstrumentation struct{}

func (logInstrumentation) StartSpan(ctx context.Context, info *tracing.SpanInfo) (context.Context, tracing.Span) {
    log.Printf("-> %s (%s) at %s", info.Name(), info.Subtype(), info.Path())
    return ctx, &logSpan{name: info.Name()}
}

type logSpan struct{ name string }

// TraceInfo returns empty ids, so Genkit assigns its own.
func (*logSpan) TraceInfo() tracing.TraceInfo { return tracing.TraceInfo{} }

func (*logSpan) SetMetadata(md map[string]string) {}

func (s *logSpan) End(res *tracing.SpanResult) {
    if err := res.Err(); err != nil {
        log.Printf("<- %s failed: %v (source: %t)", s.name, err, res.IsFailureSource())
        return
    }
    log.Printf("<- %s", s.name)
}

The contract:

• tracing.SpanInfo describes the span: Name, Type, Subtype, Path, IsRoot, Input, Init, Labels, and Metadata.
• tracing.SpanResult carries the outcome: Output, Err, and IsFailureSource. IsFailureSource is true only on the span where an error originated, not on the spans that passed it up, so a backend that counts failures counts each one once.
• End is called exactly once, in reverse start order across providers, and also when the operation panics.
• SetMetadata receives what tracing.SetSpanMetadata records, and can be called from any goroutine.
• TraceInfo reports the provider’s own trace and span ids, or empty ids if it has none. Return a nil Span from StartSpan to ignore a span.

Observe the spans of one run

tracing.WithInstrumentation adds providers to a context instead of the whole process. Every span started under that context reaches them, nested runs and subagents included. They run after the process-wide providers, and a nested scope cannot remove them. Middleware uses this to react to what its own run does, with no setup by the application.

aix.Observer is a provider that reports each model call, typed:

Preview

Observer is in the experimental ai/exp package and can change in a minor release.

import (
    "context"
    "log"

    "github.com/firebase/genkit/go/ai"
    aix "github.com/firebase/genkit/go/ai/exp"
    "github.com/firebase/genkit/go/core/tracing"
    "github.com/firebase/genkit/go/genkit"
)

ctx = tracing.WithInstrumentation(ctx, aix.Observer{
    ModelDone: func(ctx context.Context, call *aix.ModelCall) {
        if u := call.Response.Usage; u != nil {
            log.Printf("%s used %d tokens", call.Model, u.TotalTokens)
        }
    },
})
text, err := genkit.GenerateText(ctx, g, ai.WithPrompt("Tell me a joke."))

ModelDone runs after every model call under the context, failed calls included. call.Response is never nil: a call that failed without a response gets the failed record that Generate returns, and call.Err holds the error. A response that a middleware hook supplies without calling the model, such as a cache hit, is not reported. The Budget middleware is built on this; see Middleware.

Shutdown and flushing

The Genkit instance itself needs no shutdown. There is no Close or Shutdown method; background work is released by cancelling the context you passed to genkit.Init.

Spans buffered by your OpenTelemetry SDK are flushed by the provider you registered: defer tp.Shutdown, as in Export to any OTel backend, or call tp.ForceFlush if the process keeps running afterwards. Spans sent to the Developer UI are saved when each span ends, so a completed span is never lost. Log export has no flush hook, so a short-lived go run . can still lose its last few log records; sleep briefly before exit if that matters.

Production observability

The Genkit Monitoring dashboard helps you understand the overall health of your Genkit features. It is also useful for debugging stability and content issues that may indicate problems with your LLM prompts and/or Genkit Flows. See the Getting Started guide for more details.