Get started with Genkit Monitoring

This quickstart guide describes how to set up Genkit Monitoring for your deployed Genkit features, so that you can collect and view real-time telemetry data. With Genkit Monitoring, you get visibility into how your Genkit features are performing in production.

Key capabilities of Genkit Monitoring include:

• Viewing quantitative metrics like Genkit feature latency, errors, and token usage.
• Inspecting traces to see your Genkit’s feature steps, inputs, and outputs, to help with debugging and quality improvement.
• Exporting production traces to run evals within Genkit.

Setting up Genkit Monitoring requires completing tasks in both your codebase and on the Google Cloud Console.

Before you begin

1. If you haven’t already, create a Firebase project.

   In the Firebase console, click Add a project, then follow the on-screen instructions. You can create a new project or add Firebase services to an already-existing Google Cloud project.

2. Ensure your project is on the Blaze pricing plan.

   Genkit Monitoring relies on telemetry data written to Google Cloud Logging, Metrics, and Trace, which are paid services. View the Google Cloud Observability pricing page for pricing details and to learn about free-of-charge tier limits.

3. Write a Genkit feature by following the Get Started Guide, and prepare your code for deployment by using one of the following guides:

   Deploy your flows with Cloud Run or to any platform that runs a Go binary.

Step 1. Add the Firebase plugin

Add the firebase plugin to your module:

go get github.com/firebase/genkit/go/plugins/firebase

The Firebase and Google Cloud telemetry plugins are the same code

firebase.EnableFirebaseTelemetry copies its options into a googlecloud.GoogleCloudTelemetryOptions and calls googlecloud.EnableGoogleCloudTelemetry. The option structs are field for field identical. The only behavioral difference is project-ID resolution: the Firebase entry point checks FIREBASE_PROJECT_ID before GOOGLE_CLOUD_PROJECT and GCLOUD_PROJECT.

Use firebase.EnableFirebaseTelemetry if you want the Firebase Genkit Monitoring dashboard, and googlecloud.EnableGoogleCloudTelemetry otherwise. Do not call both.

Environment-based configuration

There is no environment variable that turns telemetry on. The Go plugin has no equivalent of ENABLE_FIREBASE_MONITORING, so you always call firebase.EnableFirebaseTelemetry in code.

Two environment variables still affect what happens after you call it:

• GENKIT_ENV: when set to dev, nothing is exported unless you also set ForceDevExport: true.
• FIREBASE_PROJECT_ID, then GOOGLE_CLOUD_PROJECT, then GCLOUD_PROJECT: checked in that order to resolve the destination project when you leave ProjectID empty.

Programmatic configuration

Call firebase.EnableFirebaseTelemetry before genkit.Init. It returns nothing, so there is no error to check. Pass nil for the defaults, or a *firebase.FirebaseTelemetryOptions to tweak settings like the metric export interval.

package main

import (
    "context"
    "fmt"
    "log"
    "net/http"
    "os"

    "github.com/firebase/genkit/go/ai"
    "github.com/firebase/genkit/go/genkit"
    "github.com/firebase/genkit/go/plugins/firebase"
    "github.com/firebase/genkit/go/plugins/googlegenai"
    "github.com/firebase/genkit/go/plugins/server"
)

func main() {
    ctx := context.Background()

    // Enable telemetry with default options, before genkit.Init.
    firebase.EnableFirebaseTelemetry(nil)

    g := genkit.Init(ctx,
        genkit.WithPlugins(&googlegenai.GoogleAI{}),
        genkit.WithDefaultModel("googleai/gemini-flash-latest"),
    )

    flow := genkit.DefineFlow(g, "jokesFlow", func(ctx context.Context, topic string) (string, error) {
        resp, err := genkit.Generate(ctx, g, ai.WithPrompt("Tell a short joke about %s.", topic))
        if err != nil {
            return "", fmt.Errorf("failed to generate joke: %w", err)
        }
        return resp.Text(), nil
    })

    mux := http.NewServeMux()
    mux.HandleFunc("POST /jokesFlow", genkit.Handler(flow))

    port := os.Getenv("PORT")
    if port == "" {
        port = "8080"
    }
    log.Fatal(server.Start(ctx, "0.0.0.0:"+port, mux))
}

The full option list is on Advanced configuration.

Step 2. Enable the required APIs

Make sure that the following APIs are enabled for your Google Cloud project:

• Cloud Logging API
• Cloud Trace API
• Cloud Monitoring API

These APIs should be listed in the API dashboard for your project.

Step 3. Set up permissions

The Firebase plugin needs to use a service account to authenticate with Google Cloud Logging, Metrics, and Trace services.

Grant the following roles to whichever service account is configured to run your code within the Google Cloud IAM Console. For Cloud Functions for Firebase and Cloud Run, that’s typically the default compute service account.

• Monitoring Metric Writer (roles/monitoring.metricWriter)
• Cloud Trace Agent (roles/cloudtrace.agent)
• Logs Writer (roles/logging.logWriter)

Step 4. (Optional) Test your configuration locally

Before deploying, you can run your Genkit code locally to confirm that telemetry data is being collected, and is viewable in the Genkit Monitoring dashboard.

1. In your Genkit code, set ForceDevExport to true to send telemetry from your local environment. Lower the export interval at the same time so you do not wait five minutes for the first metric:

   interval := 5000 // Milliseconds. Google Cloud rejects anything below 5000.
   
   firebase.EnableFirebaseTelemetry(&firebase.FirebaseTelemetryOptions{
       ForceDevExport:             true,
       MetricExportIntervalMillis: &interval,
   })

   MetricExportIntervalMillis is a *int, so it needs an addressable variable. Leaving it nil means 5000 in dev and 300000 in production.

2. Use your service account to authenticate and test your configuration.

   Tip

   In order to impersonate the service account, you will need to have the roles/iam.serviceAccountTokenCreator IAM role applied to your user account.

   With the Google Cloud CLI tool, authenticate using the service account:

gcloud auth application-default login --impersonate-service-account SERVICE_ACCT_EMAIL

3. Run and invoke your Genkit feature, and then view metrics on the Genkit Monitoring dashboard. Allow for up to 5 minutes to collect the first metric. You can reduce this delay by lowering the metric export interval in the telemetry configuration.

4. If metrics are not appearing in the Genkit Monitoring dashboard, view the Troubleshooting guide for steps to debug.

Step 5. Re-build and deploy code

Re-build, deploy, and invoke your Genkit feature to start collecting data. After Genkit Monitoring receives your metrics, you can view them by visiting the Genkit Monitoring dashboard

Note

It may take up to 5 minutes to collect the first metric (based on the default MetricExportIntervalMillis setting in the telemetry configuration).

Correlating a user report to a trace

A support ticket is only useful if you can find the request it describes. Read the current trace ID inside the flow and hand it back to the caller.

package main

import (
    "context"
    "fmt"

    "github.com/firebase/genkit/go/ai"
    "github.com/firebase/genkit/go/genkit"
    "go.opentelemetry.io/otel/trace"
)

type jokeResponse struct {
    Joke    string `json:"joke"`
    TraceID string `json:"traceId"`
}

func defineJokesFlow(g *genkit.Genkit) {
    genkit.DefineFlow(g, "jokesFlow", func(ctx context.Context, topic string) (jokeResponse, error) {
        traceID := trace.SpanContextFromContext(ctx).TraceID().String()

        resp, err := genkit.Generate(ctx, g, ai.WithPrompt("Tell a short joke about %s.", topic))
        if err != nil {
            // Return the ID on the error path too. That is the path users report.
            return jokeResponse{TraceID: traceID}, fmt.Errorf("failed to generate joke: %w", err)
        }
        return jokeResponse{Joke: resp.Text(), TraceID: traceID}, nil
    })
}

An X-Trace-Id response header works just as well if you do not want to change the response body.

With the ID in hand:

• Search for it directly in Cloud Trace, or in the Genkit Monitoring trace viewer.
• Paste projects/<project-id>/traces/<trace-id> into Cloud Logging to pull every log line for that request. Genkit writes the trace field of each log entry in exactly that format. See Telemetry collection.

To capture the ID from outside the flow body, use tracing.WithTelemetryCallback(ctx, func(traceID, spanID string) { ... }) from github.com/firebase/genkit/go/core/tracing and pass the resulting context into the flow.

Export to any OTLP backend

Firebase and Google Cloud are not the only destinations. core/tracing exports the SDK tracer provider as application API, so you can attach any OpenTelemetry span exporter to it: Datadog, Honeycomb, Grafana, or a self-hosted collector.

go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc

package main

import (
    "context"
    "log"

    "github.com/firebase/genkit/go/core/tracing"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

// registerOTLP attaches an OTLP exporter to Genkit's tracer provider and
// returns a shutdown function that flushes buffered spans.
func registerOTLP(ctx context.Context) func(context.Context) error {
    // Reads OTEL_EXPORTER_OTLP_ENDPOINT and the usual OTEL_* variables.
    exp, err := otlptracegrpc.New(ctx)
    if err != nil {
        log.Fatalf("failed to build OTLP exporter: %v", err)
    }

    tp := tracing.TracerProvider()
    tp.RegisterSpanProcessor(sdktrace.NewBatchSpanProcessor(exp))
    return tp.Shutdown
}

Call registerOTLP before genkit.Init and defer the returned shutdown function. For metrics, build an OTLP metric.Exporter and install it with otel.SetMeterProvider; Genkit records its counters and histograms through the global meter provider.

Order matters if you also enable Firebase or Google Cloud telemetry

EnableFirebaseTelemetry and EnableGoogleCloudTelemetry build a fresh TracerProvider and install it as the global one, which discards any span processor registered earlier. To send spans to both destinations, call registerOTLP after the telemetry plugin, not before.