<SYSTEM>Observing AI Workflows: Guidance about Genkit's various observability features and how to use them.</SYSTEM>

# Advanced Configuration

> This guide covers advanced configuration options for Genkit's Firebase telemetry plugin, including fine-tuning telemetry collection, export settings, and disabling specific data types.

This guide focuses on advanced configuration options for deployed features using the Firebase telemetry plugin. Detailed descriptions of each configuration option can be found in our [JS API reference documentation](https://js.api.genkit.dev/interfaces/_genkit-ai_google-cloud.GcpTelemetryConfigOptions.html).

This documentation will describe how to fine-tune which telemetry is collected, how often, and from what environments.

## Default Configuration

[Section titled “Default Configuration”](#default-configuration)

The Firebase telemetry plugin provides default options, out of the box, to get you up and running quickly. These are the provided defaults:

```typescript
{
  autoInstrumentation: true,
  autoInstrumentationConfig: {
    '@opentelemetry/instrumentation-dns': { enabled: false },
  }
  disableMetrics: false,
  disableTraces: false,
  disableLoggingInputAndOutput: false,
  forceDevExport: false,
  // 5 minutes
  metricExportIntervalMillis: 300_000,
  // 5 minutes
  metricExportTimeoutMillis: 300_000,
  // See https://js.api.genkit.dev/interfaces/_genkit-ai_google-cloud.GcpTelemetryConfigOptions.html#sampler
  sampler: AlwaysOnSampler()
}
```

## Export local telemetry

[Section titled “Export local telemetry”](#export-local-telemetry)

To export telemetry when running locally set the `forceDevExport` option to `true`.

```typescript
import { enableFirebaseTelemetry } from '@genkit-ai/firebase';


enableFirebaseTelemetry({ forceDevExport: true });
```

During development and testing, you can decrease latency by adjusting the export interval and timeout.

Note: Shipping to production with a frequent export interval may increase the cost for exported telemetry.

```typescript
import { enableFirebaseTelemetry } from '@genkit-ai/firebase';


enableFirebaseTelemetry({
  forceDevExport: true,
  metricExportIntervalMillis: 10_000, // 10 seconds
  metricExportTimeoutMillis: 10_000, // 10 seconds
});
```

## Adjust auto instrumentation

[Section titled “Adjust auto instrumentation”](#adjust-auto-instrumentation)

The Firebase telemetry plugin will automatically collect traces and metrics for popular frameworks using OpenTelemetry [zero-code instrumentation](https://opentelemetry.io/docs/zero-code/js/).

A full list of available instrumentations can be found in the [auto-instrumentations-node](https://github.com/open-telemetry/opentelemetry-js-contrib/blob/main/metapackages/auto-instrumentations-node/README.md#supported-instrumentations) documentation.

To selectively disable or enable instrumentations that are eligible for auto instrumentation, update the `autoInstrumentationConfig` field:

```typescript
import { enableFirebaseTelemetry } from '@genkit-ai/firebase';


enableFirebaseTelemetry({
  autoInstrumentationConfig: {
    '@opentelemetry/instrumentation-fs': { enabled: false },
    '@opentelemetry/instrumentation-dns': { enabled: false },
    '@opentelemetry/instrumentation-net': { enabled: false },
  },
});
```

## Disable telemetry

[Section titled “Disable telemetry”](#disable-telemetry)

Genkit Monitoring leverages a combination of logging, tracing, and metrics to capture a holistic view of your Genkit interactions, however, you can also disable each of these elements independently if needed.

### Disable input and output logging

[Section titled “Disable input and output logging”](#disable-input-and-output-logging)

By default, the Firebase telemetry plugin will capture inputs and outputs for each Genkit feature or step.

To help you control how customer data is stored, you can disable the logging of input and output by adding the following to your configuration:

```typescript
import { enableFirebaseTelemetry } from '@genkit-ai/firebase';


enableFirebaseTelemetry({
  disableLoggingInputAndOutput: true,
});
```

With this option set, input and output attributes will be redacted in the Genkit Monitoring trace viewer and will be missing from Google Cloud logging.

### Disable metrics

[Section titled “Disable metrics”](#disable-metrics)

To disable metrics collection, add the following to your configuration:

```typescript
import { enableFirebaseTelemetry } from '@genkit-ai/firebase';


enableFirebaseTelemetry({
  disableMetrics: true,
});
```

With this option set, you will no longer see stability metrics in the Genkit Monitoring dashboard and will be missing from Google Cloud Metrics.

### Disable traces

[Section titled “Disable traces”](#disable-traces)

To disable trace collection, add the following to your configuration:

```typescript
import { enableFirebaseTelemetry } from '@genkit-ai/firebase';


enableFirebaseTelemetry({
  disableTraces: true,
});
```

With this option set, you will no longer see traces in the Genkit Monitoring feature page, have access to the trace viewer, or see traces present in Google Cloud Tracing.

# Authentication and authorization

> Learn how to authenticate and authorize the Firebase telemetry plugin for Genkit, covering API enablement, user authentication, and deployment to Google Cloud or other environments.

The Firebase telemetry plugin requires a Google Cloud or Firebase project ID and application credentials.

If you don’t have a Google Cloud project and account, you can set one up in the [Firebase Console](https://console.firebase.google.com/) or in the [Google Cloud Console](https://cloud.google.com). All Firebase project IDs are Google Cloud project IDs.

## Enable APIs

[Section titled “Enable APIs”](#enable-apis)

Prior to adding the plugin, make sure the following APIs are enabled for your project:

* [Cloud Logging API](https://console.cloud.google.com/apis/library/logging.googleapis.com)
* [Cloud Trace API](https://console.cloud.google.com/apis/library/cloudtrace.googleapis.com)
* [Cloud Monitoring API](https://console.cloud.google.com/apis/library/monitoring.googleapis.com)

These APIs should be listed in the [API dashboard](https://console.cloud.google.com/apis/dashboard) for your project. Click to learn more about how to [enable and disable APIs](https://support.google.com/googleapi/answer/6158841).

## User Authentication

[Section titled “User Authentication”](#user-authentication)

To export telemetry from your local development environment to Genkit Monitoring, you will need to authenticate yourself with Google Cloud.

The easiest way to authenticate as yourself is using the gcloud CLI, which will automatically make your credentials available to the framework through [Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials).

If you don’t have the gcloud CLI installed, first follow the [installation instructions](https://cloud.google.com/sdk/docs/install#installation_instructions).

1. Authenticate using the `gcloud` CLI:

   ```bash
   gcloud auth application-default login
   ```

2. Set your project ID

   ```bash
   gcloud config set project PROJECT_ID
   ```

## Deploy to Google Cloud

[Section titled “Deploy to Google Cloud”](#deploy-to-google-cloud)

If deploying your code to a Google Cloud or Firebase environment (Cloud Functions, Cloud Run, App Hosting, etc), the project ID and credentials will be discovered automatically with [Application Default Credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc).

You will need to apply the following roles to the service account that is running your code (i.e. ‘attached service account’) using the [IAM Console](https://console.cloud.google.com/iam-admin/iam):

* `roles/monitoring.metricWriter`
* `roles/cloudtrace.agent`
* `roles/logging.logWriter`

Not sure which service account is the right one? See the [Find or create your service account](#find-or-create-your-service-account) section.

## Deploy outside of Google Cloud (with ADC)

[Section titled “Deploy outside of Google Cloud (with ADC)”](#deploy-outside-of-google-cloud-with-adc)

If possible, use [Application Default Credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc) to make credentials available to the plugin.

Typically this involves generating a service account key and deploying those credentials to your production environment.

1. Follow the instructions to set up a [service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating).

2. Ensure the service account has the following roles:

   * `roles/monitoring.metricWriter`
   * `roles/cloudtrace.agent`
   * `roles/logging.logWriter`

3. Deploy the credential file to production (**do not** check into source code)

4. Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable as the path to the credential file.

   ```bash
   GOOGLE_APPLICATION_CREDENTIALS = "path/to/your/key/file"
   ```

Not sure which service account is the right one? See the [Find or create your service account](#find-or-create-your-service-account) section.

## Deploy outside of Google Cloud (without ADC)

[Section titled “Deploy outside of Google Cloud (without ADC)”](#deploy-outside-of-google-cloud-without-adc)

In some serverless environments, you may not be able to deploy a credential file.

1. Follow the instructions to set up a [service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating).

2. Ensure the service account has the following roles:

   * `roles/monitoring.metricWriter`
   * `roles/cloudtrace.agent`
   * `roles/logging.logWriter`

3. Download the credential file.

4. Assign the contents of the credential file to the `GCLOUD_SERVICE_ACCOUNT_CREDS` environment variable as follows:

```bash
GCLOUD_SERVICE_ACCOUNT_CREDS='{
  "type": "service_account",
  "project_id": "your-project-id",
  "private_key_id": "your-private-key-id",
  "private_key": "your-private-key",
  "client_email": "your-client-email",
  "client_id": "your-client-id",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "your-cert-url"
}'
```

Not sure which service account is the right one? See the [Find or create your service account](#find-or-create-your-service-account) section.

## Find or create your service account

[Section titled “Find or create your service account”](#find-or-create-your-service-account)

To find the appropriate service account:

1. Navigate to the [service accounts page](https://console.cloud.google.com/iam-admin/serviceaccounts) in the Google Cloud Console
2. Select your project
3. Find the appropriate service account. Common default service accounts are as follows:

* Firebase functions & Cloud Run

  `PROJECT ID-compute@developer.gserviceaccount.com`

* App Engine

  `PROJECT ID@appspot.gserviceaccount.com`

* App Hosting

  `firebase-app-hosting-compute@PROJECT ID.iam.gserviceaccount.com`

If you are deploying outside of the Google ecosystem or don’t want to use a default service account, you can [create a service account](https://cloud.google.com/iam/docs/service-accounts-create#creating) in the Google Cloud console.

# Get started with Genkit Monitoring

> This quickstart guide explains how to set up Genkit Monitoring for your deployed Genkit features to collect and view real-time telemetry data, including metrics, traces, and production trace exports for evaluations.

This quickstart guide describes how to set up Genkit Monitoring for your deployed Genkit features, so that you can collect and view real-time telemetry data. With Genkit Monitoring, you get visibility into how your Genkit features are performing in production.

Key capabilities of Genkit Monitoring include:

* Viewing quantitative metrics like Genkit feature latency, errors, and token usage.
* Inspecting traces to see your Genkit’s feature steps, inputs, and outputs, to help with debugging and quality improvement.
* Exporting production traces to run evals within Genkit.

Setting up Genkit Monitoring requires completing tasks in both your codebase and on the Google Cloud Console.

## Before you begin

[Section titled “Before you begin”](#before-you-begin)

1. If you haven’t already, create a Firebase project.

   In the [Firebase console](https://console.firebase.google.com), click **Add a project**, then follow the on-screen instructions. You can create a new project or add Firebase services to an already-existing Google Cloud project.

2. Ensure your project is on the [Blaze pricing plan](https://firebase.google.com/pricing).

   Genkit Monitoring relies on telemetry data written to Google Cloud Logging, Metrics, and Trace, which are paid services. View the [Google Cloud Observability pricing](https://cloud.google.com/stackdriver/pricing) page for pricing details and to learn about free-of-charge tier limits.

3. Write a Genkit feature by following the [Get Started Guide](/docs/get-started), and prepare your code for deployment by using one of the following guides:

   a. [Deploy flows using Cloud Functions for Firebase](/docs/firebase) b. [Deploy flows using Cloud Run](/docs/cloud-run) c. [Deploy flows to any Node.js platform](/docs/deploy-node)

## Step 1. Add the Firebase plugin

[Section titled “Step 1. Add the Firebase plugin”](#step-1-add-the-firebase-plugin)

Install the `@genkit-ai/firebase` plugin in your project:

```bash
npm install @genkit-ai/firebase
```

### Environment-based configuration

[Section titled “Environment-based configuration”](#environment-based-configuration)

If you intend to use the default configuration for Firebase Genkit Monitoring, you can enable telemetry by setting the `ENABLE_FIREBASE_MONITORING` environment variable in your deployment environment.

```bash
export ENABLE_FIREBASE_MONITORING=true
```

### Programmatic configuration

[Section titled “Programmatic configuration”](#programmatic-configuration)

You can also enable Firebase Genkit Monitoring in code. This is useful if you want to tweak any configuration settings like the metric export interval or to set up your local environment to export telemetry data.

Import `enableFirebaseTelemetry` into your Genkit configuration file (the file where `genkit(...)` is initalized), and call it:

```typescript
import { enableFirebaseTelemetry } from '@genkit-ai/firebase';


enableFirebaseTelemetry();
```

## Step 2. Enable the required APIs

[Section titled “Step 2. Enable the required APIs”](#step-2-enable-the-required-apis)

Make sure that the following APIs are enabled for your Google Cloud project:

* [Cloud Logging API](https://console.cloud.google.com/apis/library/logging.googleapis.com)
* [Cloud Trace API](https://console.cloud.google.com/apis/library/cloudtrace.googleapis.com)
* [Cloud Monitoring API](https://console.cloud.google.com/apis/library/monitoring.googleapis.com)

These APIs should be listed in the [API dashboard](https://console.cloud.google.com/apis/dashboard) for your project.

## Step 3. Set up permissions

[Section titled “Step 3. Set up permissions”](#step-3-set-up-permissions)

The Firebase plugin needs to use a *service account* to authenticate with Google Cloud Logging, Metrics, and Trace services.

Grant the following roles to whichever service account is configured to run your code within the [Google Cloud IAM Console](https://console.cloud.google.com/iam-admin/iam). For Cloud Functions for Firebase and Cloud Run, that’s typically the default compute service account.

* **Monitoring Metric Writer** (`roles/monitoring.metricWriter`)
* **Cloud Trace Agent** (`roles/cloudtrace.agent`)
* **Logs Writer** (`roles/logging.logWriter`)

## Step 4. (Optional) Test your configuration locally

[Section titled “Step 4. (Optional) Test your configuration locally”](#step-4-optional-test-your-configuration-locally)

Before deploying, you can run your Genkit code locally to confirm that telemetry data is being collected, and is viewable in the Genkit Monitoring dashboard.

1. In your Genkit code, set `forceDevExport` to `true` to send telemetry from your local environment.

2. Use your service account to authenticate and test your configuration.

   With the [Google Cloud CLI tool](https://cloud.google.com/sdk/docs/install?authuser=0), authenticate using the service account:

   ```bash
   gcloud auth application-default login --impersonate-service-account SERVICE_ACCT_EMAIL
   ```

3. Run and invoke your Genkit feature, and then view metrics on the [Genkit Monitoring dashboard](https://console.firebase.google.com/project/_/genai_monitoring). Allow for up to 5 minutes to collect the first metric. You can reduce this delay by setting `metricExportIntervalMillis` in the telemetry configuration.

4. If metrics are not appearing in the Genkit Monitoring dashboard, view the [Troubleshooting](/docs/observability/troubleshooting) guide for steps to debug.

## Step 5. Re-build and deploy code

[Section titled “Step 5. Re-build and deploy code”](#step-5-re-build-and-deploy-code)

Re-build, deploy, and invoke your Genkit feature to start collecting data. After Genkit Monitoring receives your metrics, you can view them by visiting the [Genkit Monitoring dashboard](https://console.firebase.google.com/project/_/genai_monitoring)

# Telemetry Collection

> This document details the metrics, trace attributes, and logs collected by the Firebase telemetry plugin for Genkit, along with information on latency, quotas, and cost.

The Firebase telemetry plugin exports a combination of metrics, traces, and logs to Google Cloud Observability. This document details which metrics, trace attributes, and logs will be collected and what you can expect in terms of latency, quotas, and cost.

## Telemetry delay

[Section titled “Telemetry delay”](#telemetry-delay)

There may be a slight delay before telemetry from a given invocation is available in Firebase. This is dependent on your export interval (5 minutes by default).

## Quotas and limits

[Section titled “Quotas and limits”](#quotas-and-limits)

There are several quotas that are important to keep in mind:

* [Cloud Trace Quotas](http://cloud.google.com/trace/docs/quotas)
* [Cloud Logging Quotas](http://cloud.google.com/logging/quotas)
* [Cloud Monitoring Quotas](http://cloud.google.com/monitoring/quotas)

## Cost

[Section titled “Cost”](#cost)

Cloud Logging, Cloud Trace, and Cloud Monitoring have generous free-of-charge tiers. Specific pricing can be found at the following links:

* [Cloud Logging Pricing](http://cloud.google.com/stackdriver/pricing#google-cloud-observability-pricing)
* [Cloud Trace Pricing](https://cloud.google.com/trace#pricing)
* [Cloud Monitoring Pricing](https://cloud.google.com/stackdriver/pricing#monitoring-pricing-summary)

## Metrics

[Section titled “Metrics”](#metrics)

The Firebase telemetry plugin collects a number of different metrics to support the various Genkit action types detailed in the following sections.

### Feature metrics

[Section titled “Feature metrics”](#feature-metrics)

Features are the top-level entry-point to your Genkit code. In most cases, this will be a flow. Otherwise, this will be the top-most span in a trace.

| Name                    | Type      | Description             |
| ----------------------- | --------- | ----------------------- |
| genkit/feature/requests | Counter   | Number of requests      |
| genkit/feature/latency  | Histogram | Execution latency in ms |

Each feature metric contains the following dimensions:

| Name          | Description                                                                      |
| ------------- | -------------------------------------------------------------------------------- |
| name          | The name of the feature. In most cases, this is the top-level Genkit flow        |
| status        | ’success’ or ‘failure’ depending on whether or not the feature request succeeded |
| error         | Only set when `status=failure`. Contains the error type that caused the failure  |
| source        | The Genkit source language. Eg. ‘ts’                                             |
| sourceVersion | The Genkit framework version                                                     |

### Action metrics

[Section titled “Action metrics”](#action-metrics)

Actions represent a generic step of execution within Genkit. Each of these steps will have the following metrics tracked:

| Name                   | Type      | Description                                   |
| ---------------------- | --------- | --------------------------------------------- |
| genkit/action/requests | Counter   | Number of times this action has been executed |
| genkit/action/latency  | Histogram | Execution latency in ms                       |

Each action metric contains the following dimensions:

| Name          | Description                                                                                          |
| ------------- | ---------------------------------------------------------------------------------------------------- |
| name          | The name of the action                                                                               |
| featureName   | The name of the parent feature being executed                                                        |
| path          | The path of execution from the feature root to this action. eg. ‘/myFeature/parentAction/thisAction’ |
| status        | ’success’ or ‘failure’ depending on whether or not the action succeeded                              |
| error         | Only set when `status=failure`. Contains the error type that caused the failure                      |
| source        | The Genkit source language. Eg. ‘ts’                                                                 |
| sourceVersion | The Genkit framework version                                                                         |

### Generate metrics

[Section titled “Generate metrics”](#generate-metrics)

These are special action metrics relating to actions that interact with a model. In addition to requests and latency, input and output are also tracked, with model specific dimensions that make debugging and configuration tuning easier.

| Name                                 | Type      | Description                                |
| ------------------------------------ | --------- | ------------------------------------------ |
| genkit/ai/generate/requests          | Counter   | Number of times this model has been called |
| genkit/ai/generate/latency           | Histogram | Execution latency in ms                    |
| genkit/ai/generate/input/tokens      | Counter   | Input tokens                               |
| genkit/ai/generate/output/tokens     | Counter   | Output tokens                              |
| genkit/ai/generate/input/characters  | Counter   | Input characters                           |
| genkit/ai/generate/output/characters | Counter   | Output characters                          |
| genkit/ai/generate/input/images      | Counter   | Input images                               |
| genkit/ai/generate/output/images     | Counter   | Output images                              |
| genkit/ai/generate/input/audio       | Counter   | Input audio files                          |
| genkit/ai/generate/output/audio      | Counter   | Output audio files                         |

Each generate metric contains the following dimensions:

| Name          | Description                                                                                          |
| ------------- | ---------------------------------------------------------------------------------------------------- |
| modelName     | The name of the model                                                                                |
| featureName   | The name of the parent feature being executed                                                        |
| path          | The path of execution from the feature root to this action. eg. ‘/myFeature/parentAction/thisAction’ |
| latencyMs     | The response time taken by the model                                                                 |
| status        | ’success’ or ‘failure’ depending on whether or not the feature request succeeded                     |
| error         | Only set when `status=failure`. Contains the error type that caused the failure                      |
| source        | The Genkit source language. Eg. ‘ts’                                                                 |
| sourceVersion | The Genkit framework version                                                                         |

## Traces

[Section titled “Traces”](#traces)

All Genkit actions are automatically instrumented to provide detailed traces for your AI features. Locally, traces are visible in the Developer UI. For deployed apps enable Genkit Monitoring to get the same level of visibility.

The following sections describe what trace attributes you can expect based on the Genkit action type for a particular span in the trace.

### Root Spans

[Section titled “Root Spans”](#root-spans)

Root spans have special attributes to help disambiguate the state attributes for the whole trace versus an individual span.

| Attribute name   | Description                                                                                                             |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------- |
| genkit/feature   | The name of the parent feature being executed                                                                           |
| genkit/isRoot    | Marked true if this span is the root span                                                                               |
| genkit/rootState | The state of the overall execution as `success` or `error`. This does not indicate that this step failed in particular. |

### Flow

[Section titled “Flow”](#flow)

| Attribute name          | Description                                                                                                |
| ----------------------- | ---------------------------------------------------------------------------------------------------------- |
| genkit/input            | The input to the flow. This will always be `<redacted>` because of trace attribute size limits.            |
| genkit/metadata/subtype | The type of Genkit action. For flows it will be `flow`.                                                    |
| genkit/name             | The name of this Genkit action. In this case the name of the flow                                          |
| genkit/output           | The output generated in the flow. This will always be `<redacted>` because of trace attribute size limits. |
| genkit/path             | The fully qualified execution path that lead to this step in the trace, including type information.        |
| genkit/state            | The state of this span’s execution as `success` or `error`.                                                |
| genkit/type             | The type of Genkit primitive that corresponds to this span. For flows, this will be `action`.              |

### Util

[Section titled “Util”](#util)

| Attribute name | Description                                                                                                |
| -------------- | ---------------------------------------------------------------------------------------------------------- |
| genkit/input   | The input to the util. This will always be `<redacted>` because of trace attribute size limits.            |
| genkit/name    | The name of this Genkit action. In this case the name of the flow                                          |
| genkit/output  | The output generated in the util. This will always be `<redacted>` because of trace attribute size limits. |
| genkit/path    | The fully qualified execution path that lead to this step in the trace, including type information.        |
| genkit/state   | The state of this span’s execution as `success` or `error`.                                                |
| genkit/type    | The type of Genkit primitive that corresponds to this span. For flows, this will be `util`.                |

### Model

[Section titled “Model”](#model)

| Attribute name          | Description                                                                                                 |
| ----------------------- | ----------------------------------------------------------------------------------------------------------- |
| genkit/input            | The input to the model. This will always be `<redacted>` because of trace attribute size limits.            |
| genkit/metadata/subtype | The type of Genkit action. For models it will be `model`.                                                   |
| genkit/model            | The name of the model.                                                                                      |
| genkit/name             | The name of this Genkit action. In this case the name of the model.                                         |
| genkit/output           | The output generated by the model. This will always be `<redacted>` because of trace attribute size limits. |
| genkit/path             | The fully qualified execution path that lead to this step in the trace, including type information.         |
| genkit/state            | The state of this span’s execution as `success` or `error`.                                                 |
| genkit/type             | The type of Genkit primitive that corresponds to this span. For flows, this will be `action`.               |

### Tool

[Section titled “Tool”](#tool)

| Attribute name          | Description                                                                                                 |
| ----------------------- | ----------------------------------------------------------------------------------------------------------- |
| genkit/input            | The input to the model. This will always be `<redacted>` because of trace attribute size limits.            |
| genkit/metadata/subtype | The type of Genkit action. For tools it will be `tool`.                                                     |
| genkit/name             | The name of this Genkit action. In this case the name of the model.                                         |
| genkit/output           | The output generated by the model. This will always be `<redacted>` because of trace attribute size limits. |
| genkit/path             | The fully qualified execution path that lead to this step in the trace, including type information.         |
| genkit/state            | The state of this span’s execution as `success` or `error`.                                                 |
| genkit/type             | The type of Genkit primitive that corresponds to this span. For flows, this will be `action`.               |

## Logs

[Section titled “Logs”](#logs)

For deployed apps with Genkit Monitoring, logs are used to capture input, output, and configuration metadata that provides rich detail about each step in your AI feature.

All logs will include the following shared metadata fields:

| Field name        | Description                                                                                                                       |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| insertId          | Unique id for the log entry                                                                                                       |
| jsonPayload       | Container for variable information that is unique to each log type                                                                |
| labels            | `{module: genkit}`                                                                                                                |
| logName           | `projects/weather-gen-test-next/logs/genkit_log`                                                                                  |
| receivedTimestamp | Time the log was received by Cloud                                                                                                |
| resource          | Information about the source of the log including deployment information region, and projectId                                    |
| severity          | The log level written. See Cloud’s [LogSeverity](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#logseverity) |
| spanId            | Identifier for the span that created this log                                                                                     |
| timestamp         | Time that the client logged a message                                                                                             |
| trace             | Identifier for the trace of the format `projects/<project-id>/traces/<trace-id>`                                                  |
| traceSampled      | Boolean representing whether the trace was sampled. Logs are not sampled.                                                         |

Each log type will have a different json payload described in each section.

### Input

[Section titled “Input”](#input)

JSON payload:

| Field name | Description                                                                                  |
| ---------- | -------------------------------------------------------------------------------------------- |
| message    | `[genkit] Input[<path>, <featureName>]` including `(message X of N)` for multi-part messages |
| metadata   | Additional context including the input message sent to the action                            |

Metadata:

| Field name       | Description                                                                                                                                                     |
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| content          | The input message content sent to this Genkit action                                                                                                            |
| featureName      | The name of the Genkit flow, action, tool, util, or helper.                                                                                                     |
| messageIndex \*  | Index indicating the order of messages for inputs that contain multiple messages. For single messages, this will always be 0.                                   |
| model \*         | Model name.                                                                                                                                                     |
| path             | The execution path that generated this log of the format `step1 > step2 > step3`                                                                                |
| partIndex \*     | Index indicating the order of parts within a message for multi-part messages. This is typical when combining text and images in a single input.                 |
| qualifiedPath    | The execution path that generated this log, including type information of the format: `/{flow1,t:flow}/{generate,t:util}/{modelProvider/model,t:action,s:model` |
| totalMessages \* | The total number of messages for this input. For single messages, this will always be 1.                                                                        |
| totalParts \*    | Total number of parts for this message. For single-part messages, this will always be 1.                                                                        |

(\*) Starred items are only present on Input logs for model interactions.

### Output

[Section titled “Output”](#output)

JSON payload:

| Field name | Description                                                                                   |
| ---------- | --------------------------------------------------------------------------------------------- |
| message    | `[genkit] Output[<path>, <featureName>]` including `(message X of N)` for multi-part messages |
| metadata   | Additional context including the input message sent to the action                             |

Metadata:

| Field name                      | Description                                                                                                                                                     |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| candidateIndex \* (deprecated)  | Index indicating the order of candidates for outputs that contain multiple candidates. For logs with single candidates, this will always be 0.                  |
| content                         | The output message generated by the Genkit action                                                                                                               |
| featureName                     | The name of the Genkit flow, action, tool, util, or helper.                                                                                                     |
| messageIndex \*                 | Index indicating the order of messages for inputs that contain multiple messages. For single messages, this will always be 0.                                   |
| model \*                        | Model name.                                                                                                                                                     |
| path                            | The execution path that generated this log of the format \`step1 > step2 > step3                                                                                |
| partIndex \*                    | Index indicating the order of parts within a message for multi-part messages. This is typical when combining text and images in a single output.                |
| qualifiedPath                   | The execution path that generated this log, including type information of the format: `/{flow1,t:flow}/{generate,t:util}/{modelProvider/model,t:action,s:model` |
| totalCandidates \* (deprecated) | Total number of candidates generated as output. For single-candidate messages, this will always be 1.                                                           |
| totalParts \*                   | Total number of parts for this message. For single-part messages, this will always be 1.                                                                        |

(\*) Starred items are only present on Output logs for model interactions.

### Config

[Section titled “Config”](#config)

JSON payload:

| Field name | Description                                                       |
| ---------- | ----------------------------------------------------------------- |
| message    | `[genkit] Config[<path>, <featureName>]`                          |
| metadata   | Additional context including the input message sent to the action |

Metadata:

| Field name    | Description                                                                                                                                                     |
| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| featureName   | The name of the Genkit flow, action, tool, util, or helper.                                                                                                     |
| model         | Model name.                                                                                                                                                     |
| path          | The execution path that generated this log of the format \`step1 > step2 > step3                                                                                |
| qualifiedPath | The execution path that generated this log, including type information of the format: `/{flow1,t:flow}/{generate,t:util}/{modelProvider/model,t:action,s:model` |
| source        | The Genkit library language used. This will always be set to ‘ts’ as it is the only supported language.                                                         |
| sourceVersion | The Genkit library version.                                                                                                                                     |
| temperature   | Model temperature used.                                                                                                                                         |

### Paths

[Section titled “Paths”](#paths)

JSON payload:

| Field name | Description                                                       |
| ---------- | ----------------------------------------------------------------- |
| message    | `[genkit] Paths[<path>, <featureName>]`                           |
| metadata   | Additional context including the input message sent to the action |

Metadata:

| Field name | Description                                                      |
| ---------- | ---------------------------------------------------------------- |
| flowName   | The name of the Genkit flow, action, tool, util, or helper.      |
| paths      | An array containing all execution paths for the collected spans. |

# Genkit Monitoring - Troubleshooting

> This guide provides solutions to common issues encountered when using Genkit Monitoring, including problems with traces, metrics, and telemetry export.

The following sections detail solutions to common issues that developers run into when using Genkit Monitoring.

## I can’t see traces or metrics in Genkit Monitoring

[Section titled “I can’t see traces or metrics in Genkit Monitoring”](#i-cant-see-traces-or-metrics-in-genkit-monitoring)

1. Ensure that the following APIs are enabled for your underlying Google Cloud project:

   * [Cloud Logging API](https://console.cloud.google.com/apis/library/logging.googleapis.com)
   * [Cloud Trace API](https://console.cloud.google.com/apis/library/cloudtrace.googleapis.com)
   * [Cloud Monitoring API](https://console.cloud.google.com/apis/library/monitoring.googleapis.com)

2. Ensure that the following roles are applied to the service account that is running your code (or service account that has been configured as part of the plugin options) in [Cloud IAM](https://console.cloud.google.com/iam-admin/iam).

   * **Monitoring Metric Writer** (`roles/monitoring.metricWriter`)
   * **Cloud Trace Agent** (`roles/cloudtrace.agent`)
   * **Logs Writer** (`roles/logging.logWriter`)

3. Inspect the application logs for errors writing to Cloud Logging, Cloud Trace, and Cloud Monitoring. On Google Cloud infrastructure such as Firebase Functions and Cloud Run, even when telemetry is misconfigured, logs to `stdout/stderr` are automatically ingested by the Cloud Logging Agent, allowing you to diagnose issues in the in the [Cloud Logging Console](https://console.cloud.google.com/logs).

4. Debug locally:

   Enable dev export:

   ```typescript
   enableFirebaseTelemetry({
     forceDevExport: true,
   });
   ```

   To test with your personal user credentials, use the [gcloud CLI](https://cloud.google.com/sdk/docs/install) to authenticate with Google Cloud. Doing so can help diagnose enabled or disabled APIs, but does not test the gcloud auth application-default login.

   Alternatively, impersonating the service account lets you test production-like access. You must have the `roles/iam. serviceAccountTokenCreator` IAM role applied to your user account in order to impersonate service accounts:

   ```bash
   gcloud auth application-default login --impersonate-service-account <SERVICE_ACCT_EMAIL>
   ```

   See the [ADC](https://cloud.google.com/docs/authentication/set-up-adc-local-dev-environment) documentation for more information.

## Request count does not match traces count

[Section titled “Request count does not match traces count”](#request-count-does-not-match-traces-count)

At low volumes (<1 query per second), you may notice that your metric counts, like requests or failed paths, do not match the number of traces shown in the traces table. Below are three common reasons for this happening.

### Metric and trace export intervals can be different

[Section titled “Metric and trace export intervals can be different”](#metric-and-trace-export-intervals-can-be-different)

In some cases, the dashboard shows traces that have exported but metrics that have not, or vice versa.

You can reduce the likelihood of this happening by adjusting the metric export interval to be more frequent. By default, metrics are exported every 5 minutes. The minimum allowable export interval is 5 seconds.

```typescript
enableFirebaseTelemetry({
  // Override the export interval to 3 minutes
  metricExportIntervalMillis: 180_000,
  // Override the export timeout to 3 minutes
  metricExportTimeoutMillis: 180_000,
});
```

### Intermittent network issues

[Section titled “Intermittent network issues”](#intermittent-network-issues)

Occasionally you may have transient network issues that result in a failure to upload telemetry data. These failures are logged to Google Cloud Logging. To see the specific failure reason, look for a log that starts with —

> Unable to send telemetry to Google Cloud: Error: Send TimeSeries failed:

### Telemetry upload reliability in Firebase Functions or Cloud Run

[Section titled “Telemetry upload reliability in Firebase Functions or Cloud Run”](#telemetry-upload-reliability-in-firebase-functions-or-cloud-run)

When your Genkit codei is hosted in Google Cloud Run or Cloud Functions for Firebase, telemetry-data upload may be less reliable as the container switches to the “idle” [lifecycle state](https://cloud.google.com/blog/topics/developers-practitioners/lifecycle-container-cloud-run). If higher reliability is important to you, consider changing [CPU allocation](https://cloud.google.com/run/docs/configuring/cpu-allocation) to **Instance-based billing** (previously called **CPU always allocated**) in the Google Cloud Console.

To switch to instance-based billing, run

```bash
gcloud run services update YOUR-SERVICE --no-cpu-throttling
```