Creating Genkit plugins

Genkit’s capabilities are designed to be extended by plugins. Genkit plugins are configurable modules that can provide models, retrievers, trace stores, and more. You’ve already seen plugins in action just by using Genkit:

import (

    "github.com/firebase/genkit/go/ai"
    "github.com/firebase/genkit/go/genkit"
    "github.com/firebase/genkit/go/plugins/googlegenai"
    "github.com/firebase/genkit/go/plugins/server"
)

g := genkit.Init(ctx,
    genkit.WithPlugins(
        &googlegenai.GoogleAI{APIKey: ...},
        &googlegenai.VertexAI{ProjectID: "my-project", Location: "us-central1"},
    ),
)

The Vertex AI plugin takes configuration (such as the user’s Google Cloud project ID) and registers a variety of new models, embedders, and more with the Genkit registry. The registry serves as a lookup service for named actions at runtime, and powers Genkit’s local UI for running and inspecting models, prompts, and more.

Creating a Plugin

In Go, a Genkit plugin is a package that adheres to a small set of conventions. A single module can contain several plugins.

Provider ID

Every plugin must have a unique identifier string that distinguishes it from other plugins. Genkit uses this identifier as a namespace for every resource your plugin defines, to prevent naming conflicts with other plugins.

For example, if your plugin has an ID yourplugin and provides a model called text-generator, the full model identifier will be yourplugin/text-generator.

This provider ID needs to be exported and you should define it once for your plugin and use it consistently when required by a Genkit function.

package yourplugin

const providerID = "yourplugin"

Standard Exports

Every plugin should define and export the following symbols to conform to the genkit.Plugin interface:

• A struct type that encapsulates all of the configuration options accepted by the plugin.

  For any plugin options that are secret values, such as API keys, you should offer both a config option and a default environment variable to configure it. This lets your plugin take advantage of the secret-management features offered by many hosting providers (such as Cloud Secret Manager, which you can use with Cloud Run). For example:

type MyPlugin struct {
    APIKey string
    // Other options you may allow to configure...
}

• A Name() method on the struct that returns the provider ID.

• An Init() method on the struct with a declaration like the following:

func (m *MyPlugin) Init(ctx context.Context, g *genkit.Genkit) error

In this function, perform any setup steps required by your plugin. For example:

• Confirm that any required configuration values are specified and assign default values to any unspecified optional settings.
• Verify that the given configuration options are valid together.
• Create any shared resources required by the rest of your plugin. For example, create clients for any services your plugin accesses.

To the extent possible, the resources provided by your plugin shouldn’t assume that any other plugins have been installed before this one.

This method will be called automatically during genkit.Init() when the user passes the plugin into the WithPlugins() option.

Building Plugin Features

A single plugin can activate many new things within Genkit. For example, the Vertex AI plugin activates several new models as well as an embedder. Here’s how to build some common plugin types.

Model Plugins

Genkit model plugins add one or more generative AI models to the Genkit registry. A model represents any generative model that is capable of receiving a prompt as input and generating text, media, or data as output.

Model definitions

Generally, a model plugin will make one or more genkit.DefineModel() calls in its Init function—once for each model the plugin is providing an interface to.

A model definition consists of three components:

1. Metadata declaring the model’s capabilities.
2. A configuration type with any specific parameters supported by the model.
3. A generation function that accepts an ai.ModelRequest and returns an ai.ModelResponse, presumably using an AI model to generate the latter.

At a high level, here’s what it looks like in code:

package myplugin

import (
    "context"
    "fmt"

    "github.com/firebase/genkit/go/ai"
    "github.com/firebase/genkit/go/genkit"
)

const providerID = "myProvider" // Unique ID for your plugin provider

// MyModelConfig defines the configuration options for your model.
// Embed ai.GenerationCommonConfig for common options.
type MyModelConfig struct {
    ai.GenerationCommonConfig
    AnotherCustomOption string `json:"anotherCustomOption,omitempty"`
    CustomOption        int    `json:"customOption,omitempty"`
}

// DefineModel registers your custom model with Genkit.
func DefineMyModel(g *genkit.Genkit) {
    genkit.DefineModel(g, providerID+"/my-model",
        &ai.ModelOptions{
            Label: "My Model", // User-friendly label
            Supports: &ai.ModelSupports{
                Multiturn:  true,  // Does the model support multi-turn chats?
                SystemRole: true,  // Does the model support system messages?
                Media:      false, // Can the model accept media input?
                Tools:      false, // Does the model support function calling (tools)?
            },
            Versions: []string{"my-model-001"}, // List supported versions/aliases
        },
        // The generation function
        func(ctx context.Context, mr *ai.ModelRequest, cb ai.ModelStreamCallback) (*ai.ModelResponse, error) {
            // Verify that the request includes a configuration that conforms to your schema.
            var cfg MyModelConfig
            if mr.Config != nil {
                // Attempt to cast the config; handle potential type mismatch
                if typedCfg, ok := mr.Config.(*MyModelConfig); ok {
                    cfg = *typedCfg
                } else {
                    // Handle incorrect config type if necessary, or rely on default values
                    // For simplicity, this example proceeds with default cfg if cast fails
                }
            }
            // Now 'cfg' holds the configuration, either from the request or default.

            // Use your custom logic to convert Genkit's ai.ModelRequest into a form
            // usable by the model's native API.
            apiRequest, err := apiRequestFromGenkitRequest(mr, cfg) // Pass config too
            if err != nil {
                return nil, fmt.Errorf("failed to create API request: %w", err)
            }

            // Send the request to the model API, using your own code or the model
            // API's client library.
            apiResponse, err := callModelAPI(ctx, apiRequest) // Pass context if needed
            if err != nil {
                return nil, fmt.Errorf("model API call failed: %w", err)
            }

            // Use your custom logic to convert the model's response to Genkit's ai.ModelResponse.
            response, err := genResponseFromAPIResponse(apiResponse)
            if err != nil {
                return nil, fmt.Errorf("failed to convert API response: %w", err)
            }

            return response, nil
        },
    )
}

// Placeholder for the function that converts Genkit request to your API's format
func apiRequestFromGenkitRequest(mr *ai.ModelRequest, cfg MyModelConfig) (interface{}, error) {
    // Implementation depends on your specific model API
    fmt.Printf("Converting Genkit request with config: %+v\n", cfg)
    // ... conversion logic ...
    return "your-api-request-format", nil // Replace with actual request object
}

// Placeholder for the function that calls your model's API
func callModelAPI(ctx context.Context, apiRequest interface{}) (interface{}, error) {
    // Implementation depends on your specific model API client library
    // ... API call logic ...
    return "your-api-response-format", nil // Replace with actual response object
}

// Placeholder for the function that converts your API's response to Genkit's format
func genResponseFromAPIResponse(apiResponse interface{}) (*ai.ModelResponse, error) {
    // Implementation depends on your specific model API response format
    // ... conversion logic ...
    return &ai.ModelResponse{
        Candidates: []*ai.Candidate{
            {
                Message: &ai.Message{
                    Content: []*ai.Part{ai.NewTextPart("Generated response text")},
                    Role:    ai.RoleModel,
                },
                FinishReason: ai.FinishReasonStop,
            },
        },
    }, nil // Replace with actual response conversion
}

// Example Plugin implementation
type MyPlugin struct{}

func (p *MyPlugin) Name() string {
    return providerID
}

func (p *MyPlugin) Init(ctx context.Context, g *genkit.Genkit) error {
    DefineMyModel(g)
    // Define other models or resources here
    return nil
}

// Ensure MyPlugin implements genkit.Plugin
var _ genkit.Plugin = &MyPlugin{}

Declaring model capabilities

Every model definition must contain, as part of its metadata, an ai.ModelInfo value that declares which features the model supports. Genkit uses this information to determine certain behaviors, such as verifying whether certain inputs are valid for the model. For example, if the model doesn’t support multi-turn interactions, then it’s an error to pass it a message history.

Note that these declarations refer to the capabilities of the model as provided by your plugin, and do not necessarily map one-to-one to the capabilities of the underlying model and model API. For example, even if the model API doesn’t provide a specific way to define system messages, your plugin might still declare support for the system role, and implement it as special logic that inserts system messages into the user prompt.

Defining your model’s config schema

To specify the generation options a model supports, define and export a configuration type. Genkit has an ai.GenerationCommonConfig type that contains options frequently supported by generative AI model services, which you can embed or use outright.

Your generation function should verify that the request contains the correct options type.

Transforming requests and responses

The generation function carries out the primary work of a Genkit model plugin: transforming the ai.ModelRequest from Genkit’s common format into a format that is supported by your model’s API, and then transforming the response from your model into the ai.ModelResponse format used by Genkit.

Sometimes, this may require massaging or manipulating data to work around model limitations. For example, if your model does not natively support a system message, you may need to transform a prompt’s system message into a user-model message pair.

Exports

In addition to the resources that all plugins must export, a model plugin should also export the following:

• A generation config type, as discussed earlier.

• A Model() function, which returns references to your plugin’s defined models. Often, this can be:

func Model(g *genkit.Genkit, name string) *ai.Model {
    return genkit.LookupModel(g, providerID, name)
}

• A ModelRef function, which creates a model reference paired with its config that can validate the type and be passed around together:

func ModelRef(name string, config *MyModelConfig) *ai.ModelRef {
    return ai.NewModelRef(name, config)
}

• Optional: A DefineModel() function, which lets users define models that your plugin can provide, but that you do not automatically define. There are two main reasons why you might want to provide such a function:

  • Your plugin provides access to too many models to practically register each one. For example, the Ollama plugin can provide access to dozens of different models, with more added frequently. For this reason, it doesn’t automatically define any models, and instead requires the user to call DefineModel() for each model they want to use.

  • To give your users the ability to use newly-released models that you have not yet added to your plugin.

  A plugin’s DefineModel() function is typically a frontend to genkit.DefineModel() that defines a generation function, but lets the user specify the model name and model capabilities.

Telemetry Plugins

The Genkit libraries are instrumented with OpenTelemetry to support collecting traces, metrics, and logs. Genkit users can export this telemetry data to monitoring and visualization tools by installing a plugin that configures the OpenTelemetry Go SDK to export to a particular OpenTelemetry-capable system.

Genkit includes a plugin that configures OpenTelemetry to export data to Google Cloud Monitoring and Cloud Logging. To support other monitoring systems, you can extend Genkit by writing a telemetry plugin.

Exporters and Loggers

The primary job of a telemetry plugin is to configure OpenTelemetry to export data to a particular service. To do so, you need the following:

• An implementation of OpenTelemetry’s SpanExporter interface that exports data to the service of your choice.
• An implementation of OpenTelemetry’s metric.Exporter interface that exports data to the service of your choice.
• Either a slog.Logger or an implementation of the slog.Handler interface, that exports logs to the service of your choice.

Depending on the service you’re interested in exporting to, this might be a relatively minor effort or a large one.

Because OpenTelemetry is an industry standard, many monitoring services already have libraries that implement these interfaces. For example, the googlecloud plugin for Genkit makes use of the opentelemetry-operations-go library, maintained by the Google Cloud team. Similarly, many monitoring services provide libraries that implement the standard slog interfaces.

On the other hand, if no such libraries are available for your service, implementing the necessary interfaces can be a substantial project.

Check the OpenTelemetry registry or the monitoring service’s docs to see if integrations are already available.

If you need to build these integrations yourself, take a look at the source of the official OpenTelemetry exporters and the page A Guide to Writing slog Handlers.

Building the plugin

Dependencies

Every telemetry plugin needs to import the Genkit core library and several OpenTelemetry libraries:

// Import the Genkit core library.

"github.com/firebase/genkit/go/genkit"

// Import the OpenTelemetry libraries.
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/sdk/metric"
"go.opentelemetry.io/otel/sdk/trace"

If you are building a plugin around an existing OpenTelemetry or slog integration, you will also need to import them.

Config

A telemetry plugin should, at a minimum, support the following configuration options:

type Config struct {
    // Export even in the dev environment.
    ForceExport bool

    // The interval for exporting metric data.
    // The default is 60 seconds.
    MetricInterval time.Duration

    // The minimum level at which logs will be written.
    // Defaults to [slog.LevelInfo].
    LogLevel slog.Leveler
}

Most plugins will also include configuration settings for the service it’s exporting to (API key, project name, and so on).

Init()

The Init() function of a telemetry plugin should do all of the following:

• Return early if Genkit is running in a development environment (such as when running with with genkit start) and the Config.ForceExport option isn’t set:

shouldExport := cfg.ForceExport || os.Getenv("GENKIT_ENV") != "dev"
if !shouldExport {
  return nil
}

• Initialize your trace span exporter and register it with Genkit:

spanProcessor := trace.NewBatchSpanProcessor(YourCustomSpanExporter{})
genkit.RegisterSpanProcessor(g, spanProcessor)

• Initialize your metric exporter and register it with the OpenTelemetry library:

r := metric.NewPeriodicReader(
  YourCustomMetricExporter{},
  metric.WithInterval(cfg.MetricInterval),
)
mp := metric.NewMeterProvider(metric.WithReader(r))
otel.SetMeterProvider(mp)

Use the user-configured collection interval (Config.MetricInterval) when initializing the PeriodicReader.

• Register your slog handler as the default logger:

logger := slog.New(YourCustomHandler{
  Options: &slog.HandlerOptions{Level: cfg.LogLevel},
})
slog.SetDefault(logger)

You should configure your handler to honor the user-specified minimum log level (Config.LogLevel).

PII redaction

Because most generative AI flows begin with user input of some kind, it’s a likely possibility that some flow traces contain personally-identifiable information (PII). To protect your users’ information, you should redact PII from traces before you export them.

If you are building your own span exporter, you can build this functionality into it.

If you’re building your plugin around an existing OpenTelemetry integration, you can wrap the provided span exporter with a custom exporter that carries out this task. For example, the googlecloud plugin removes the genkit:input and genkit:output attributes from every span before exporting them using a wrapper similar to the following:

type redactingSpanExporter struct {
    trace.SpanExporter
}

func (e *redactingSpanExporter) ExportSpans(ctx context.Context, spanData []trace.ReadOnlySpan) error {
    var redacted []trace.ReadOnlySpan
    for _, s := range spanData {
        redacted = append(redacted, redactedSpan{s})
    }
    return e.SpanExporter.ExportSpans(ctx, redacted)
}

func (e *redactingSpanExporter) Shutdown(ctx context.Context) error {
    return e.SpanExporter.Shutdown(ctx)
}

type redactedSpan struct {
    trace.ReadOnlySpan
}

func (s redactedSpan) Attributes() []attribute.KeyValue {
    // Omit input and output, which may contain PII.
    var ts []attribute.KeyValue
    for _, a := range s.ReadOnlySpan.Attributes() {
        if a.Key == "genkit:input" || a.Key == "genkit:output" {
            continue
        }
        ts = append(ts, a)
    }
    return ts
}

Troubleshooting

If you’re having trouble getting data to show up where you expect, OpenTelemetry provides a useful diagnostic tool that helps locate the source of the problem.

Publishing a Plugin

Genkit plugins can be published as normal Go packages. To increase discoverability, your package should have genkit somewhere in its name so it can be found with a simple search on pkg.go.dev. Any of the following are good choices:

• github.com/yourorg/genkit-plugins/servicename
• github.com/yourorg/your-repo/genkit/servicename