Middleware

Genkit allows you to use middleware to modify the behavior of generate() calls. Middleware can be used for various purposes, such as retrying failed requests, falling back to different models, or injecting tools and context.

You can use pre-packaged middleware or build your own custom middleware.

Middleware in Dart

Genkit Dart uses a registry-based system for middleware, similar to plugins. This allows middleware to be resolved by name and configured via schemas, enabling support for the Genkit Developer UI.

Available middleware

retry and simulateConstrainedGeneration are built into package:genkit and need no setup. The filesystem, skills, and toolApproval middleware come from the genkit_middleware package; register their plugins on the Genkit instance so the references resolve:

import 'package:genkit/genkit.dart';
import 'package:genkit_google_genai/genkit_google_genai.dart';
import 'package:genkit_middleware/genkit_middleware.dart';

final ai = Genkit(
  plugins: [
    googleAI(),
    FilesystemPlugin(),
    SkillsPlugin(),
    ToolApprovalPlugin(),
  ],
);

1. FileSystem middleware (filesystem)

Grants the model access to the local filesystem by injecting standard file manipulation tools (list_files, read_file, write_file, search_and_replace). All operations are safely restricted to a specified root directory.

final response = await ai.generate(
  model: googleAI.gemini('gemini-flash-latest'),
  prompt: 'Create a hello world node app in the workspace',
  use: [
    filesystem(rootDirectory: './workspace'),
  ],
);

Configuration options:

• rootDirectory (required): The root directory to which all filesystem operations are restricted.

Note: Unlike the JavaScript version, the Dart FileSystem middleware does not currently support allowWriteAccess or toolNamePrefix options and always enables write operations.

2. Skills middleware (skills)

Automatically scans a directory for SKILL.md files and injects them into the system prompt. It also provides a use_skill tool the model can use to retrieve more specific skills on demand.

final response = await ai.generate(
  prompt: 'How do I run tests in this repo?',
  use: [
    skills(skillPaths: ['./skills']),
  ],
);

Configuration options:

• skillPaths (optional): Paths to directories containing skills (defaults to ['skills']).

3. Tool approval middleware (toolApproval)

Restricts execution of tools to an approved list. If the model calls a tool that isn’t on the list, the generation stops with FinishReason.interrupted so you can ask the user for confirmation before resuming.

final response = await ai.generate(
  prompt: 'write a file',
  tools: [writeFileTool],
  use: [
    toolApproval(approved: []), // Empty list: every tool call needs approval
  ],
);

if (response.finishReason == FinishReason.interrupted) {
  final interrupt = response.interrupts.first;

  // Ask user for approval...
  final approved = await askUser(interrupt.toolRequest.name);

  if (approved) {
    // Restart the tool call, marked as approved
    final response2 = await ai.generate(
      messages: response.messages,
      tools: [writeFileTool],
      interruptRestart: [
        interrupt.restart({'tool-approved': true}),
      ],
      use: [
        toolApproval(approved: []),
      ],
    );
  }
}

Configuration options:

• approved (optional): List of approved tool names. Replaces the default list set with ToolApprovalPlugin(approvedTools: [...]).

4. Retry middleware (retry)

Automatically retries failed model generations on transient error codes (like resourceExhausted, unavailable) using exponential backoff with jitter. It is built into package:genkit, so there is no plugin to register.

import 'package:genkit/genkit.dart';

final response = await ai.generate(
  model: googleAI.gemini('gemini-flash-latest'),
  prompt: 'Reliable request',
  use: [
    retry(
      maxRetries: 3,
      initialDelay: const Duration(milliseconds: 500),
      maxDelay: const Duration(seconds: 30),
    ),
  ],
);

Configuration options:

• maxRetries (optional): Maximum number of retry attempts (default: 3).
• statuses (optional): A list of StatusCode values that should trigger a retry. Defaults to unavailable, deadlineExceeded, resourceExhausted, aborted, internal.
• initialDelay (optional): The delay before the first retry, as a Duration (default: 1 second).
• maxDelay (optional): The maximum capped delay between retries, as a Duration (default: 1 minute).
• backoffFactor (optional): Exponential backoff multiplier (default: 2.0).
• noJitter (optional): Set to true to disable the random factor (Full Jitter) added to the delay (default: false).
• noRetryModel (optional): Set to true to skip retrying model calls (default: false).
• retryTools (optional): Whether to retry tool calls (default: false).

Note: The onError callback is available when instantiating RetryMiddleware directly, but cannot be passed via the retry() helper.

5. Simulated constrained generation (simulateConstrainedGeneration)

With an outputSchema, Genkit asks the model for native constrained output. For a model or provider without native support, simulateConstrainedGeneration() puts the schema in the prompt instead. Nothing enforces the schema, so expect the occasional malformed reply.

final response = await ai.generate(
  model: myModel,
  prompt: 'Generate a person named John Doe, age 30',
  outputSchema: Person.$schema,
  use: [simulateConstrainedGeneration()],
);
final person = response.output; // Still typed and parsed

Building your own custom middleware

A custom middleware is a class that extends GenerateMiddleware, plus an optional configuration schema. Register it so it can be referenced from use: and shows up (and can be configured) in the Developer UI.

1. Define the configuration schema

Use the @Schema() annotation to define the configuration options for your middleware.

import 'package:schemantic/schemantic.dart';

part 'logger.g.dart';

@Schema()
abstract class $LoggerOptions {
  bool? get enableColor;
  int? get maxLogLength;
}

2. Implement the middleware logic

Create a class that extends GenerateMiddleware and override the hooks you need (model, tool, or generate).

class LoggerMiddleware extends GenerateMiddleware {
  final bool enableColor;
  final int maxLogLength;

  LoggerMiddleware({
    this.enableColor = false,
    this.maxLogLength = 1000,
  });

  @override
  Future<ModelResponse> model(
    ModelRequest request,
    ActionFnArg<ModelResponseChunk, ModelRequest, void> ctx,
    Future<ModelResponse> Function(
      ModelRequest request,
      ActionFnArg<ModelResponseChunk, ModelRequest, void> ctx,
    ) next,
  ) async {
    // Custom interception logic here...
    return next(request, ctx);
  }
}

3. Register the middleware

In app code, ai.defineGenerateMiddleware registers the middleware and returns its definition. Call the definition to build a reference for use:. The create callback receives the resolved config and a GenerateMiddlewareContext (ctx), and runs once per generate call.

final logger = ai.defineGenerateMiddleware<LoggerOptions>(
  name: 'logger',
  configSchema: LoggerOptions.$schema,
  create: (config, ctx) => LoggerMiddleware(
    enableColor: config?.enableColor ?? false,
    maxLogLength: config?.maxLogLength ?? 1000,
  ),
);

final response = await ai.generate(
  model: googleAI.gemini('gemini-flash-latest'),
  prompt: 'Hello world',
  use: [
    logger(LoggerOptions(enableColor: true, maxLogLength: 500)),
    retry(maxRetries: 2),
  ],
);

The ctx argument carries an ephemeral GenkitAI instance (ctx.ai) backed by the active registry. This lets a middleware run its own AI operations (generate, generateStream, embed) and resolve other registered actions at request time, which is useful for building guardrails, classifiers, or routers. For example, a safety middleware can screen a request with the model before letting it proceed:

class SafetyMiddleware extends GenerateMiddleware {
  final GenkitAI ai;

  SafetyMiddleware(this.ai);

  @override
  Future<GenerateResult> generate(
    GenerateTurnState envelope,
    ActionFnArg<ModelResponseChunk, GenerateActionOptions, void> ctx,
    Future<GenerateResult> Function(
      GenerateTurnState envelope,
      ActionFnArg<ModelResponseChunk, GenerateActionOptions, void> ctx,
    ) next,
  ) async {
    // Use the ephemeral GenkitAI to classify the request before proceeding.
    final verdict = await ai.generate(
      prompt: 'Reply "block" if this request is unsafe, otherwise "allow": '
          '${envelope.request.messages.last.text}',
    );
    if (verdict.text.trim().toLowerCase().startsWith('block')) {
      throw GenkitException('Request blocked by safety middleware.');
    }
    return next(envelope, ctx);
  }
}

final safety = ai.defineGenerateMiddleware<void>(
  name: 'safety',
  // `ctx.ai` is an ephemeral GenkitAI the middleware can use for its own
  // generate/embed calls at request time.
  create: (config, ctx) => SafetyMiddleware(ctx.ai),
);

// use: [safety()]

4. Package it in a plugin

To share a middleware across apps or publish it, build the definition with generateMiddleware (it doesn’t register anything by itself) and return it from a GenkitPlugin. A small helper with named parameters makes it pleasant to use:

final loggerDef = generateMiddleware<LoggerOptions>(
  // The name should be reasonably unique to avoid conflicts with other plugins.
  name: 'logger',
  configSchema: LoggerOptions.$schema,
  create: (config, ctx) => LoggerMiddleware(
    enableColor: config?.enableColor ?? false,
    maxLogLength: config?.maxLogLength ?? 1000,
  ),
);

class LoggerPlugin extends GenkitPlugin {
  @override
  String get name => 'logger';

  @override
  List<GenerateMiddlewareDef> middleware() => [loggerDef];
}

GenerateMiddlewareRef<LoggerOptions> logger({
  bool? enableColor,
  int? maxLogLength,
}) => loggerDef(
  LoggerOptions(enableColor: enableColor, maxLogLength: maxLogLength),
);

Users register the plugin and reference the middleware through the helper:

final ai = Genkit(plugins: [googleAI(), LoggerPlugin()]);

final response = await ai.generate(
  model: googleAI.gemini('gemini-flash-latest'),
  prompt: 'Hello world',
  use: [logger(enableColor: true, maxLogLength: 500)],
);

For more complex examples of building custom middleware, you can refer to the source code of the built-in middleware in the Genkit Dart GitHub repository.