Skip to content

Middleware

Genkit allows you to use middleware to modify the behavior of generate() calls. Middleware can be used for various purposes, such as retrying failed requests, falling back to different models, or injecting tools and context.

You can use pre-packaged middleware or build your own custom middleware.

The official Genkit middleware for Python is available in the genkit-middleware package.

Terminal window
uv add genkit-middleware

Passing a middleware instance in use=[...] works without any setup. Register the Middleware plugin to also list the built-ins in the Dev UI and to name them in a .prompt file’s use::

from genkit import Genkit
from genkit_google_genai import GoogleAI
from genkit_middleware import Middleware
ai = Genkit(
plugins=[
GoogleAI(),
Middleware(),
],
)

The genkit-middleware package provides several middleware classes out of the box.

Grants the model access to a root directory by injecting list_files and read_file. When allow_write_access=True (default False), it also injects write_file and edit_file. All paths are restricted to that root.

from genkit import Genkit
from genkit_google_genai import GoogleAI
from genkit_middleware import Filesystem
ai = Genkit(plugins=[GoogleAI()])
response = await ai.generate(
model=GoogleAI.gemini('gemini-flash-latest'),
prompt='Create a hello world node app in the workspace',
use=[
Filesystem(root_dir='./workspace', allow_write_access=True),
],
)

Configuration options:

  • root_dir (required): The root directory to which all filesystem operations are restricted.
  • allow_write_access (optional): If True, adds write/edit tools (defaults to False — read-only).
  • tool_name_prefix (optional): Prefix to add to the name of the injected tools.

Scans each directory in skill_paths for subfolders that contain a SKILL.md file, reads name and description from its YAML frontmatter, and lists the skills in the system prompt. It also adds a use_skill tool that loads a skill’s full SKILL.md on demand.

from genkit import Genkit
from genkit_google_genai import GoogleAI
from genkit_middleware import Skills
ai = Genkit(plugins=[GoogleAI()], model=GoogleAI.gemini('gemini-flash-latest'))
response = await ai.generate(
prompt='How do I run tests in this repo?',
use=[
Skills(skill_paths=['./skills']),
],
)

Configuration options:

  • skill_paths (optional): Paths to directories containing skills (defaults to ['skills']).

3. Tool approval middleware (ToolApproval)

Section titled “3. Tool approval middleware (ToolApproval)”

Restricts execution of tools to an approved list. If the model attempts to call an unapproved tool, generation finishes interrupted (FinishReason.INTERRUPTED) so you can prompt the user for confirmation before resuming.

from genkit import FinishReason, Genkit
from genkit_google_genai import GoogleAI
from genkit_middleware import ToolApproval
from pydantic import BaseModel, Field
ai = Genkit(plugins=[GoogleAI()], model=GoogleAI.gemini('gemini-flash-latest'))
class WriteFileInput(BaseModel):
path: str = Field(description='File path')
content: str = Field(description='File contents')
@ai.tool()
async def write_file_tool(input: WriteFileInput) -> str:
"""Write a text file."""
# Persist input.content to input.path in a real app.
return f'Wrote {input.path}'
# 1. Initial attempt
response = await ai.generate(
prompt='write a file',
tools=[write_file_tool],
use=[
ToolApproval(allowed_tools=[]), # Empty list means all tool calls trigger interrupt
],
)
if response.finish_reason == FinishReason.INTERRUPTED:
interrupt = response.interrupts[0]
# 2. Ask user for approval, then build a restart part with approval metadata
approved_part = interrupt.restart(
resumed_metadata={'tool_approved': True},
)
# 3. Resume execution
response = await ai.generate(
messages=response.messages,
tools=[write_file_tool],
use=[
ToolApproval(allowed_tools=[]),
],
resume_restart=approved_part,
)
print(response.text)

Configuration options:

  • allowed_tools (optional): List of approved tool names that can run without interruption.

Automatically retries failed model generations on transient error codes (like RESOURCE_EXHAUSTED, UNAVAILABLE) using exponential backoff with jitter.

from genkit import Genkit
from genkit_google_genai import GoogleAI
from genkit_middleware import Retry
ai = Genkit(plugins=[GoogleAI()])
response = await ai.generate(
model=GoogleAI.gemini('gemini-pro-latest'),
prompt='Analyze quarterly variance across all regional ledgers.',
use=[
Retry(
max_retries=3,
initial_delay_ms=1000,
backoff_factor=2.0,
),
],
)

Configuration options:

  • max_retries (optional): The maximum number of times to retry a failed request (default: 3).
  • statuses (optional): An array of status names that should trigger a retry (default: ['UNAVAILABLE', 'DEADLINE_EXCEEDED', 'RESOURCE_EXHAUSTED', 'ABORTED', 'INTERNAL']).
  • initial_delay_ms (optional): The initial delay between retries in milliseconds (default: 1000).
  • max_delay_ms (optional): The maximum delay between retries in milliseconds (default: 60000).
  • backoff_factor (optional): The factor by which the delay increases after each retry (exponential backoff, default: 2.0).
  • no_jitter (optional): Whether to disable jitter on the delay (default: False).

A GenkitError is retried only when its status is in statuses; any other exception is retried. If a GenkitError carries retry_after_ms, Retry waits at least that long, capped at max_delay_ms.

Automatically switches to a different model if the primary model fails on a specific set of error codes. Useful for falling back to a smaller/faster model when a large model exceeds quota limits.

from genkit import Genkit
from genkit_google_genai import GoogleAI
from genkit_middleware import Fallback
ai = Genkit(plugins=[GoogleAI()])
response = await ai.generate(
model=GoogleAI.gemini('gemini-pro-latest'),
prompt='Summarize the incident timeline.',
use=[
Fallback(
models=[GoogleAI.gemini('gemini-flash-latest')], # try flash if pro fails
statuses=['RESOURCE_EXHAUSTED'],
),
],
)

Configuration options:

  • models (required): A list of model names or ModelRef values to try in order.
  • statuses (optional): A list of status names that should trigger a fallback (default: ['UNAVAILABLE', 'DEADLINE_EXCEEDED', 'RESOURCE_EXHAUSTED', 'ABORTED', 'INTERNAL', 'NOT_FOUND', 'UNIMPLEMENTED']).

Each fallback model must resolve on the app, so register its plugin. A string entry runs with that model’s default config; a ModelRef entry runs with the ref’s own config. The primary request’s config is not carried over. Errors that aren’t a GenkitError with a listed status skip the fallbacks.

Gives the model read_artifact and write_artifact tools over the active agent session’s artifacts, and lists the current artifacts in the system prompt each turn. Artifacts depends on agent sessions, so it lives in genkit_middleware.exp and only works when passed to an agent defined with genkit.exp.Genkit. The Middleware() plugin doesn’t register it.

from genkit.exp import Genkit
from genkit_google_genai import GoogleAI
from genkit_middleware.exp import Artifacts
ai = Genkit(plugins=[GoogleAI()])
builder = ai.define_agent(
name='builder',
model=GoogleAI.gemini('gemini-flash-latest'),
system='You are a code generator. Use write_artifact to create files.',
use=[Artifacts()],
)
chat = builder.chat()
await chat.send('Write a README.md for a pizza ordering CLI.')
print([artifact.name for artifact in chat.artifacts])
# => ['README.md']

Configuration options:

  • readonly (optional): If True, only read_artifact is provided (default: False). Use it on an orchestrator that reads what its sub-agents produced; see Multi-agent delegation.

Implement custom middleware by subclassing Middleware from genkit.middleware and setting a unique name class attribute. Decorating the class with @ai.middleware registers it on the Genkit instance so it appears in the Developer UI.

The base class shares its name with the genkit_middleware.Middleware plugin. If a module needs both, alias one: from genkit_middleware import Middleware as MiddlewarePlugin.

Middleware can override four hooks:

  • tools(self, ctx): Injects tools into the generation loop.
  • wrap_generate(self, params, ctx, next_fn): Wraps each turn of the tool loop: one model call plus the tool calls it requested. Later turns run inside its next_fn.
  • wrap_model(self, params, ctx, next_fn): Wraps each model call.
  • wrap_tool(self, params, ctx, next_fn): Wraps each tool call.

Here is a custom middleware that logs model requests and responses:

from pydantic import BaseModel
from genkit import Genkit, ModelResponse
from genkit.middleware import Middleware, MiddlewareContext, ModelHookParams
from genkit_google_genai import GoogleAI
ai = Genkit(plugins=[GoogleAI()])
class LoggerConfig(BaseModel):
verbose: bool = False
@ai.middleware
class LoggerMiddleware(Middleware[LoggerConfig]):
name = 'logger_middleware'
description = 'Logs requests and responses'
async def wrap_model(
self,
params: ModelHookParams,
ctx: MiddlewareContext,
next_fn,
) -> ModelResponse:
if self.config.verbose:
print(f'Request: {params.request}')
resp = await next_fn(params, ctx)
if self.config.verbose:
print(f'Response: {resp}')
return resp

Pass an instance in use:

response = await ai.generate(
model=GoogleAI.gemini('gemini-flash-latest'),
prompt='Hello',
use=[LoggerMiddleware(verbose=True)],
)

Middleware in use=[...] composes left to right, with the first entry as the outermost wrapper. Each next_fn runs the next inner layer:

use=[
Retry(max_retries=3), # outer: retries the whole inner stack
Fallback(models=[GoogleAI.gemini('gemini-flash-latest')]), # inner: tries the backup on failure
]
# effective chain: Retry { Fallback { model } }

Order matters. Retry outside Fallback retries the entire fallback cascade as a unit. Swap them and Retry exhausts its attempts on the primary model before Fallback switches models.

For more examples of custom middleware, see the built-in implementations in py/packages/genkit-middleware.