Middleware
Genkit allows you to use middleware to modify the behavior of generate() calls. Middleware can be used for various purposes, such as retrying failed requests, falling back to different models, or injecting tools and context.
You can use pre-packaged middleware or build your own custom middleware.
Installation
Section titled “Installation”The official Genkit middleware for Python is available in the genkit-middleware package.
uv add genkit-middlewarePassing a middleware instance in use=[...] works without any setup. Register the Middleware plugin to also list the built-ins in the Dev UI and to name them in a .prompt file’s use::
from genkit import Genkitfrom genkit_google_genai import GoogleAIfrom genkit_middleware import Middleware
ai = Genkit( plugins=[ GoogleAI(), Middleware(), ],)Available middleware
Section titled “Available middleware”The genkit-middleware package provides several middleware classes out of the box.
1. FileSystem middleware (Filesystem)
Section titled “1. FileSystem middleware (Filesystem)”Grants the model access to a root directory by injecting list_files and
read_file. When allow_write_access=True (default False), it also injects
write_file and edit_file. All paths are restricted to that root.
from genkit import Genkitfrom genkit_google_genai import GoogleAIfrom genkit_middleware import Filesystem
ai = Genkit(plugins=[GoogleAI()])
response = await ai.generate( model=GoogleAI.gemini('gemini-flash-latest'), prompt='Create a hello world node app in the workspace', use=[ Filesystem(root_dir='./workspace', allow_write_access=True), ],)Configuration options:
root_dir(required): The root directory to which all filesystem operations are restricted.allow_write_access(optional): IfTrue, adds write/edit tools (defaults toFalse— read-only).tool_name_prefix(optional): Prefix to add to the name of the injected tools.
2. Skills middleware (Skills)
Section titled “2. Skills middleware (Skills)”Scans each directory in skill_paths for subfolders that contain a SKILL.md file, reads name and description from its YAML frontmatter, and lists the skills in the system prompt. It also adds a use_skill tool that loads a skill’s full SKILL.md on demand.
from genkit import Genkitfrom genkit_google_genai import GoogleAIfrom genkit_middleware import Skills
ai = Genkit(plugins=[GoogleAI()], model=GoogleAI.gemini('gemini-flash-latest'))
response = await ai.generate( prompt='How do I run tests in this repo?', use=[ Skills(skill_paths=['./skills']), ],)Configuration options:
skill_paths(optional): Paths to directories containing skills (defaults to['skills']).
3. Tool approval middleware (ToolApproval)
Section titled “3. Tool approval middleware (ToolApproval)”Restricts execution of tools to an approved list. If the model attempts to call an unapproved tool, generation finishes interrupted (FinishReason.INTERRUPTED) so you can prompt the user for confirmation before resuming.
from genkit import FinishReason, Genkitfrom genkit_google_genai import GoogleAIfrom genkit_middleware import ToolApprovalfrom pydantic import BaseModel, Field
ai = Genkit(plugins=[GoogleAI()], model=GoogleAI.gemini('gemini-flash-latest'))
class WriteFileInput(BaseModel): path: str = Field(description='File path') content: str = Field(description='File contents')
@ai.tool()async def write_file_tool(input: WriteFileInput) -> str: """Write a text file.""" # Persist input.content to input.path in a real app. return f'Wrote {input.path}'
# 1. Initial attemptresponse = await ai.generate( prompt='write a file', tools=[write_file_tool], use=[ ToolApproval(allowed_tools=[]), # Empty list means all tool calls trigger interrupt ],)
if response.finish_reason == FinishReason.INTERRUPTED: interrupt = response.interrupts[0]
# 2. Ask user for approval, then build a restart part with approval metadata approved_part = interrupt.restart( resumed_metadata={'tool_approved': True}, )
# 3. Resume execution response = await ai.generate( messages=response.messages, tools=[write_file_tool], use=[ ToolApproval(allowed_tools=[]), ], resume_restart=approved_part, ) print(response.text)Configuration options:
allowed_tools(optional): List of approved tool names that can run without interruption.
4. Retry middleware (Retry)
Section titled “4. Retry middleware (Retry)”Automatically retries failed model generations on transient error codes (like RESOURCE_EXHAUSTED, UNAVAILABLE) using exponential backoff with jitter.
from genkit import Genkitfrom genkit_google_genai import GoogleAIfrom genkit_middleware import Retry
ai = Genkit(plugins=[GoogleAI()])
response = await ai.generate( model=GoogleAI.gemini('gemini-pro-latest'), prompt='Analyze quarterly variance across all regional ledgers.', use=[ Retry( max_retries=3, initial_delay_ms=1000, backoff_factor=2.0, ), ],)Configuration options:
max_retries(optional): The maximum number of times to retry a failed request (default: 3).statuses(optional): An array of status names that should trigger a retry (default:['UNAVAILABLE', 'DEADLINE_EXCEEDED', 'RESOURCE_EXHAUSTED', 'ABORTED', 'INTERNAL']).initial_delay_ms(optional): The initial delay between retries in milliseconds (default: 1000).max_delay_ms(optional): The maximum delay between retries in milliseconds (default: 60000).backoff_factor(optional): The factor by which the delay increases after each retry (exponential backoff, default: 2.0).no_jitter(optional): Whether to disable jitter on the delay (default:False).
A GenkitError is retried only when its status is in statuses; any other exception is retried. If a GenkitError carries retry_after_ms, Retry waits at least that long, capped at max_delay_ms.
5. Fallback middleware (Fallback)
Section titled “5. Fallback middleware (Fallback)”Automatically switches to a different model if the primary model fails on a specific set of error codes. Useful for falling back to a smaller/faster model when a large model exceeds quota limits.
from genkit import Genkitfrom genkit_google_genai import GoogleAIfrom genkit_middleware import Fallback
ai = Genkit(plugins=[GoogleAI()])
response = await ai.generate( model=GoogleAI.gemini('gemini-pro-latest'), prompt='Summarize the incident timeline.', use=[ Fallback( models=[GoogleAI.gemini('gemini-flash-latest')], # try flash if pro fails statuses=['RESOURCE_EXHAUSTED'], ), ],)Configuration options:
models(required): A list of model names orModelRefvalues to try in order.statuses(optional): A list of status names that should trigger a fallback (default:['UNAVAILABLE', 'DEADLINE_EXCEEDED', 'RESOURCE_EXHAUSTED', 'ABORTED', 'INTERNAL', 'NOT_FOUND', 'UNIMPLEMENTED']).
Each fallback model must resolve on the app, so register its plugin. A string entry runs with that model’s default config; a ModelRef entry runs with the ref’s own config. The primary request’s config is not carried over. Errors that aren’t a GenkitError with a listed status skip the fallbacks.
6. Artifacts middleware (Artifacts)
Section titled “6. Artifacts middleware (Artifacts)”Gives the model read_artifact and write_artifact tools over the active agent session’s artifacts, and lists the current artifacts in the system prompt each turn. Artifacts depends on agent sessions, so it lives in genkit_middleware.exp and only works when passed to an agent defined with genkit.exp.Genkit. The Middleware() plugin doesn’t register it.
from genkit.exp import Genkitfrom genkit_google_genai import GoogleAIfrom genkit_middleware.exp import Artifacts
ai = Genkit(plugins=[GoogleAI()])
builder = ai.define_agent( name='builder', model=GoogleAI.gemini('gemini-flash-latest'), system='You are a code generator. Use write_artifact to create files.', use=[Artifacts()],)
chat = builder.chat()await chat.send('Write a README.md for a pizza ordering CLI.')print([artifact.name for artifact in chat.artifacts])# => ['README.md']Configuration options:
readonly(optional): IfTrue, onlyread_artifactis provided (default:False). Use it on an orchestrator that reads what its sub-agents produced; see Multi-agent delegation.
Building your own custom middleware
Section titled “Building your own custom middleware”Implement custom middleware by subclassing Middleware from genkit.middleware and setting a unique name class attribute. Decorating the class with @ai.middleware registers it on the Genkit instance so it appears in the Developer UI.
The base class shares its name with the genkit_middleware.Middleware plugin. If a module needs both, alias one: from genkit_middleware import Middleware as MiddlewarePlugin.
Middleware can override four hooks:
tools(self, ctx): Injects tools into the generation loop.wrap_generate(self, params, ctx, next_fn): Wraps each turn of the tool loop: one model call plus the tool calls it requested. Later turns run inside itsnext_fn.wrap_model(self, params, ctx, next_fn): Wraps each model call.wrap_tool(self, params, ctx, next_fn): Wraps each tool call.
Here is a custom middleware that logs model requests and responses:
from pydantic import BaseModel
from genkit import Genkit, ModelResponsefrom genkit.middleware import Middleware, MiddlewareContext, ModelHookParamsfrom genkit_google_genai import GoogleAI
ai = Genkit(plugins=[GoogleAI()])
class LoggerConfig(BaseModel): verbose: bool = False
@ai.middlewareclass LoggerMiddleware(Middleware[LoggerConfig]): name = 'logger_middleware' description = 'Logs requests and responses'
async def wrap_model( self, params: ModelHookParams, ctx: MiddlewareContext, next_fn, ) -> ModelResponse: if self.config.verbose: print(f'Request: {params.request}')
resp = await next_fn(params, ctx)
if self.config.verbose: print(f'Response: {resp}')
return respPass an instance in use:
response = await ai.generate( model=GoogleAI.gemini('gemini-flash-latest'), prompt='Hello', use=[LoggerMiddleware(verbose=True)],)Composition order
Section titled “Composition order”Middleware in use=[...] composes left to right, with the first entry as the outermost wrapper. Each next_fn runs the next inner layer:
use=[ Retry(max_retries=3), # outer: retries the whole inner stack Fallback(models=[GoogleAI.gemini('gemini-flash-latest')]), # inner: tries the backup on failure]# effective chain: Retry { Fallback { model } }Order matters. Retry outside Fallback retries the entire fallback cascade as a unit. Swap them and Retry exhausts its attempts on the primary model before Fallback switches models.
For more examples of custom middleware, see the built-in implementations in py/packages/genkit-middleware.