Middleware

Genkit allows you to use middleware to modify the behavior of generate() calls. Middleware can be used for various purposes, such as retrying failed requests, falling back to different models, or injecting tools and context.

You can use pre-packaged middleware or build your own custom middleware.

The official Genkit middleware for JavaScript is available in the @genkit-ai/middleware package.

Installation

npm install @genkit-ai/middleware
# or
yarn add @genkit-ai/middleware
# or
pnpm add @genkit-ai/middleware

Available middleware

The @genkit-ai/middleware package provides several useful middleware options out of the box. This list represents the middleware built and maintained by the Genkit team, but there may also be community-built middleware available.

1. FileSystem middleware (filesystem)

Grants the model access to the local filesystem by injecting standard file manipulation tools (list_files, read_file, write_file, search_and_replace). All operations are safely restricted to a specified root directory.

import { genkit } from 'genkit';
import { filesystem } from '@genkit-ai/middleware';

const ai = genkit({ ... });

const response = await ai.generate({
  model: googleAI.model('gemini-flash-latest'),
  prompt: 'Create a hello world node app in the workspace',
  use: [
    filesystem({ rootDirectory: './workspace' })
  ]
});

Configuration options:

• rootDirectory (required): The root directory to which all filesystem operations are restricted.
• allowWriteAccess (optional): If true, allows write access to the filesystem (defaults to false).
• toolNamePrefix (optional): Prefix to add to the name of the injected tools.

2. Skills middleware (skills)

Automatically scans a directory for SKILL.md files (and their YAML frontmatter) and injects them into the system prompt. It also provides a use_skill tool the model can use to retrieve more specific skills on demand.

import { genkit } from 'genkit';
import { skills } from '@genkit-ai/middleware';

const ai = genkit({ ... });

const response = await ai.generate({
  prompt: 'How do I run tests in this repo?',
  use: [
    skills({ skillPaths: ['./skills'] })
  ]
});

3. Tool approval middleware (toolApproval)

Restricts execution of tools to an approved list. If the model attempts to call an unapproved tool, it throws a ToolInterruptError allowing you to prompt the user for manual confirmation before resuming.

import { genkit, restartTool } from 'genkit';
import { toolApproval } from '@genkit-ai/middleware';

const ai = genkit({ ... });

// 1. Initial attempt
const response = await ai.generate({
  prompt: 'write a file',
  tools: [writeFileTool],
  use: [
    toolApproval({ approved: [] }) // Empty list means call triggers interrupt
  ]
});

if (response.finishReason === 'interrupted') {
  const interrupt = response.interrupts[0];

  // 2. Ask user for approval, then recreate the tool request with approval
  const approvedPart = restartTool(interrupt, { toolApproved: true });

  // 3. Resume execution
  const resumedResponse = await ai.generate({
    messages: response.messages,
    resume: { restart: [approvedPart] },
    use: [
      toolApproval({ approved: [] })
    ]
  });
}

4. Retry middleware (retry)

Automatically retries failed model generations on transient error codes (like RESOURCE_EXHAUSTED, UNAVAILABLE) using exponential backoff with jitter.

import { genkit } from 'genkit';
import { retry } from '@genkit-ai/middleware';

const ai = genkit({ ... });

const response = await ai.generate({
  model: googleAI.model('gemini-pro-latest'),
  prompt: 'Heavy reasoning task...',
  use: [
    retry({
      maxRetries: 3,
      initialDelayMs: 1000,
      backoffFactor: 2
    })
  ]
});

Configuration options:

• maxRetries (optional): The maximum number of times to retry a failed request (default: 3).
• statuses (optional): An array of StatusName values that should trigger a retry (default: ['UNAVAILABLE', 'DEADLINE_EXCEEDED', 'RESOURCE_EXHAUSTED', 'ABORTED', 'INTERNAL']).
• initialDelayMs (optional): The initial delay between retries in milliseconds (default: 1000).
• maxDelayMs (optional): The maximum delay between retries in milliseconds (default: 60000).
• backoffFactor (optional): The factor by which the delay increases after each retry (exponential backoff, default: 2).
• noJitter (optional): Whether to disable jitter on the delay (default: false).

5. Fallback middleware (fallback)

Automatically switches to a different model if the primary model fails on a specific set of error codes. Useful for falling back to a smaller/faster model when a large model exceeds quota limits.

import { genkit } from 'genkit';
import { fallback } from '@genkit-ai/middleware';

const ai = genkit({ ... });

const response = await ai.generate({
  model: googleAI.model('gemini-pro-latest'),
  prompt: 'Try the pro model first...',
  use: [
    fallback({
      models: [googleAI.model('gemini-flash-latest')], // try flash if pro fails
      statuses: ['RESOURCE_EXHAUSTED']
    })
  ]
});

Configuration options:

• models (required): An array of model references to try in order.
• statuses (optional): An array of StatusName values that should trigger a fallback (default: ['UNAVAILABLE', 'DEADLINE_EXCEEDED', 'RESOURCE_EXHAUSTED', 'ABORTED', 'INTERNAL', 'NOT_FOUND', 'UNIMPLEMENTED']).
• isolateConfig (optional): If true, the fallback model will not inherit the original request’s configuration (default: false).

6. Agents middleware (agents)

Lets an orchestrator delegate to sub-agents. It adds one delegate_to_<agent> tool per configured agent and lists the agents in the system prompt. The continue_task tool picks up failed, aborted, or finished tasks of sub-agents that have a session store. With async: true, delegations can also run in the background.

import { agents } from '@genkit-ai/middleware';

const orchestrator = ai.defineAgent({
  name: 'orchestrator',
  system: 'Delegate research to the researcher and coding to the coder.',
  use: [agents({ agents: ['researcher', 'coder'], async: true })],
});

See Multi-agent delegation for the options, background delegation, and task continuation.

7. Artifacts middleware (artifacts)

Gives the model read_artifact and write_artifact tools over the session’s artifacts, and lists the current artifacts in the system prompt each turn. Pass readonly: true to provide only read_artifact, for example on an orchestrator that reads what its sub-agents produced.

import { artifacts } from '@genkit-ai/middleware';

const builder = ai.defineAgent({
  name: 'builder',
  system: 'You are a code generator. Use write_artifact to create files.',
  use: [artifacts()],
});

8. Context compression middleware (contextCompression)

Keeps a conversation inside a token budget through long chats and tool loops. When the context grows past maxInputTokens, the middleware compresses what the model receives, cheapest strategy first, while the history you get back keeps every message.

import { genkit } from 'genkit';
import { contextCompression, resolveCompressedHistory } from '@genkit-ai/middleware';

const ai = genkit({ ... });

const response = await ai.generate({
  model: googleAI.model('gemini-flash-latest'),
  prompt: 'Research this topic thoroughly.',
  tools: [searchTool],
  use: [
    contextCompression({
      maxInputTokens: 80000,
      deduplicateToolResponses: { matchBy: 'name-and-input' },
      toolResponses: { maxChars: 2000, preserveRecent: 2 },
      summarize: {
        model: googleAI.model('gemini-flash-lite-latest'),
        preserveRecent: 6,
      },
    }),
  ],
});

// The full transcript, with compression state in message metadata:
console.log(response.messages.length);

// The messages the model actually received:
const active = resolveCompressedHistory(response.messages);

Compression runs when the estimated tokens, or the input tokens the model reported for the previous call, exceed maxInputTokens, or when the conversation holds more than maxMessages messages. The strategies run in order:

1. deduplicateToolResponses replaces older duplicate tool responses with a short notice.
2. toolResponses truncates older tool responses, leaving the newest tool messages intact. The same pass cuts any single tool response over maxToolResponseChars with a [TRUNCATED: ...] marker. That cap applies on every turn, whether or not compression is triggered.
3. summarize has a model fold older messages into a summary, sized so that the system messages, the summary, and the kept messages fit within maxMessages. It is skipped when the cheaper strategies already saved skipSummarizationThreshold of the context and brought it under budget.
4. Only if nothing was summarized, older non-system messages are dropped, keeping the history starting on a user turn: down to maxMessages, or down to preserveRecent if the context is still over budget. By default a notice tells the model that context was removed.

The model gets a view, the history keeps the transcript. By default (preserveOriginalMessages: true), compression state is recorded in message metadata under contextCompression rather than replacing messages. Later turns build on earlier compactions, so compression carries across generate() calls and agent sessions as long as you pass the history forward. A history that the Go middleware compressed resolves the same way.

Order. List contextCompression after middleware that adds context (filesystem, skills, artifacts, agents) so it accounts for what they add.

Configuration options:

• maxInputTokens (optional): Compress when the context exceeds this many tokens (default: Infinity).
• maxMessages (optional): Compress when the conversation holds more than this many messages, and trim it back to this many. The newest turn is always kept, even when the system messages and the truncation notice already fill the cap.
• preserveRecent (optional): How many of the newest non-system messages are kept when older ones are dropped, and the default window for summarize (default: 4).
• preserveSystem (optional): Always keep system messages (default: true).
• maxToolResponseChars (optional): Hard cap on the characters of any one tool response, applied on every turn (default: 400000). <= 0 or Infinity disables it.
• deduplicateToolResponses (optional): matchBy is 'name-and-input' (default) or 'name-only'; keepRecent is how many of the newest responses per group stay intact (default: 1); notice replaces the default notice.
• toolResponses (optional): maxChars is the most characters an older tool response keeps; preserveRecent is how many of the newest tool messages stay intact (default: 2).
• summarize (optional): model (required) writes the summaries; preserveRecent is how many of the newest messages stay out of the summary (default: the top-level preserveRecent when set, 6 otherwise); prompt replaces the default prompt, with {conversation} standing in for the messages to summarize.
• skipSummarizationThreshold (optional): Skip summarize when the cheaper strategies saved at least this fraction (0..1) of the context and brought it under maxInputTokens.
• insertTruncationNotice / truncationNotice (optional): Whether to insert a notice when messages are dropped (default: true), and its text.
• preserveOriginalMessages (optional): Keep the original messages in request.messages and response.messages (default: true). Set to false to overwrite them with the compressed view.

Building your own custom middleware

You can implement your own custom middleware to extend Genkit’s functionality. Genkit provides a generateMiddleware helper to create structured middleware with configuration schemas.

Middleware can intercept different phases of execution by providing hooks:

• model: Intercepts the call to the model.
• tool: Intercepts tool execution.
• generate: Intercepts the high-level generation loop.

Here is an example of a custom middleware that logs requests and responses:

import { generateMiddleware, z } from 'genkit';

export const loggerMiddleware = generateMiddleware(
  {
    name: 'loggerMiddleware',
    description: 'Logs requests and responses',
    configSchema: z.object({
      verbose: z.boolean().optional(),
    }),
  },
  ({ config, ai }) => {
    return {
      model: async (req, ctx, next) => {
        if (config?.verbose) {
          console.log('Request:', JSON.stringify(req));
        }
        const resp = await next(req, ctx);
        if (config?.verbose) {
          console.log('Response:', JSON.stringify(resp));
        }
        return resp;
      },
    };
  },
);

To use it:

const response = await ai.generate({
  model: googleAI.model('gemini-flash-latest'),
  prompt: 'Hello',
  use: [loggerMiddleware({ verbose: true })],
});

For more complex examples of building custom middleware, you can refer to the source code of the built-in middleware in the Genkit GitHub repository.