Skip to content
Genkit

Pause generation using interrupts

Interrupts are a special kind of tool that can pause the LLM generation-and-tool-calling loop to return control back to you. When you’re ready, you can then resume generation by sending replies that the LLM processes for further generation.

The most common uses for interrupts fall into a few categories:

  • Human-in-the-Loop: Enabling the user of an interactive AI to clarify needed information or confirm the LLM’s action before it is completed, providing a measure of safety and confidence.
  • Async Processing: Starting an asynchronous task that can only be completed out-of-band, such as sending an approval notification to a human reviewer or kicking off a long-running background process.
  • Exit from an Autonomous Task: Providing the model a way to mark a task as complete, in a workflow that might iterate through a long series of tool calls.

Before you begin

Section titled “Before you begin”

All of the examples documented here assume that you have already set up a project with Genkit dependencies installed. If you want to run the code examples on this page, first complete the steps in the Get started guide.

Before diving too deeply, you should also be familiar with the following concepts:

Overview of interrupts

Section titled “Overview of interrupts”

At a high level, this is what an interrupt looks like when interacting with an LLM:

  1. The calling application prompts the LLM with a request. The prompt includes a list of tools, including at least one for an interrupt that the LLM can use to generate a response.
  2. The LLM generates either a complete response or a tool call request in a specific format. To the LLM, an interrupt call looks like any other tool call.
  3. If the LLM calls an interrupting tool, the Genkit library automatically pauses generation rather than immediately passing responses back to the model for additional processing.
  4. The developer checks whether an interrupt call is made, and performs whatever task is needed to collect the information needed for the interrupt response.
  5. The developer resumes generation by passing an interrupt response to the model. This action triggers a return to Step 2.

Defining tools with interrupts

Section titled “Defining tools with interrupts”

Use genkit.DefineTool() and call ai.InterruptWith() to trigger a strongly-typed interrupt. Define a struct to carry the interrupt metadata:

// InterruptMetadata carries information about why the tool was interrupted.
type InterruptMetadata struct {
    Reason  string   `json:"reason"`
    Choices []string `json:"choices,omitempty"`
}


askQuestion := genkit.DefineTool(g, "askQuestion",
    "use this to ask the user a clarifying question",
    func(ctx *ai.ToolContext, input QuestionInput) (string, error) {
        return "", ai.InterruptWith(ctx, InterruptMetadata{
            Reason:  "need_clarification",
            Choices: input.Choices,
        })
    },
)

Use interrupts

Section titled “Use interrupts”

Interrupts are passed into the WithTools() option when generating content, just like other types of tools:

resp, err := genkit.Generate(ctx, g,
    ai.WithPrompt("Ask me a movie trivia question."),
    ai.WithTools(askQuestion),
)

Genkit immediately returns a response on receipt of an interrupt tool call.

Respond to interrupts

Section titled “Respond to interrupts”

Check the response for interrupts and handle them. Use ai.InterruptAs() to extract strongly-typed metadata from the interrupt:

// Check if generation was interrupted
if resp.FinishReason == ai.FinishReasonInterrupted {
    for _, interrupt := range resp.Interrupts() {
        if meta, ok := ai.InterruptAs[InterruptMetadata](interrupt); ok {
            fmt.Printf("Interrupt reason: %s\n", meta.Reason)
        }
    }
}

Responding to an interrupt is done using tool.RespondWith() and ai.WithToolResponses() on a subsequent Generate call, passing in the existing message history:

resp, err := genkit.Generate(ctx, g,
    ai.WithPrompt("Help me plan a backyard BBQ."),
    ai.WithSystem("Ask clarifying questions until you have a complete solution."),
    ai.WithTools(askQuestion),
)
if err != nil {
    return err
}


for resp.FinishReason == ai.FinishReasonInterrupted {
    var responses []*ai.Part


    for _, interrupt := range resp.Interrupts() {
        // Use RespondWith to provide a strongly-typed response
        part, err := askQuestion.RespondWith(interrupt, getUserAnswer(interrupt))
        if err != nil {
            return err
        }
        responses = append(responses, part)
    }


    resp, err = genkit.Generate(ctx, g,
        ai.WithMessages(resp.History()...),
        ai.WithTools(askQuestion),
        ai.WithToolResponses(responses...),
    )
    if err != nil {
        return err
    }
}


fmt.Println(resp.Text())

Tools with restartable interrupts

Section titled “Tools with restartable interrupts”

Another common pattern is the need to confirm an action that the LLM suggests before actually performing it. For example, a payments app might want the user to confirm certain kinds of transfers.

Define a restartable tool

Section titled “Define a restartable tool”

Use ctx.IsResumed() to check if the tool is being restarted after an interrupt:

type TransferInput struct {
    ToAccount string  `json:"toAccount"`
    Amount    float64 `json:"amount"`
}


type TransferOutput struct {
    Status     string  `json:"status"`
    Message    string  `json:"message,omitempty"`
    NewBalance float64 `json:"newBalance,omitempty"`
}


type TransferInterrupt struct {
    Reason    string  `json:"reason"` // "insufficient_balance" or "confirm_large"
    ToAccount string  `json:"toAccount"`
    Amount    float64 `json:"amount"`
    Balance   float64 `json:"balance,omitempty"`
}


transferMoney := genkit.DefineTool(g, "transferMoney",
    "Transfers money to another account.",
    func(ctx *ai.ToolContext, input TransferInput) (TransferOutput, error) {
        // Check for insufficient balance
        if input.Amount > accountBalance {
            return TransferOutput{}, ai.InterruptWith(ctx, TransferInterrupt{
                Reason:  "insufficient_balance",
                ToAccount: input.ToAccount,
                Amount:    input.Amount,
                Balance:   accountBalance,
            })
        }


        // Require confirmation for large transfers (only on first execution)
        if !ctx.IsResumed() && input.Amount > 100 {
            return TransferOutput{}, ai.InterruptWith(ctx, TransferInterrupt{
                Reason:    "confirm_large",
                ToAccount: input.ToAccount,
                Amount:    input.Amount,
            })
        }


        // Execute the transfer
        accountBalance -= input.Amount
        return TransferOutput{"completed", "Transfer successful", accountBalance}, nil
    },
)

Restart tools after interruption

Section titled “Restart tools after interruption”

Use tool.RestartWith() and ai.WithToolRestarts() to re-execute an interrupted tool. You can optionally replace the input using ai.WithNewInput():

resp, err := genkit.Generate(ctx, g,
    ai.WithPrompt("Transfer $200 to account ABC123"),
    ai.WithTools(transferMoney),
)
if err != nil {
    return err
}


for resp.FinishReason == ai.FinishReasonInterrupted {
    var restarts, responses []*ai.Part


    for _, interrupt := range resp.Interrupts() {
        meta, ok := ai.InterruptAs[TransferInterrupt](interrupt)
        if !ok {
            continue
        }


        switch meta.Reason {
        case "insufficient_balance":
            if userConfirms("Transfer $%.2f instead?", meta.Balance) {
                // Restart with adjusted amount using WithNewInput
                part, err := transferMoney.RestartWith(interrupt,
                    ai.WithNewInput(TransferInput{meta.ToAccount, meta.Balance}))
                if err != nil {
                    return err
                }
                restarts = append(restarts, part)
            } else {
                // Provide a response directly without re-running the tool
                part, err := transferMoney.RespondWith(interrupt,
                    TransferOutput{"cancelled", "Transfer cancelled by user.", accountBalance})
                if err != nil {
                    return err
                }
                responses = append(responses, part)
            }


        case "confirm_large":
            if userConfirms("Confirm transfer of $%.2f?", meta.Amount) {
                // Simple restart (re-run with same input)
                part, err := transferMoney.RestartWith(interrupt)
                if err != nil {
                    return err
                }
                restarts = append(restarts, part)
            } else {
                part, err := transferMoney.RespondWith(interrupt,
                    TransferOutput{"cancelled", "Transfer cancelled by user.", accountBalance})
                if err != nil {
                    return err
                }
                responses = append(responses, part)
            }
        }
    }


    resp, err = genkit.Generate(ctx, g,
        ai.WithMessages(resp.History()...),
        ai.WithTools(transferMoney),
        ai.WithToolRestarts(restarts...),
        ai.WithToolResponses(responses...),
    )
    if err != nil {
        return err
    }
}


fmt.Println(resp.Text())

Access original input after replacement

Section titled “Access original input after replacement”

When you use ai.WithNewInput(), you can access the original input inside the tool using ai.OriginalInputAs():

transferMoney := genkit.DefineTool(g, "transferMoney",
    "Transfers money to another account.",
    func(ctx *ai.ToolContext, input TransferInput) (TransferOutput, error) {
        // ... interrupt logic ...


        accountBalance -= input.Amount
        message := fmt.Sprintf("Transferred $%.2f to %s", input.Amount, input.ToAccount)


        // Check if input was replaced and include original amount in message
        if orig, ok := ai.OriginalInputAs[TransferInput](ctx); ok {
            message = fmt.Sprintf("Transferred $%.2f to %s (adjusted from $%.2f)",
                input.Amount, input.ToAccount, orig.Amount)
        }


        return TransferOutput{"completed", message, accountBalance}, nil
    },
)