Passing information through context

There are different categories of information that a developer working with an LLM may be handling simultaneously:

• Input: Information that is directly relevant to guide the LLM’s response for a particular call. An example of this is the text that needs to be summarized.
• Generation Context: Information that is relevant to the LLM, but isn’t specific to the call. An example of this is the current time or a user’s name.
• Execution Context: Information that is important to the code surrounding the LLM call but not to the LLM itself. An example of this is a user’s current auth token.

Genkit provides a consistent context object that can propagate generation and execution context throughout the process. This context is made available to all actions including tools.

Context is automatically propagated to all actions called within the scope of execution: Context passed to the generate() method is available to tools called within the generation loop.

Why is context important?

As a best practice, you should provide the minimum amount of information to the LLM that it needs to complete a task. This is important for multiple reasons:

• The less extraneous information the LLM has, the more likely it is to perform well at its task.
• If an LLM needs to pass around information like user or account IDs to tools, it can potentially be tricked into leaking information.

Context gives you a side channel of information that can be used by any of your code but doesn’t necessarily have to be sent to the LLM. As an example, it can allow you to restrict tool queries to the current user’s available scope.

Context structure

Context must be a Map<String, dynamic>, but its properties are yours to decide.

One of the most common uses of context is to store information about the current user. We recommend adding auth context in the following format:

{
  'auth': {
    'uid': "...", // the user's unique identifier
    'token': {...}, // the decoded claims of a user's id token
    'rawToken': "...", // the user's raw encoded id token
    // ...any other fields
  }
}

The context object can store any information that you might need to know somewhere else in the flow of execution.

Use context in an action

To use context within an action, you can access the context property of the helper object (second argument) supplied to your function definition:

// Define a schema for user notes
@Schema()
class UserNote {
  final String title;
  final String content;
  UserNote({required this.title, required this.content});
}

final searchNotes = ai.defineTool(
  name: 'searchNotes',
  description: "search the current user's notes for info",
  inputSchema: .string(),
  outputSchema: .list(UserNote.$schema),
  fn: (query, ctx) async {
    final auth = ctx.context?['auth'];
    if (auth == null || auth['uid'] == null) {
      throw Exception("Must be called by a signed-in user.");
    }
    return searchUserNotes(auth['uid'], query);
  },
);

Provide context at runtime

To provide context to an action, you pass the context map as a parameter when calling the action.

final response = await ai.generate(
  prompt: "Find references to ocelots in my notes.",
  // the context will propagate to tool calls
  toolNames: ['searchNotes'],
  context: {'auth': currentUser},
);

Context propagation

By default, when you provide context it is automatically propagated to all actions called as a result of your original call. If your generation calls tools, the same context is provided.