Retrieval-augmented generation (RAG)

Genkit provides abstractions that help you build retrieval-augmented generation (RAG) flows, as well as plugins that provide integrations with related tools.

What is RAG?

Retrieval-augmented generation is a technique used to incorporate external sources of information into an LLM’s responses. It’s important to be able to do so because, while LLMs are typically trained on a broad body of material, practical use of LLMs often requires specific domain knowledge (for example, you might want to use an LLM to answer customers’ questions about your company’s products).

One solution is to fine-tune the model using more specific data. However, this can be expensive both in terms of compute cost and in terms of the effort needed to prepare adequate training data.

In contrast, RAG works by incorporating external data sources into a prompt at the time it’s passed to the model. For example, you could imagine the prompt, “What is Bart’s relationship to Lisa?” might be expanded (“augmented”) by prepending some relevant information, resulting in the prompt, “Homer and Marge’s children are named Bart, Lisa, and Maggie. What is Bart’s relationship to Lisa?”

This approach has several advantages:

• It can be more cost-effective because you don’t have to retrain the model.
• You can continuously update your data source and the LLM can immediately make use of the updated information.
• You now have the potential to cite references in your LLM’s responses.

On the other hand, using RAG naturally means longer prompts, and some LLM API services charge for each input token you send. Ultimately, you must evaluate the cost tradeoffs for your applications.

RAG is a very broad area and there are many different techniques used to achieve the best quality RAG. The core Genkit framework offers three main abstractions to help you do RAG:

• Indexers: keep track of your documents so relevant ones can be retrieved for a query.
• Embedders: transforms documents into a vector representation.
• Retrievers: retrieve documents from an “index”, given a query.

These definitions are broad on purpose because Genkit is un-opinionated about what an “index” is or how exactly documents are retrieved from it. Genkit only provides a Document format and everything else is defined by the retriever or indexer implementation provider.

Embedders

An embedder is a function that takes content (text, images, audio, etc.) and creates a numeric vector that encodes the semantic meaning of the original content. As mentioned above, embedders are leveraged as part of the process of indexing. However, they can also be used independently to create embeddings without an index.

Retrievers

A retriever is a concept that encapsulates logic related to any kind of document retrieval. The most popular retrieval cases typically include retrieval from vector stores. However, in Genkit a retriever can be any function that returns data.

To create a retriever, you can use one of the provided implementations or create your own.

Indexers

The index is responsible for keeping track of your documents so that you can quickly retrieve the relevant ones for a query. This is most often a vector database: it stores each document alongside its embedding, and retrieves documents whose embeddings sit close to the embedding of the query.

Before you can retrieve documents you have to ingest them. A typical ingestion pipeline does three things:

1. Split large documents into chunks, so that only the relevant portion augments your prompt and so each chunk fits comfortably in the model’s context window. Genkit does not ship a chunker; any Go text splitter works.
2. Generate an embedding for each chunk.
3. Write the chunk and its vector to the store.

Ingestion is a batch job, not something that runs per request. Run it once for a stable corpus, or on a trigger whenever the source data changes.

Go has no separate indexer action type. Indexing goes through the store handle that the plugin’s DefineRetriever returns alongside the retriever, for example *localvec.DocStore with localvec.Index.

Supported retrievers and embedders

Genkit provides retriever support through its plugin system:

Vector store	Use it when
Dev local vector store	Prototyping on one machine. Development only; do not use it in production.
Pinecone	You want a managed cloud vector database with no database to operate.
AlloyDB for PostgreSQL	Your data already lives in AlloyDB and you want pgvector search next to it.
Cloud SQL for PostgreSQL	Same, on managed PostgreSQL.
Cloud Firestore vector search	Your documents are already Firestore documents.
Vertex AI Vector Search with BigQuery	Your corpus is in BigQuery and you want Vertex AI to serve the index.
Vertex AI Vector Search with Firestore	Same, with Firestore as the document store.
Self-managed pgvector	You run your own PostgreSQL. This is a code template, not a plugin.

Embedding model support is provided through the following plugins:

Plugin	Embedders
Google Generative AI	googleai/gemini-embedding-001 and the other Gemini API text embedders.
Vertex AI	vertexai/text-embedding-004, vertexai/text-embedding-005 and others.

Defining a RAG flow

The following examples show how you could ingest a collection of restaurant menu PDF documents into a vector database and retrieve them for use in a flow that determines what food items are available. The rag sample is a runnable version of the same shape against the local vector store, and the pgvector sample is the PostgreSQL equivalent. Note: Although retriever functions are defined using Genkit, users are expected to add their own functionality to index the documents.

Prerequisites

This walkthrough registers the Vertex AI plugin, which authenticates with Application Default Credentials. Before you run it:

gcloud auth application-default login
export GOOGLE_CLOUD_PROJECT=your-project-id
export GOOGLE_CLOUD_LOCATION=us-central1

To use googleai/... models and embedders instead, register &googlegenai.GoogleAI{} and set GEMINI_API_KEY. Provider prefixes are not interchangeable: a googleai/ name only resolves if the Google AI plugin is registered, and likewise for vertexai/.

Install dependencies

In this example, we will use the textsplitter library from langchaingo and the ledongthuc/pdf PDF parsing Library:

go get github.com/tmc/langchaingo/textsplitter
go get github.com/ledongthuc/pdf

Neither is a Genkit dependency. Any Go text splitter and any PDF reader work here; these two just keep the example short.

The complete program

Everything below is one main package. Later sections walk through the pieces.

package main

import (
    "context"
    "fmt"
    "io"
    "log"

    "github.com/firebase/genkit/go/ai"
    "github.com/firebase/genkit/go/genkit"
    "github.com/firebase/genkit/go/plugins/googlegenai"
    "github.com/firebase/genkit/go/plugins/localvec"
    "github.com/firebase/genkit/go/plugins/server"
    "github.com/ledongthuc/pdf"
    "github.com/tmc/langchaingo/textsplitter"
)

func main() {
    ctx := context.Background()

    // Initialize Genkit with the Vertex AI plugin.
    g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.VertexAI{}))

    // Initialize the local vector store plugin.
    if err := localvec.Init(); err != nil {
        log.Fatal(err)
    }

    embedder := genkit.LookupEmbedder(g, "vertexai/text-embedding-004")
    if embedder == nil {
        log.Fatal("embedder vertexai/text-embedding-004 is not registered")
    }

    // Define the retriever and document store. We keep menuDocStore for the
    // indexer flow and menuPdfRetriever for the retrieval flow.
    menuDocStore, menuPdfRetriever, err := localvec.DefineRetriever(
        g,
        "menuQA",
        localvec.Config{
            Dir:      ".genkit/localvec",
            Embedder: embedder,
        },
        nil,
    )
    if err != nil {
        log.Fatal(err)
    }

    splitter := textsplitter.NewRecursiveCharacter(
        textsplitter.WithChunkSize(200),
        textsplitter.WithChunkOverlap(20),
    )

    genkit.DefineFlow(g, "indexMenu",
        func(ctx context.Context, path string) (map[string]any, error) {
            // Extract plain text from the PDF. Wrap the logic in Run so it
            // appears as a step in your traces.
            pdfText, err := genkit.Run(ctx, "extract", func() (string, error) {
                return readPDF(path)
            })
            if err != nil {
                return nil, err
            }

            // Split the text into chunks. Wrap the logic in Run so it appears
            // as a step in your traces.
            docs, err := genkit.Run(ctx, "chunk", func() ([]*ai.Document, error) {
                chunks, err := splitter.SplitText(pdfText)
                if err != nil {
                    return nil, err
                }

                var docs []*ai.Document
                for i, chunk := range chunks {
                    docs = append(docs, ai.DocumentFromText(chunk, map[string]any{
                        "id":     fmt.Sprintf("%s#%d", path, i),
                        "source": path,
                    }))
                }
                return docs, nil
            })
            if err != nil {
                return nil, err
            }

            // Add chunks to the index using the vector store.
            if err := localvec.Index(ctx, docs, menuDocStore); err != nil {
                return nil, err
            }

            return map[string]any{
                "success":          true,
                "documentsIndexed": len(docs),
            }, nil
        })

    genkit.DefineFlow(g, "menuQA",
        func(ctx context.Context, question string) (string, error) {
            // Retrieve text relevant to the user's question.
            resp, err := genkit.Retrieve(ctx, g,
                ai.WithRetriever(menuPdfRetriever),
                ai.WithConfig(&localvec.RetrieverOptions{K: 3}),
                ai.WithTextDocs(question))
            if err != nil {
                return "", err
            }
            if len(resp.Documents) == 0 {
                return "I don't have that in the indexed material.", nil
            }

            // Call Generate, including the menu information in your prompt.
            return genkit.GenerateText(ctx, g,
                ai.WithModelName("vertexai/gemini-flash-latest"),
                ai.WithDocs(resp.Documents...),
                ai.WithSystem(`
You are acting as a helpful AI assistant that can answer questions about the
food available on the menu at Genkit Grub Pub.
Use only the context provided to answer the question. If you don't know, do not
make up an answer. Do not add or change items on the menu.`),
                ai.WithPrompt(question))
        })

    // Keep the process alive so the CLI and the developer UI can reach the
    // flows. Ctrl-C stops it.
    log.Fatal(server.Start(ctx, "127.0.0.1:3400", nil))
}

// readPDF extracts plain text from a PDF. Excerpted from
// https://github.com/ledongthuc/pdf
func readPDF(path string) (string, error) {
    f, r, err := pdf.Open(path)
    if f != nil {
        defer f.Close()
    }
    if err != nil {
        return "", err
    }

    reader, err := r.GetPlainText()
    if err != nil {
        return "", err
    }

    bytes, err := io.ReadAll(reader)
    if err != nil {
        return "", err
    }

    return string(bytes), nil
}

Chunking config

The textsplitter call above configures the chunking function to return document segments of 200 characters, with an overlap between chunks of 20 characters. More chunking options for this library can be found in the langchaingo documentation.

Document metadata

ai.DocumentFromText(text string, metadata map[string]any) *ai.Document builds a document from a string. ai.Document has two fields: Content []*ai.Part and Metadata map[string]any. The local vector store persists the whole document, so whatever metadata you attach at index time comes back on every retrieved document. Use it for IDs, titles and source paths. The id key doubles as the citation marker the model sees; see Citations.

Run the indexer flow

genkit flow:run indexMenu '"menu.pdf"' -- go run .

Run this from the module directory. menu.pdf is resolved relative to that directory.

After running the indexMenu flow, the vector database will be seeded with documents and ready to be used in Genkit flows with retrieval steps.

Calling a retriever

genkit.Retrieve(ctx, g, ai.WithRetriever(r), ai.WithTextDocs(q)) is the form to reach for. It resolves the retriever through the registry and records the call as a traced step. Three variants exist and none of them is deprecated:

Call	Use it when
genkit.Retrieve(ctx, g, opts...)	Default. You have a *genkit.Genkit.
ai.Retrieve(ctx, reg, opts...)	Same call, with the registry passed explicitly, for code that has no *genkit.Genkit.
r.Retrieve(ctx, req)	You hold the ai.Retriever and want to build the ai.RetrieverRequest yourself.

You can name a registered retriever instead of holding its value. Plugin retrievers register under a provider prefix; the local vector store uses devLocalVectorStore/<name>:

resp, err := genkit.Retrieve(ctx, g,
    ai.WithRetrieverName("devLocalVectorStore/menuQA"),
    ai.WithDocs(ai.DocumentFromText(question, nil)))

When retrieval comes back empty or weak

ai.RetrieverResponse carries Documents and nothing else. There is no score field, so you cannot filter on relevance after the fact. Guard on the count before you call the model, as the menuQA flow above does:

if len(resp.Documents) == 0 {
    return "I don't have that in the indexed material.", nil
}

A relevance floor is only available where the store’s own options support one, for example a MIN_SCORE predicate in a pgvector query you write yourself. The local vector store ranks by cosine similarity internally but returns the top K documents with no scores, so K is its only relevance control.

Citations

Documents passed with ai.WithDocs land on the request’s Docs field. What happens next depends on the model:

• A model that declares Supports.Context receives the documents natively. Genkit inserts no markers, and rendering citations in your UI is up to you.
• Every other model goes through Genkit’s augment-with-context middleware, which appends the documents to the last user message as lines of the form - [<id>]: <text>.

The bracketed key comes from ai.AugmentWithContextOptions.CitationKey. With the default settings it is Metadata["ref"], else Metadata["id"], else the zero-based index of the document. Set stable IDs at index time, as the indexer flow above does, to make those citations addressable.

Only the document text and that one metadata value reach the model. The rest of Metadata stays on your side, so you do not need to duplicate document text into your prompt.

Write your own retrievers

It’s also possible to create your own retriever. This is useful if your documents are managed in a document store that is not supported in Genkit (eg: MySQL, Google Drive, etc.). The Genkit SDK provides flexible methods that let you provide custom code for fetching documents.

You can also define custom retrievers that build on top of existing retrievers in Genkit and apply advanced RAG techniques (such as reranking or prompt extension) on top.

For example, suppose you have a custom re-ranking function you want to use. The following example defines a custom retriever that applies your function to the menu retriever defined earlier:

type CustomMenuRetrieverOptions struct {
    K          int `json:"k,omitempty"`
    PreRerankK int `json:"preRerankK,omitempty"`
}

advancedMenuRetriever := genkit.DefineRetrieverAction(
    g, "custom/advancedMenuRetriever", nil,
    func(ctx context.Context, req *ai.RetrieverRequest, opts *CustomMenuRetrieverOptions) (*ai.RetrieverResponse, error) {
        // The config type parameter is a pointer, so it is nil when the caller
        // sends no config at all.
        if opts == nil {
            opts = &CustomMenuRetrieverOptions{}
        }
        // Set fields to default values when the caller left them unset.
        if opts.K == 0 {
            opts.K = 3
        }
        if opts.PreRerankK == 0 {
            opts.PreRerankK = 10
        }

        // Call the retriever as in the simple case.
        resp, err := genkit.Retrieve(ctx, g,
            ai.WithRetriever(menuPdfRetriever),
            ai.WithDocs(req.Query),
            ai.WithConfig(&localvec.RetrieverOptions{K: opts.PreRerankK}),
        )
        if err != nil {
            return nil, err
        }

        // Re-rank the returned documents using your custom function.
        // Note: Genkit does not currently provide a built-in reranker;
        // you would implement this logic yourself
        rerankedDocs := rerank(resp.Documents)
        resp.Documents = rerankedDocs[:opts.K]

        return resp, nil
    },
)

genkit.DefineRetrieverAction infers the retriever’s config schema from the type of its last function parameter and validates every request against it, so your function receives a typed value instead of an any it has to assert. A request that carries a key the schema does not allow fails with INVALID_ARGUMENT before your code runs.

Note: The rerank function is a placeholder for your own logic and is not provided by the Genkit framework.

The request your retriever receives

ai.RetrieverRequest has exactly two fields:

• Query *ai.Document: the query, expressed as a document so you can hand it straight to an embedder.
• Options any: the raw per-request config, as set by ai.WithConfig.

Retrievers defined with genkit.DefineRetrieverAction get that config decoded into the typed last parameter and should ignore req.Options, which the framework normalizes to the same value.

genkit.DefineRetriever(g, name, opts, fn) is the older non-generic form. Its callback is ai.RetrieverFunc, func(ctx context.Context, req *ai.RetrieverRequest) (*ai.RetrieverResponse, error), so it has to read and type-assert req.Options itself. It is deprecated in favor of DefineRetrieverAction.

Next steps

• Learn about tool calling to give your RAG system access to external APIs and functions

• Explore full-stack agents for coordinating multiple AI agents with RAG capabilities

• See the evaluation guide for testing and improving your RAG system’s performance

• Check out the vector database plugins for production-ready RAG implementations