Recipe · Knowledge retrieval

Wire RAG into an ElevenLabs Agent

Connect external documents to your ElevenLabs Conversational AI agent so it retrieves relevant context before answering — and attributes answers to source titles.

Time: ~20 min Difficulty: intermediate Companion skill: add-rag-to-elevenlabs-agent

Why RAG

An ElevenLabs agent without RAG answers from its training data. That is fine for general knowledge. It is wrong for specific knowledge: your product docs, your pricing, your runbooks, your blog posts. The agent will hallucinate confidently about things it was never trained on.

Retrieval-augmented generation fixes this. Before the agent answers, it queries a vectorized knowledge base of your documents, pulls the relevant chunks, and grounds its response in what you actually wrote. The result is specific, attributable, and updatable.

What you'll build

Prerequisites

Step 1 — Create knowledge-base entries

Each document becomes a knowledge-base entry via the ElevenLabs API. You can do this manually in the dashboard for a quick test, but for a real pipeline you'll script it.

First, create a knowledge base document from a URL:

curl -X POST https://api.elevenlabs.io/v1/convai/knowledge-base \
  -H "xi-api-key: $ELEVENLABS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Product Docs",
    "type": "url",
    "url": "https://your-site.com/docs/"
  }'

Or upload a file directly:

curl -X POST https://api.elevenlabs.io/v1/convai/knowledge-base \
  -H "xi-api-key: $ELEVENLABS_API_KEY" \
  -H "Content-Type: multipart/form-data" \
  -F "name=Runbook" \
  -F "file=@/path/to/runbook.pdf"

Save the returned id for each entry — you'll attach it to the agent in Step 3.

Step 2 — Trigger vectorization

ElevenLabs needs to vectorize the knowledge base before the agent can query it. This happens server-side when you specify a model. The default is e5_mistral_7b_instruct, which is good for general text.

curl -X POST https://api.elevenlabs.io/v1/convai/knowledge-base/{id}/vectorize \
  -H "xi-api-key: $ELEVENLABS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "e5_mistral_7b_instruct"
  }'

Vectorization is async. Poll for status or just wait a minute before testing queries.

Step 3 — Attach to the agent and enable RAG

Update your agent with a PATCH request that enables RAG and attaches the knowledge base entries:

curl -X PATCH https://api.elevenlabs.io/v1/convai/agents/{agent_id} \
  -H "xi-api-key: $ELEVENLABS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": {
      "rag": {
        "enabled": true,
        "knowledge_base": ["kb-entry-id-1", "kb-entry-id-2"]
      }
    },
    "conversation_config": {
      "agent": {
        "prompt": {
          "system_prompt": "You are a helpful assistant. When answering questions, retrieve relevant context from the knowledge base and attribute your answers to the source title. If the answer is not in the knowledge base, say so."
        }
      }
    }
  }'
Gotcha — the prompt matters. Without explicit instructions, the agent may retrieve but not cite, or ignore the retrieval entirely. The system prompt must name the behavior: retrieve, then attribute. Test this explicitly — ask a question you know is in the docs and verify the agent names the source.

Step 4 — Auto-sync with a GitHub Action

For static sites on GitHub Pages, you want the knowledge base to update every time the site rebuilds. The trigger is page_build, not push. A push trigger fires before GitHub Pages finishes deploying, which causes a ReadabilityError because the URL isn't live yet.

Save this workflow to .github/workflows/sync-knowledge-base.yml in your site's repo:

name: Sync Knowledge Base

on:
  page_build:

jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Re-create knowledge base entries
        env:
          ELEVENLABS_API_KEY: ${{ secrets.ELEVENLABS_API_KEY }}
        run: |
          # Delete old entries (optional — or just update in place)
          # This script re-creates and re-attaches knowledge base documents
          python3 scripts/sync_kb.py

      - name: Vectorize
        env:
          ELEVENLABS_API_KEY: ${{ secrets.ELEVENLABS_API_KEY }}
        run: |
          # Poll until vectorization is complete
          python3 scripts/vectorize_kb.py

The page_build event fires after GitHub Pages finishes, so the URLs in your knowledge base entries are guaranteed live. Add your ELEVENLABS_API_KEY to the repo's GitHub Secrets.

Gotcha — ReadabilityError on push. If you use on: push and your knowledge base entries reference URLs on GitHub Pages, the ElevenLabs crawler may hit a 404 because Pages hasn't finished deploying. The fix is on: page_build. This is documented poorly and cost me an hour.

Step 5 — Verify end-to-end

  1. Open your ElevenLabs agent in the dashboard. The knowledge base section should show your attached entries with green checkmarks (vectorized).
  2. Start a conversation with the agent. Ask a question whose answer is in your docs.
  3. Verify the agent names the source document (e.g., "According to the Product Docs...").
  4. Ask a question not in the knowledge base. The agent should say it doesn't know, rather than hallucinating.
  5. Push a change to your site. Wait for the page_build action to run. Ask the same question again — the answer should reflect the updated content.

When not to use this

RAG is the wrong pattern when:

What's next

Companion materials

Seth Shoultes builds at garagedoorscience.com and writes about it at sethshoultes.com/blog.