LlamaIndex with AI Gateway

LlamaIndex connects large language models (LLMs) to your enterprise data to build knowledge assistants. Connect LlamaIndex to AI Gateway to use its model catalog and provider routing.

1. Create a new project

   First, create a new directory for your project and initialize it:

   terminal

   mkdir llamaindex-ai-gateway
   cd llamaindex-ai-gateway

2. Install dependencies

   Install the required LlamaIndex packages along with the python-dotenv package:

   terminal

   pip install llama-index-llms-vercel-ai-gateway llama-index python-dotenv

3. Configure environment variables

   Create a .env file with your Edgenivo AI Gateway API key:

   .env

   AI_GATEWAY_API_KEY=your-api-key-here

   If you're using the AI Gateway from within a Edgenivo deployment, you can also use the VERCEL_OIDC_TOKEN environment variable which will be automatically provided.

4. Create your LlamaIndex application

   Create a new file called main.py with the following code:

   main.py

   from dotenv import load_dotenv
   from llama_index.llms.vercel_ai_gateway import VercelAIGateway
   from llama_index.core.llms import ChatMessage
   import os
    
   load_dotenv()
    
   llm = VercelAIGateway(
       api_key=os.getenv("AI_GATEWAY_API_KEY"),
       max_tokens=200000,
       context_window=64000,
       model="anthropic/claude-opus-5",
   )
    
   message = ChatMessage(role="user", content="Tell me a story in 250 words")
   resp = llm.stream_chat([message])
   for r in resp:
       print(r.delta, end="")

   The following code:

   • Initializes a VercelAIGateway LLM instance with your API key
   • Configures the model to use Anthropic's Claude Opus 5 (anthropic/claude-opus-5) through AI Gateway
   • Creates a chat message and streams the response

5. Running the application

   Run your application using Python:

   terminal

   python main.py

   You should see a streaming response from the AI model.