Tools, APIs, and the Agent Ecosystem

Tools, APIs, and the Agent Ecosystem

Modern AI agents don't just generate text — they use tools to interact with the world. This lesson covers the ecosystem that makes agents powerful.

Function Calling and Tool Use

LLMs can decide when and how to call external functions:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string", "description": "City name"},
                "units": {"type": "string", "enum": ["celsius", "fahrenheit"]}
            },
            "required": ["location"]
        }
    }
}]

# The LLM decides whether to call the tool based on the user's question
response = client.chat.completions.create(
    model="gpt-4",
    messages=messages,
    tools=tools
)

RAG (Retrieval-Augmented Generation)

RAG gives agents access to external knowledge:

1. Index: Convert documents into embeddings, store in vector database
2. Retrieve: When user asks a question, find relevant document chunks
3. Generate: Pass retrieved context to LLM along with the question

# Simplified RAG pipeline
query_embedding = embed(user_question)
relevant_docs = vector_db.similarity_search(query_embedding, k=5)
context = "\n".join([doc.content for doc in relevant_docs])

response = llm.generate(
    f"Context: {context}\n\nQuestion: {user_question}\nAnswer:"
)

Vector Databases

Specialized databases for storing and searching embeddings:

Database	Type	Best For
Pinecone	Cloud	Production, managed
Weaviate	Self-hosted/Cloud	Flexible, open-source
ChromaDB	Local	Prototyping, small scale
Qdrant	Self-hosted/Cloud	High performance
pgvector	PostgreSQL extension	Existing Postgres users

Agent Frameworks Overview

Frameworks that simplify building agents:

• LangChain: Most popular, chains + agents + tools + memory
• LlamaIndex: Focused on data indexing and retrieval (RAG)
• CrewAI: Multi-agent collaboration framework
• AutoGen: Microsoft's multi-agent conversation framework
• LangGraph: Graph-based agent workflows (from LangChain team)

The Agent Stack

┌─────────────────────────────────────┐
│         Your Application            │
├─────────────────────────────────────┤
│    Agent Framework (LangChain)      │
├─────────────────────────────────────┤
│  LLM Provider  │  Vector DB  │ Tools│
│  (OpenAI/etc)  │  (Pinecone) │(APIs)│
└─────────────────────────────────────┘

Key Takeaways

• Tool use transforms LLMs from text generators into action-taking agents
• RAG gives agents access to up-to-date, domain-specific knowledge
• Vector databases enable semantic search over large document collections
• Frameworks like LangChain and LlamaIndex accelerate agent development
• The modern agent stack combines LLMs + tools + memory + retrieval