Arun Baby

Becoming Unlabelable

7 minute read

“Giving the Brain Hands to Act: The Interface Between Intelligence and Infrastructure.”

1. Introduction: From Text to Action

An LLM isolated in a box is just a text generator. It can hallucinatively describe the weather, but it cannot check the actual temperature. To become an Agent, it needs to interact with the outside world. This is achieved through Tool Calling (often called Function Calling).

Tool calling is the bridge between the Probabilistic World of AI (where 2+2 might equal 5 if the context is weird) and the Deterministic World of Software (where 2+2 is always 4). It is the mechanism that transforms an LLM from a “Chatbot” into a “Controller.”

In this post, we will dissect the anatomy of a tool call, explore standard patterns like JSON Schema, and discuss the critical architecture of a robust Runtime Execution Environment.

2. The Mechanics of Tool Calling

How does a model “call” a function? It doesn’t. A neural network cannot execute code. It can only emit tokens. Tool Calling is a parse-execute loop.

2.1 The Lifecycle of an Action

Let’s trace a user request: “What is the weather in Tokyo?” through the system.

  1. Tool Definition: You provide the model with a “menu” of functions in the System Prompt (usually in strict JSON Schema format).
    • Prompt: “You have a tool get_weather(city: str). Use it if needed.”
  2. Reasoning: The model analyzes the user request against the menu. It performs Semantic Matching.
    • Thinking: “User asks for weather. ‘Tokyo’ is a city. This matches get_weather.”
  3. Generation: The model generates a special token or formatted string (e.g., a JSON object) representing the intent to call the function: 
    { "tool": "get_weather", "arguments": { "city": "Tokyo" } }
  4. Pause (Stop Sequence): The inference engine recognizes that the model has output a “Tool Call Block” and stops generating. It freezes the model state and returns control to the Python script (the Orchestrator).
  5. Execution (The Runtime):
    • Your code parses the JSON.
    • Validation: Is “Tokyo” a string? Is it valid?
    • Execution: Your code calls the actual Python function requests.get(...).
    • Result: It captures the return value: {"temp": 18, "condition": "Cloudy"}.
  6. Context Injection (The Feedback): You do not show the result to the user yet. You append a new message to the chat history:
    • Role: Tool, Content: {"temp": 18, "condition": "Cloudy"}.
  7. Final Response: You invoke the model again. It sees its own previous request + the tool output. It now performs Synthesis.
    • Output: “It’s 18°C and cloudy in Tokyo.”

2.2 JSON Schema: The Protocol

Standardization is key. The industry has converged on JSON Schema (OpenAPI) to define tools. Models like GPT-4 are fine-tuned to read this specific format.

  • Crucial Tip: The description field in the schema is not just documentation; it is part of the Prompt. Be verbose.
    • Bad: "description": "Get weather"
    • Good: "description": "Get current weather for a specific city. Do not use for general climate questions. Returns temp in Celsius."
    • Impact: The model reads this description to decide when to call the tool.

3. Design Patterns for Reliability

LLMs are clumsy. They make mistakes. They hallucinate arguments (e.g., inventing a parameter force=True that your function doesn’t accept). Your runtime must be defensive.

3.1 Pydantic Validation (The Shield)

You should never pass raw LLM output to a function.

  • Pattern: Wrap every tool in a Pydantic Model.
  • Logic: Before executing, pass the LLM’s argument dict into the Pydantic model.
  • Self-Correction: If validation fails (e.g., “Field ‘city’ is missing”), catch the error and return it to the LLM.
    • System Response to Agent: Error: Invalid argument. Missing 'city'.
    • Agent: “Ah, sorry.” (Tries again with city). This Feedback Loop allows the agent to fix its own typos without crashing the program.

3.2 Robust Outputs (Error Propagation)

What happens if your API returns a 500-line stack trace?

  • Problem: It runs up your token bill and fills the context window with noise. The agent uses the stack trace to hallucinate a weird answer.
  • Fix: Catch exceptions in the tool wrapper. Return a concise, safe string.
    • Bad: Traceback (most recent call last): File "main.py"...
    • Good: Error: The database connection timed out. Please try again later.
  • Why: This allows the Agent to reason about the failure (“Okay, I’ll apologize to the user or try a different database”) rather than getting confused.

3.3 Atomic vs. Mega Tools

  • Mega Tool: manage_user(action, id, data) - Hard for the LLM to understand all the permutations. It has to guess the schema for data based on action. It often fails.
  • Atomic Tools: create_user, delete_user, update_email.
  • Rule: Smaller, specific tools reduce hallucination rates. Adhere to the Single Responsibility Principle.

4. Scaling: The “Too Many Tools” Problem

GPT-4 has a context limit. If you have 5,000 internal APIs (like AWS or Stripe), you cannot paste all 5,000 JSON schemas into the prompt. It would cost $5 per query and confuse the model.

4.1 Solution: Tool RAG (Retrieval)

Treat tool definitions like documents.

  1. Embed the descriptions of all 5,000 tools into a Vector Database.
  2. When a user query comes in (“How do I refund a charge?”), embed the query.
  3. Search the Vector DB for the top 5 most relevant tools (stripe_refund, stripe_get_charge, etc.).
  4. Inject only those 5 definitions into the context prompt. This allows agents to have infinite toolkits.

4.2 Handling Asynchronous Actions

Some tools take time (e.g., generate_video, provision_server). You can’t keep the HTTP connection open for 10 minutes waiting for the tool.

  • Pattern:
    1. Agent calls start_job().
    2. Tool returns immediately: {"job_id": "123", "status": "pending"}.
    3. Agent reasons: “I have started the job. I will check back later.”
    4. Agent (or a scheduler) periodically calls check_status(job_id).

5. Security: The Sandbox

The most dangerous tool is run_python_code. If an agent can run code, it can os.system('rm -rf /') or os.environ['AWS_KEY'].

5.1 Sandboxing Strategies

NEVER run agent code on your host.

  1. Transational Containers: Spin up a Docker container for the session. Destroy it after.
  2. WebAssembly (Wasm): Run code in a browser-like sandbox (e.g., Pyodide).
  3. Cloud Sandboxes: Use services like E2B or Modal that provide secure, isolated VMs specifically for AI agents.

5.2 Permission Scopes & HITL

  • Read-Only: Give agents “Viewer” access by default.
  • Human-in-the-Loop (HITL): Tag dangerous tools (delete_db, send_money) as requires_approval=True.
    • When the agent calls it, the runtime pauses.
    • The user gets a pop-up: “Agent wants to delete DB. Allow?”
    • Only on “Yes” does the code execute.

6. Code: A Semantic Tool Router (Conceptual)

Instead of showing the parsing code (which varies by library), let’s look at the Router Logic structure.

class ToolExecutor:
    """
    The 'Hands' of the agent.
    Responsible for safe execution and validation.
    """
    def execute(self, tool_name: str, raw_arguments: dict) -> str:
        
        # 1. Lookup
        tool = self.registry.get(tool_name)
        if not tool:
            return "Error: Tool not found."
            
        # 2. Validation (Pydantic)
        try:
            validated_args = tool.schema(**raw_arguments)
        except ValidationError as e:
            # CRITICAL: Return the validation error to the LLM
            # giving it a chance to fix its typo.
            return f"Error: Invalid Arguments. {e}"

        # 3. Security Check (HITL)
        if tool.requires_approval:
            permission = self.human_interface.request(tool_name, validated_args)
            if not permission:
                return "Error: User denied permission."

        # 4. Execution (Sandboxed)
        try:
            result = tool.run(validated_args)
            return str(result)
        except Exception as e:
            # 5. Error Sanitization
            return "Error: Internal tool failure. Please try again."

# Usage in the Agent Loop:
# if output.is_tool_call:
#     result = executor.execute(output.name, output.args)
#     memory.add("ToolResult", result)

7. Summary

Tool Calling is what makes AI useful.

  • Standardized Interfaces (JSON Schema) allow models to understand the world.
  • Defensive Coding (Validation Loops) allow models to correct their own mistakes.
  • Strict Security (Sandboxing) ensures the agent doesn’t burn down the house.

By mastering these fundamentals, you can build agents that don’t just talk, but do—transforming business workflows from manual drudgery to autonomous execution.