OpenAI | Jacob Aloysious

Building and Evaluating an AI Agent: A Hands-On Case Study

Sun, 11 Oct 2026 00:00:00 +0000

I wanted to learn agent evaluations end to end, not just read about them. With Codex as a coding and teaching partner, I built a fictional customer-support agent, deployed it, and worked through a course project one step at a time: define behavior, test deterministic rules, measure a baseline, make a targeted change, and check what the evidence supports.

Part 1 of a two-part series: this post covers setup and evals. Part 2 is an engineer-focused observability deep dive.

The result is a learning project, not a production support platform. The useful outcome was learning how to make agent behavior testable and improve it without treating a small demo score as a reliability guarantee.

Try it: Live demo · Source repository · Course syllabus

Start with policy, not prompting

The example is an online shop with fictional customers, orders, delivery promises, and refund requests. Before tuning a prompt, I wrote down what the system should allow:

Order access comes from the server’s authenticated session, never a customer ID asserted in chat.
Address changes must check ownership, shipment status, serviceability, and delivery timing.
A mismatch between the requested state and postal PIN needs separate, explicit confirmation.
Refund requests outside policy are declined automatically; eligible requests go to a human for review. The agent cannot approve or issue money.

That policy shaped the trust boundary. The model can select a tool and formulate arguments. Server-side tools apply ownership and policy checks against trusted session and fixture data. Tool descriptions help the model choose an action; they are not authorization controls. The human review capability is intentionally not exposed to the agent.

How the project fits together

The browser sends a chat request to /api/demo. A Vercel Node.js function runs the OpenAI Agents SDK. The SDK sends model requests to OpenAI, executes registered JavaScript tools on the server when requested, and feeds tool results back into the model. It then returns a response and correlation IDs to the browser.

Browser → Vercel function → Agents SDK → OpenAI model
│ │
└─ local policy tools
→ tool result → model response

Vercel hosts and runs the application; it does not decide which tool the agent needs. The model proposes a tool call; the server executes that function and enforces policy. One browser submission can contain multiple model turns while remaining a single inbound HTTP request.

The project is in GitHub so each lesson leaves a reviewable change. GitHub Actions runs unit/API tests and deterministic policy scenarios on pushes and pull requests. It does not call the live model: those evals require an API key, incur usage, and can vary. Live chat and model evals use OPENAI_API_KEY on the server. I set it in Vercel’s project environment settings, alongside DEMO_SESSION_SECRET; neither secret belongs in the browser bundle. Saving an environment variable requires a new deployment before the running function sees it. ChatGPT subscriptions and API Platform billing are separate ( OpenAI billing details, Vercel environment variables).

Build an evaluation ladder

I used different checks for different failure surfaces:

Unit and API tests check deterministic rules such as ownership denials, refund eligibility, and side effects. They run without a model.
Deterministic policy scenarios run proposals through server tools with fixed data and a controlled clock.
Live model evals send customer language to the agent and inspect tool results, state changes, and answer requirements.
Conversation evals preserve a transcript and shared fixture state. One scenario asks for a refund before the promised delivery date, advances the fixture clock, and asks again. The first request is declined; a later eligible one can enter the human review queue.
Grader tests and adversarial cases check that the eval rejects unsafe outcomes while accepting valid alternative wording and tool paths.

The evaluator needed testing too. A correct explanation that an order was “within its promised delivery window” initially failed because the phrase check was too narrow. A cross-account case required one particular tool to report denial even though another server-checked path had the same safe outcome. We changed the rubric to consider policy outcome, side effects, and response meaning instead of one phrase or one valid route.

This is a useful quality-engineering distinction: a failed eval can mean the agent is wrong, the application is wrong, or the grader is wrong. A pass rate without failure classification hides those differences.

Measure a baseline, then make a narrow change

For the optimization exercise, I chose three ordinary tasks: check an order status, list the signed-in account’s orders, and change an eligible address while reporting its revised delivery promise. I ran the same live model eval before changing the prompt or tool description.

Run	Passes	Average tool calls	Average elapsed time
Baseline	2/3 (67%)	1.00	3,766 ms
Prompt instruction added	3/3 (100%)	1.33	4,802 ms
Prompt + tool-description guidance	3/3 (100%)	1.00	3,322 ms

The baseline failure was specific: the address-change tool succeeded and returned a revised promised date, but the final answer omitted it. I updated the rubric to require the new delivery promise, added an instruction to include the date returned by the tool, and added a grader regression test.

That fixed the answer requirement but raised average tool calls to 1.33. Inspecting the runs showed that for a clear address-change request, the agent first looked up the order and then called the address-change tool, which already checks ownership and shipment status. I clarified the tool description so the agent could call it directly when intent was clear, while retaining lookup for questions or ambiguity. The next run stayed at 3/3 and averaged one call per scenario. We reduced redundant work without weakening server checks.

Read the numbers honestly

These measurements compare the exact runs; they do not establish production quality or a statistically significant improvement. Three scenarios are a tiny sample. “100%” means 3 of 3 cases passed, not that the agent is 100% accurate. Elapsed time includes variable network and service conditions, so timing differences are suggestive only.

Tool-call count is an efficiency proxy. This eval runner does not expose token usage, so it cannot support a dollar-cost calculation. One fewer tool call may reduce work, but it does not tell us how many tokens or dollars were saved. Once I used the optimization cases to guide changes, they were no longer a pristine holdout. I created separate final-validation cases—PIN/state mismatch, shipped order, and an untrusted lost-package claim—and ran them after the change.

For a technical leader, the lesson is the measurement loop: define the customer outcome, set a baseline, inspect a failure, make a narrow change, and evaluate again. Keep outcome quality, safety/side effects, efficiency, and operational health separate; one score cannot stand in for all four.

Continue with the observability deep dive

The next post follows one submitted message across Vercel and OpenAI: how the request ID differs from the agent trace ID, what the log and span views contain, and where an engineer should investigate each failure. Read Tracing an AI Agent Across Vercel and OpenAI.

The GitHub repository contains the code and lesson notes. The live demo uses fictional data. The project taught me to treat agent behavior as something to specify and measure, then make claims no stronger than the evidence.

Tracing an AI Agent Across Vercel and OpenAI: An Observability Deep Dive

Sun, 11 Oct 2026 00:00:00 +0000

One chat message can produce a serverless invocation, several model API calls, and local tool executions. When the answer is wrong—or missing—the challenge is finding which system owns the failure.

This post follows one request through a small support-agent project and shows what to inspect in Vercel Runtime Logs, OpenAI Agents SDK Traces, and the API Platform’s request and usage views. It is the engineering companion to Building and Evaluating an AI Agent, which covers setup, eval design, baselines, and improvement decisions.

Part 2 of a two-part series: this deep dive is for engineers following a request through the runtime and agent execution layers.

One request, multiple observability views

The browser posts to /api/demo. A Vercel Node.js function calls the Agents SDK Runner.run, which sends generation requests to OpenAI. If the model requests a tool, the SDK executes the registered JavaScript function in the server process and sends its result back to the model. The model can then make another generation before the answer returns.

Customer message
│
▼
Browser ── POST /api/demo ──► Vercel Node.js function
│
├── platform request metadata
├── app console events ─────► Vercel Runtime Logs
│
└── Agents SDK Runner.run
│ traceId = this run
│ groupId = conversation
▼
OpenAI model ────────────► API request activity
│ generation span Usage Dashboard
│
└─ function span
▼
Local server tool/policy
│ result code
└──► model continues or answers
GitHub Actions / local eval runner ──► tests, outcomes, tool counts, elapsed time

Vercel executes the HTTP request; the Agents SDK orchestrates the run; the model generates text and proposes tool calls; server functions execute tools and enforce policy. One inbound request can contain multiple model calls. A later message creates a new request and trace, linked to the earlier one by the conversation group ID.

Keep the identifiers straight

In this app, api/demo.js generates a custom traceId for each submitted chat turn and uses the browser’s conversationId as the Agents SDK groupId:

const traceId = `trace_${randomUUID().replaceAll("-", "")}`;
const { conversationId } = body;
const runner = new Runner({
workflowName: "Northstar Shop support chat",
traceId,
groupId: conversationId,
traceIncludeSensitiveData: false,
});

The IDs answer different questions:

Identifier	Created by	Use it to find
Vercel request ID	Vercel	The platform invocation: route, status, timing, deployment, and runtime details.
App `traceId`	`api/demo.js`	One submitted message’s agent run, including its model and tool spans in OpenAI Traces.
`conversationId` / SDK `groupId`	Browser/app	The set of separate agent runs from one chat conversation.
OpenAI API request ID	OpenAI API	One individual model API request, useful when investigating that request’s status or service behavior.

Do not assume similarly named trace fields are interchangeable. Vercel can show its own request, invocation, or tracing metadata. Search OpenAI Traces with the app trace ID printed in the Vercel application log; use the Vercel request ID to locate the platform request. For a whole multi-message chat, use the conversation ID/group ID.

What is logged in each system

System	What it owns	What to inspect
Browser	Sends messages and renders responses.	Network result and returned IDs; not authoritative for policy or ownership.
Vercel Runtime Logs	Runs the deployed function.	Method/path, status, duration, deployment, platform metadata, app `console` lines. The app omits the full chat body.
OpenAI Agents SDK Traces	Captures the agent run.	Span hierarchy, generations, functions, timings, and run outcome. Default spans include task, agent, turn, generation, and function.
OpenAI API Platform	Serves model inference.	HTTP Requests view for individual failed/slow calls; Usage Dashboard for aggregate usage/cost. Separate from SDK traces and Vercel logs.
Server tools and policy code	Checks session, ownership, eligibility, and allowed changes.	Deterministic result codes and state. Fix policy/data bugs here and add tests.
Eval runner / GitHub Actions	Measures scenarios and gates changes.	Outcomes, pass rate, tool count, elapsed time. CI is deterministic; live evals are metered.

A log is an event, a span is a timed operation, a trace connects spans for one run, and a metric aggregates runs. A trace explains one execution; evals show behavior across cases.

The app sets traceIncludeSensitiveData: false, excluding LLM and tool inputs/outputs from traces while retaining spans. It separately logs safe result codes with the app trace ID. Thus the trace will not contain the customer’s full prompt, tool arguments, or tool-result body. See the Agents SDK tracing guide and Vercel Runtime Logs.

Correlate a Vercel log with an OpenAI trace

The following IDs are invented and the lines are shortened, but the event names and app fields match the project. Vercel’s platform request metadata is shown separately from the app’s own log messages:

Vercel platform: POST /api/demo status=200 duration=4.1s requestId=req_91…
app stdout: {"event":"agent_run_started","conversationId":"conv_7f…","traceId":"trace_4a…"}
app stdout: {"event":"agent_tool_result","traceId":"trace_4a…","tool":"request_shipping_fee_refund_review","code":"REFUND_NOT_ELIGIBLE"}
app stdout: {"event":"agent_run_completed","conversationId":"conv_7f…","traceId":"trace_4a…","turns":5}

The actual turns value is result.newItems?.length ?? 0; it is a count of SDK result items, not necessarily model generations. The tool-result line includes a trace ID but not a conversation ID. That lets you connect the tool outcome to a run, while the start/completion lines connect that run to the conversation.

In OpenAI Traces, the same run is represented as a span tree. The exact display labels can vary by SDK version; conceptually it looks like this:

Northstar Shop support chat traceId=trace_4a… groupId=conv_7f…
└─ task / agent / turn
├─ generation span model decides whether a tool is needed
├─ function span request_shipping_fee_refund_review
│ payload omitted by trace redaction
└─ generation span model forms its final answer

The function span shows that the tool ran and where it fits in the agent loop. The Vercel agent_tool_result event gives the safe outcome code, REFUND_NOT_ELIGIBLE. Payload capture is disabled, so the trace omits the message and full tool input/output. Use the API Requests view for individual model-call errors or latency, the SDK trace for agent flow, and the Usage Dashboard for aggregate usage and cost.

The API handler calls forceFlush() before responding so the trace exporter has a chance to send the spans. If flushing fails, it writes agent_trace_flush_failed with the same conversation and trace IDs. That event is useful when the Vercel request completed but the expected OpenAI trace is absent.

Triage by symptom

Follow the failure to the system that owns it:

Browser shows 5xx or times out. Start in Vercel Runtime Logs. Filter by deployment, route, time, and status; inspect duration and agent_run_failed. Check for agent_run_started and server config such as OPENAI_API_KEY. Fix the runtime or API boundary if the request failed before completion.
Request succeeds, but the model picks a poor tool or stops early. Use traceId from agent_run_started to inspect generation → function → generation. Improve instructions, tool descriptions, or eval scenarios. Keep authorization and policy checks in server code.
Right tool ran, but returned the wrong result or changed state incorrectly. Match agent_tool_result by traceId; inspect its code, session, fixture, and tool implementation. Fix domain logic and add a deterministic test.
Tool result is right, but the answer is incomplete. Compare the final generation with the rubric. Improve instructions or response construction and add a grader regression case. For critical dates or policy disclosures, deterministic text can give stronger guarantees.
Traces look fine, but pass rate or latency declines. Segment eval failures by scenario; compare tool counts and elapsed time, and check grader false positives. Improve coverage or measurement before optimizing. Three cases cannot establish a production trend.
Vercel shows a request, but OpenAI Traces does not. Search the app traceId; check agent_trace_flush_failed, deployment/time range, and whether this was hosted chat. Local deterministic controls produce no OpenAI trace.

Correlate IDs, classify the failure, and fix the layer that owns it. A real system also needs retention, access control, alerting, and a policy for sensitive trace data.

Continue the evals story

For the project setup, eval design, baseline numbers, and the changes those measurements motivated, read Building and Evaluating an AI Agent: A Hands-On Case Study. The source repository and live demo are available for trying the example with fictional data.