Article: The Agent Harness: What It Is and Two Ways to Build One
InfoQ Homepage Articles The Agent Harness: What It Is and Two Ways to Build One
AI, ML & Data Engineering InfoQ Certified AI-Assisted Engineering Program (online, Oct 19): Build the harness that holds.The Agent Harness: What It Is and Two Ways to Build One
Sep 25, 2026 17 min read
by
reviewed by
Follow us on
Youtube232K Followers Linkedin26K Followers InstagramNew RSS19K Readers X57.1k Followers Facebook21K Likes BlueskyNew Listen to this article - 0:00 Audio ready to play Your browser does not support the audio element. 0:00 0:00Key Takeaways
- The gap between a demo and a production agent is the harness: Everything you build around the model to make it a real product. Almost none of it comes from the model itself, even though it is where most of your engineering time actually goes.
- The harness splits into two halves: Development extends what the model can do, while operations keeps it running once real users show up. That operations half is mostly DevOps in a new hat.
- Harness-as-a-Service (HaaS) (like AWS AgentCore) and a self-managed one (like LangChain with Agent Router (formerly Envoy AI Gateway) on Kubernetes) give you the same capabilities, so the real question isn’t what you receive. It’s who runs it, how much you’d like to pay, and whether you’re trading control for speed or effort for portability.
- Building a production agent is more architecture than magic: You decide what the product must do, what “good” looks like, and where the hard boundaries are. Then use the model and the harness as the tools to build it.
- Grow the harness, but don’t gold-plate it. Start minimally and let the stack evolve with the agent. The value comes from the job getting done, not from a gold-plated harness built before you need it.
Spinning up a demo AI agent can take an afternoon. Building one that’s actually ready for production is a very different story.
A production agent has to stay up, stay safe, keep its costs in check, and give you enough visibility to work out what went wrong when it inevitably breaks. Almost none of that capability comes from the model. It comes from everything you build around the model: memory, tool access, model routing, guardrails, cost controls, and the traces you go digging through when your agent starts misbehaving.
That layer is what we call the agent harness.
Figure 1. Demo vs. production (Source: Image created by author).
The rest of this article is a tour of what an agent harness is and its two halves: development and operations. Then we’ll look at how teams choose what to manage themselves, from a fully managed service (Harness-as-a-Service) to a self-managed stack. Finally, I’ll build the same agent, FinBot, both ways, capability by capability. By the end, you’ll have a better sense of how to leverage the harness capabilities introduced in this article to build your production-ready agents.
Agent = Model + Harness
"Agent harness" is a concept that really only caught on this year, but the work behind it isn’t new at all. If you were building agents last year, you were already doing it: gluing tools together, hand-tuning prompts, bolting on retries and logging, and patching each failure the morning after it woke someone up. What is new is having a name for this process. The name turns out to be surprisingly useful. It lets you treat all of that capability as one thing you design on purpose, instead of a drawer full of one-off fixes.
The framing I keep coming back to is Vivek Trivedy’s, from The Anatomy of an Agent Harness: An agent is a model plus a harness. Put another way: if you’re not the model, you’re the harness. A car makes the idea click for me.
Figure 2. Agent = Model + Harness (Source: Image created by author).
The model is the engine. It is where the power comes from. But nobody ships a bare engine bolted to a pallet to a customer. The chassis, brakes, dashboard, and seatbelts are what make it something you’d put your family in. The model is only the engine; it is everything that makes the product safe, reliable, and pleasant to use is the harness you build around it.
The Two Halves of a Harness
The way I think about it, everything in the harness falls into two halves. Development is what extends the model’s reach: memory across sessions, tools and MCP, retrieval, prompts, and orchestration. Operations is what keeps the thing running reliably once real users show up: observability, evaluation, guardrails, routing, monitoring for drift and cost, deployment, and scaling.
Figure 3. The two halves of a harness (Source: Image created by author).
I want to be careful here, because it’s easy to look at a lopsided diagram like this and conclude the model barely matters. That’s not at all true. The model does the genuinely hard cognitive work. A stronger model lifts everything above it. The point is subtler: An AI agent isn’t a model, it’s a product. Building a good product takes far more than wrapping a model in an API, and most of that "far more" is the harness. A lot of it will feel familiar, too, if you’ve run services before: The operations half is essentially DevOps, wearing a new hat.
Two Ways to Build It
Figure 4. HaaS vs. self-managed (Source: Image created by author).
Learn one thing early. Both routes are the same kind of thing: The scaffolding you build an agent on and then operate it with. Under the hood they hold the same building blocks: model access, retrieval, tools and MCP, routing, guardrails, etc.
What differs is how those blocks are packaged and how you interact with them. HaaS hands you each one behind a managed API you configure; self-managed hands you the pieces to wire together and run. That difference decides how fast you ship, how much control you keep, and what the harness costs you: a vendor bill on one side, infrastructure and engineering time on the other. What changes isn’t the parts. It’s who assembles them and who gets paged when they break.
| Dimension | Harness-as-a-Service | Self-managed |
| What it is | A vendor-run runtime that exposes the whole harness, dev and ops, as managed APIs | A harness you assemble from an SDK plus open-source components |
| You still own | Policies, budgets, prompts, tools, evals | The same, plus deployment, upgrades, and on-call |
| Tied to | The provider’s runtime and APIs | The stack you chose and built |
| Examples | AWS AgentCore, Google Vertex AI Agent Engine, Azure AI Foundry Agent Service, LangGraph Platform | LangChain or LlamaIndex, plus Agent Router (formerly Envoy AI Gateway) or LiteLLM, on Kubernetes |
| Sweet spot | Shipping fast without owning deployment | Cloud-agnostic deployments, tight control |
Choosing Where to Sit on the Spectrum
Harness-as-a-Service sits at one end, self-managed at the other, and plenty of teams land somewhere in between, which is perfectly fine. There is a spectrum here; where you want to sit on it comes down to a handful of considerations:
- Team capability – can you operate Kubernetes, a gateway, and an on-call rotation, or would you genuinely rather not?
- Existing cloud commitments – already all-in on AWS, or deliberately multi-cloud?
- Cost – a predictable vendor bill, or infrastructure plus the engineering time to run it? HaaS is simpler to budget; self-managed can come out cheaper at scale, once you’re paying to operate it either way.
- Governance – do the guardrail, the audit trail, and the data path have to sit inside your own boundary? Some enterprises must run all of this in their own VPC.
- Portability – one provider is fine or must you stay cloud-agnostic?
- Expected scale – how much operational surface you’re willing to run to serve it.
- Tolerance for operational complexity – do you actually want to live with clusters, upgrades, and the occasional 3am page?
The Demo: Building FinBot, Then Three Capabilities to Watch
FinBot is a small agent, and it’s the same scenario in both builds. A user asks a finance question: "summarize Q3 revenue". FinBot pulls the quarter’s filings from a document store, hands the numbers to a code interpreter, calls a model to write the summary, and answers.
The real point is that shipping this for production isn’t about picking a smarter model. It’s about what you build around it. A great model can reason all day, but on its own it won’t remember the last turn, won’t reach the filings, won’t stop a runaway loop before it becomes an invoice, and won’t tell you what went wrong when it breaks. All of that is a harness, wrapped around this one request path; I’ll trace the same path through both builds, written to the same spec on each side but in the idioms each stack encourages.
A real harness has a lot of moving parts (go back to the two halves) and no article can carry all of them. I’ll pick a few from each half: first, how FinBot itself gets built, then three operational capabilities that, in my experience, decide production-readiness faster than anything else. Guardrails, evaluation, prompt orchestration, deployment and scaling, and multi-agent workflows don’t appear here; that’s a real gap rather than a tidy boundary, because getting a demo all the way to production means dealing with those too. Here’s why each of the ones I did pick matters, so that Option 1 and Option 2 can stay focused on how each delivers it.
Building the Agent
The development half, narrowed to what FinBot actually needs: the system prompt, the tools it can reach (an MCP server for the filings, and a container running an MCP code-execution server), retrieval over those filings, and memory across turns. This is where the two stacks feel most different. On one side it’s a config object, on the other it’s a set of processes you deploy and version.
Unified Model Access
Every provider has its own SDK, auth, and response format. Self-hosted models add more. You want one front door, so adding a model, canarying a new one, or failing over to another provider is a config change rather than an app change. Your provider keys live in one place instead of sprayed across every service.
Cost Control
One buggy loop or a careless feature can spike token usage into a runaway bill. You need metering, budgets, and graceful behavior when a limit is hit, such as a hard cap that stops the loop, or a fallback that keeps serving, rather than a surprise invoice.
Observability
An agent is a black box, and multiple providers make it worse. Without a single view of the prompt-tool-response chain, plus latency and cost, you can’t debug it, let alone tune it.
Now, the same agent and the same three capabilities, built two ways.
Meet the Two Stacks
Before the capabilities, we will take a quick orientation on what each build is actually made of. Both deliver the same blocks, but they package them differently, and the packaging is what the next sections are about.
Figure 5. AgentCore Harness architecture (Source: Image created by author).
Amazon Bedrock AgentCore gives you the production primitives (Runtime, Memory, Identity, Gateway, Observability, Code Interpreter, and Browser) and AgentCore Harness is a managed layer on top that turns "wire all these primitives together" into "fill in some config". You describe what the agent is through two calls, CreateHarness and InvokeHarness and the harness wires up the primitives underneath with a Firecracker microVM per session, managed memory, identity, and automatic tracing.
When config runs out of road, one command exports the harness to editable Strands code that runs on the same runtime. Config for the boring eighty percent, code for the twenty percent that’s actually specific to you.
Figure 6. FinBot with Agent Router (formerly Envoy AI Gateway) (Source: Image created by author).
The self-managed build keeps FinBot as a plain LangChain agent on your own cluster and hands the model-access half of the harness to a gateway on the request path.
Agent Router (formerly Envoy AI Gateway) is an open-source gateway built on Envoy Proxy and the CNCF Envoy Gateway, aimed at GenAI traffic: one OpenAI-compatible endpoint in front of many providers, token-based rate limiting, routing and failover, observability, and an ext_proc extension hook, with the control plane and data plane cleanly split. The request path is easy to follow. FinBot fires an OpenAI-style call at the gateway. The data plane (Envoy Proxy plus an ext_proc service) authenticates it, routes by model, runs guardrails, counts tokens, and injects the upstream credentials before it ever reaches the provider. A control plane of controllers watches your Kubernetes CRDs and pushes config down via xDS. It all runs on your own cluster and none of it is tied to one cloud. Any Kubernetes will do (EKS, GKE, AKS, or on-prem).
The examples run on EKS in front of Bedrock. If you move the stack to another cloud, (e.g., GKE in front of Vertex) this is what cloud-agnostic buys you: FinBot’s code and the endpoint it calls don’t change. What you reconfigure is the layer underneath, including the backend schema, model identifiers, credentials, workload identity, and deployment config.
Building FinBot: Tools, MCP, Memory and Retrieval
Option 1 runs FinBot on AWS AgentCore, a vendor-run harness runtime; Option 2 runs the same agent on LangChain behind an Agent Router on your own cluster. Both need the same four things: a system prompt, the tools it can reach, retrieval over the filings, and memory across turns. On AgentCore they are fields in one control-plane call:
import boto3
control = boto3.client("bedrock-agentcore-control")
runtime = boto3.client("bedrock-agentcore")
harness = control.create_harness(
harnessName="finbot",
executionRoleArn=ROLE_ARN,
systemPrompt=[{"text": "You are FinBot, a finance assistant."}],
model={"bedrockModelConfig": {
"modelId": "us.anthropic.claude-sonnet-4-6",
"apiFormat": "converse_stream"}},
tools=[
{"type": "agentcore_code_interpreter", "name": "code"},
{"type": "remote_mcp", "name": "filings",
"config": {"remoteMcp": {"url": "https://mcp.internal/filings"}}},
],
memory={"managedMemoryConfiguration": {
"strategies": ["SEMANTIC", "SUMMARIZATION"],
"eventExpiryDuration": 60}},
)
resp = runtime.invoke_harness(
harnessArn=harness["harnessArn"],
runtimeSessionId=SESSION_ID,
messages=[{"role": "user", "content": [{"text": "Summarize Q3 revenue."}]}],
)
Retrieval isn’t a harness field of its own: an Amazon Bedrock Knowledge Base arrives as a Gateway connector target and is attached like any other tool.
Self-managed, the same four building blocks are assembled from separate pieces you deploy and version:
from langchain.agents import create_agent
from langchain_openai import ChatOpenAI
from langchain_mcp_adapters.client import MultiServerMCPClient
from langchain_core.tools.retriever import create_retriever_tool
from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver
from psycopg_pool import AsyncConnectionPool
from psycopg.rows import dict_row
model = ChatOpenAI(model="finbot", base_url="http://ai-gateway/v1", api_key="unused",
default_headers={"x-ai-eg-model": "finbot"})
async def build_agent(): # once, at startup
pool = AsyncConnectionPool(DB_URI, open=False,
kwargs={"autocommit": True, "row_factory": dict_row})
await pool.open() # long-lived, shared
checkpointer = AsyncPostgresSaver(pool)
await checkpointer.setup() # migration: run once, not per request
mcp = MultiServerMCPClient({"filings": {"url": "http://filings-mcp:8000/mcp",
"transport": "http"},
"code": {"url": "http://code-sandbox:8000/mcp",
"transport": "http"}})
tools = [*await mcp.get_tools(),
create_retriever_tool(vector_store.as_retriever(search_kwargs={"k": 4}),
name="search_filings",
description="Search quarterly filings.")]
return create_agent(model, tools,
system_prompt="You are FinBot, a finance assistant.",
checkpointer=checkpointer)
async def answer(agent, question, session_id): # per request
return await agent.ainvoke(
{"messages": [{"role": "user", "content": question}]},
{"configurable": {"thread_id": session_id}}) # memory keyed by session
These are the same four building blocks and the same request path. On one side they are fields in a config object; on the other, they are processes with their own lifecycle, where runtimeSessionId versus thread_id is the whole difference in two identifiers.
Model Access
One front door is created for each provider, so that adding a model, canarying a new one, or failing over is a config change rather than an app change.
On AgentCore
Naming the model is config, and you can override it per call. Under the hood AgentCore handles Bedrock, OpenAI, Gemini, and any LiteLLM-compatible provider, switches provider mid-session without losing context, and keeps third-party keys in AgentCore Identity’s token vault. The model field is a union of four (bedrockModelConfig, openAiModelConfig, geminiModelConfig, liteLlmModelConfig), settable on the harness and overridable per invocation:
resp = runtime.invoke_harness(
harnessArn=harness["harnessArn"],
runtimeSessionId=SESSION_ID,
model={"openAiModelConfig": {"modelId": "gpt-4o"}}, # swap provider per call
messages=[{"role": "user", "content": [{"text": "Summarize Q3 revenue."}]}],
)
Self-Managed
Agent Router exposes one OpenAI-compatible endpoint and maps a model alias to a backend, with the provider credentials held at the gateway:
kind: AIGatewayRoute
spec:
rules:
- matches:
- headers: [{ name: x-ai-eg-model, value: finbot }]
backendRefs:
- name: bedrock-claude
modelNameOverride: us.anthropic.claude-sonnet-4-6 # alias -> real model id
---
kind: AIServiceBackend # translate OpenAI schema -> Bedrock
spec: { schema: { name: AWSBedrock } }
FinBot points a normal client at that single endpoint and never sees a provider key; switching model or provider is just the alias it sends. Credentials stay at the gateway. For Bedrock that means an AWSCredentials policy backed by EKS Pod Identity or IRSA, so no static keys sit in the cluster:
apiVersion: aigateway.envoyproxy.io/v1beta1
kind: BackendSecurityPolicy
spec:
targetRefs:
- {group: aigateway.envoyproxy.io, kind: AIServiceBackend, name: bedrock-claude}
type: AWSCredentials # Bedrock uses AWS auth
awsCredentials:
region: us-east-1 # credentials via EKS Pod Identity or IRSA
Both give you one front door. AgentCore’s is an API you call with a different config object; the gateway’s is a route you deploy. The difference surfaces in where the provider key lives: AgentCore Identity’s token vault, or an IAM role you attach to the gateway through Pod Identity or IRSA.
Cost Control
Cost control is a hard cap that stops a runaway loop, and a budget you can actually enforce before the invoice arrives.
On AgentCore
AgentCore attacks the runaway-loop failure directly, with per-invocation hard caps (maxIterations, 75 by default; maxTokens; timeoutSeconds), plus tags for cost allocation and token metrics in CloudWatch:
aws bedrock-agentcore-control update-harness \
--harness-id "finbot-UuFdkQoXSL" \
--max-iterations 50 --max-tokens 8192 --timeout-seconds 1800
Self-Managed
The gateway meters tokens as a request cost. Here a per-user daily token budget sits on the model, charged from the response. Once it is spent, the gateway rejects further requests with HTTP 429 (limit.requests is a token budget despite the name):
kind: AIGatewayRoute
spec:
llmRequestCosts:
- { metadataKey: llm_total_token, type: TotalToken }
rules:
- matches: [{ headers: [{ name: x-ai-eg-model, value: finbot }] }]
backendRefs: [{ name: bedrock-claude }]
---
kind: BackendTrafficPolicy
spec:
rateLimit:
global:
rules:
- clientSelectors: [{ headers: [{ name: x-user-id, type: Distinct }] }]
limit: { requests: 5000000, unit: Day }
cost:
request: { from: Number, number: 0 }
response: { from: Metadata, metadata: { namespace: io.envoy.ai_gateway, key: llm_total_token } }
AgentCore caps the agent, while; the gateway caps the traffic. One stops a loop mid-invocation, the other refuses the next request once a tenant’s token budget is spent. Only the second one knows what a tenant is.
Observability
This is one view of the prompt-tool-response chain, with latency and cost attached to it.
On AgentCore
This one is nearly free. Every AgentCore invocation automatically ships traces, logs, and metrics to CloudWatch (i.e., model calls, tool calls, and memory ops) in one view. There is a single account-level prerequisite; after that every invocation is traced:
# one-time, account level
aws logs put-resource-policy --policy-name MyResourcePolicy \
--policy-document file://xray-to-logs.json # logs:PutLogEvents for xray.amazonaws.com
aws xray update-trace-segment-destination --destination CloudWatchLogs
An indexing rule that sets the sampling percentage is optional.
To see what that saves you, compare it with instrumenting an agent yourself. Hosted outside the runtime, you set AGENT_OBSERVABILITY_ENABLED, OTEL_PYTHON_DISTRO and OTEL_PYTHON_CONFIGURATOR, and wrap the start command in opentelemetry-instrument. Bring your own container to AgentCore Runtime and the runtime sets its own ADOT defaults, leaving only the opentelemetry-instrument wrapper. Deploy through the AgentCore CLI and it adds the instrumentation while packaging, so you don't need the wrapper either. The harness developer writes none of it.
Self-Managed
Agent Router emits GenAI metrics on the OpenTelemetry semantic conventions (token usage, time-to-first-token, and inter-token latency) and Prometheus scrapes them:
scrape_configs:
- job_name: envoy-ai-gateway
relabel_configs:
- source_labels: [__meta_kubernetes_pod_container_port_name]
regex: "metrics|aigw-admin"
action: keep
# exposed GenAI metrics (OTel semconv):
# gen_ai.client.token.usage
# gen_ai.server.time_to_first_token
So the two questions you actually ask in an incident (who is burning tokens and who is slow) are one query each. The standard metric names keep them portable across providers:
sum(gen_ai_client_token_usage_sum{gateway_envoyproxy_io_owning_gateway_name="finbot"})
by (gen_ai_request_model, gen_ai_token_type)
Metrics answer how much and how slow. For what actually happened you need traces, and the shape is simple: two producers, one collector, and one backend.
Figure 7. Two producers, one collector, one backend (Source: Image created by author).
Both emit the same gen_ai.* vocabulary over OTLP, which is what makes their spans line up. Instrumentation and backend stay independent choices: Langfuse, Phoenix and OpenLIT each ship a LangChain SDK as well as a backend, so you can instrument with one and store in another. On the gateway, point the ext-proc at the collector:
kind: GatewayConfig
spec:
extProc:
kubernetes:
env:
- { name: OTEL_EXPORTER_OTLP_ENDPOINT, value: "http://otel-collector:4317" }
- { name: AI_GATEWAY_TRACING_SEMCONV, value: "gen_ai" } # gen_ai.* span names
Those spans cover the model calls on the request path. The rest of the loop (tool calls, the agent’s control flow, prompt assembly) never leaves the process, so instrument the app. OpenLIT is one line and speaks the same conventions:
import openlit
openlit.init(otlp_endpoint="http://otel-collector:4318") # once, at startup
The OpenTelemetry project ships its own LangChain instrumentation too, which is still beta at the time of writing. Either way, one thing catches everyone out: message content is off by default, so set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY on both sides if you want the prompts themselves in the trace, and think about who can read them once you do.
Point both at one collector and you have the prompt to tool to response chain. It is the same instrumentation work AgentCore Runtime asks for when you bring your own code; the harness is the only option that traces the loop for you.
Wrapping Up
This article introduced the idea of an agent harness and walked through two ways to build one: a managed approach with Harness-as-a-Service and a self-managed approach. We only had room to demonstrate a few of the harness's capabilities in FinBot, so we’ll leave the rest for you to explore.
Whichever path you choose, building and running an agent eventually settles into the same rhythm, and it’s more architecture than magic. You know what the product needs to do, what "good" looks like, and where the hard boundaries are. The model brings the reasoning; the harness provides the control that turns that reasoning into something you can actually ship. You define the outcome and set the boundaries, while the tools do the work within them.
About the Author
Trista Pan
Show moreShow lessThis content is in the AI, ML & Data Engineering topic
Related Topics:
-
Related Editorial
-
Related Sponsors
-
Related Sponsor
/filters:no_upscale()/sponsorship/topic/c59b0682-d008-40a2-a913-b206619a26c0/DatadogWebinarOct8-RSB-1787825775117.png)
The InfoQ Newsletter
A round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example