Agent-to-agent systems are becoming an important architectural pattern for building sophisticated AI applications. Instead of relying on a single AI agent to perform every task, an application can distribute responsibilities across multiple specialized agents. One agent might research information, another might analyze it, another might execute an external operation, and a final agent might validate the result.
The problem is that multi-agent systems introduce a new class of reliability challenges.
AI agents can take a long time to complete tasks. They may need to wait for external services, recover from failures, coordinate with other agents, and maintain state across multiple interactions. Traditional request-response architectures are often poorly suited to these requirements because they assume that a process will complete within a relatively short period of time.
This is where Temporal Nexus can provide a useful architectural foundation.
Temporal is designed around durable execution: application logic can continue across failures, retries, restarts, and long periods of waiting without requiring developers to manually reconstruct application state. Nexus extends this model toward communication and interaction between services, making it particularly interesting for agent-to-agent architectures.
This article explores how to build durable agent-to-agent services with Temporal Nexus, including architecture, service definitions, workflows, asynchronous communication, failure handling, retries, and practical coding examples.
Why Agent-to-Agent Systems Need Durable Execution
A simple AI application might look like this:
User
|
v
AI Agent
|
+--> LLM
|
+--> API
|
+--> Database
|
v
ResponseThis architecture works reasonably well when a request can be completed quickly.
A multi-agent application is considerably more complicated:
+----------------+
| Research Agent |
+-------+--------+
|
v
+------+ +----------------+
| User | ---> Coordinator Agent
+------+ +-------+--------+
|
+------------+------------+
| |
v v
+-------------+ +-------------+
| Analysis | | Execution |
| Agent | | Agent |
+------+------+ +------+------+
| |
+------------+------------+
|
v
Validation AgentSuppose the research agent takes ten minutes to gather information. The execution agent then waits for that result, performs several API calls, and eventually requires human approval.
A conventional HTTP request should not remain open for the entire operation.
Instead, the process should become a durable workflow.
Temporal allows the application to represent the process as workflow state rather than as one long-lived network request. If a worker crashes halfway through execution, the workflow can continue when another worker becomes available.
That distinction is fundamental to reliable agent-to-agent architecture.
What Temporal Nexus Adds to the Architecture
A useful way to think about Nexus is as a mechanism for exposing durable operations between Temporal-based services.
Instead of directly coupling agents through HTTP calls such as:
response = requests.post(
"http://research-agent/research",
json={"topic": topic}
)an agent can invoke a durable operation exposed by another service.
Conceptually:
Coordinator Workflow
|
| Nexus request
v
Research Service
|
v
Research Workflow
|
v
Research ActivitiesThe important architectural property is that the receiving operation can itself be backed by Temporal’s durable execution model.
The coordinator therefore does not need to assume that the remote agent will finish immediately.
This creates a much stronger boundary between agents.
Each agent can own its own workflows, activities, data, and operational responsibilities while still exposing carefully defined operations to other agents.
Designing Agent Services Around Capabilities
One of the most important design decisions is determining what an agent should expose.
Avoid exposing implementation details.
For example, a research service should not expose operations such as:
runSearchEngine
callLLM
saveIntermediateResult
parseSearchResponseThose are internal implementation details.
Instead, expose business-level capabilities:
Research a topic
Summarize evidence
Validate a source setThis gives other agents a stable interface.
A conceptual service contract might look like:
class ResearchRequest:
topic: str
depth: str
class ResearchResult:
summary: str
findings: list[str]The coordinator only needs to understand the contract.
It does not need to know whether the research agent uses one model, five models, a search engine, a database, or several activities internally.
Creating the Research Workflow
A simplified Temporal workflow could look like this:
from datetime import timedelta
from temporalio import workflow
@workflow.defn
class ResearchWorkflow:
@workflow.run
async def run(self, topic: str) -> dict:
findings = await workflow.execute_activity(
research_topic,
topic,
start_to_close_timeout=timedelta(minutes=10),
)
summary = await workflow.execute_activity(
summarize_findings,
findings,
start_to_close_timeout=timedelta(minutes=5),
)
return {
"topic": topic,
"findings": findings,
"summary": summary,
}The workflow coordinates the process, while activities perform external or potentially unreliable operations.
For example:
from temporalio import activity
@activity.defn
async def research_topic(topic: str) -> list[str]:
# Call search APIs, databases, or other services.
return [
f"Finding about {topic}",
f"Additional evidence about {topic}",
]
@activity.defn
async def summarize_findings(findings: list[str]) -> str:
# Call an LLM or summarization service.
return "Summary generated from the collected findings."This separation is especially useful for AI systems because model calls and external APIs are inherently failure-prone.
Building a Coordinator Agent
The coordinator can invoke the research capability and then pass the result to another agent.
A conceptual coordinator workflow might look like this:
@workflow.defn
class CoordinatorWorkflow:
@workflow.run
async def run(self, request: str) -> dict:
research = await call_research_service(
topic=request
)
analysis = await call_analysis_service(
evidence=research
)
return {
"research": research,
"analysis": analysis,
}The exact Nexus API depends on the Temporal SDK and Nexus version being used, but the architectural pattern remains the same: the coordinator treats the remote agent as a durable service capability rather than as a fragile HTTP endpoint.
That distinction becomes increasingly valuable as the system grows.
Keep Workflows Deterministic
Temporal workflows should remain deterministic.
This means that the workflow itself should not directly perform arbitrary network calls, generate random values using uncontrolled randomness, inspect the current system clock directly, or execute other operations that can produce different results when workflow history is replayed.
Instead, external operations belong in activities or appropriate Temporal primitives.
For example, avoid:
@workflow.defn
class BadWorkflow:
@workflow.run
async def run(self):
response = await http_client.get(
"https://example.com/api"
)
return response.json()A better architecture is:
@workflow.defn
class GoodWorkflow:
@workflow.run
async def run(self):
response = await workflow.execute_activity(
fetch_external_data,
start_to_close_timeout=timedelta(minutes=2),
)
return responseThe workflow records the activity execution in its durable history, allowing Temporal to reconstruct workflow state when necessary.
Designing Reliable Retries
AI services frequently interact with APIs that experience temporary failures.
A remote LLM provider might return a transient error. A database might be temporarily unavailable. A downstream service might experience a network timeout.
Activities can be configured with retry policies.
For example:
from temporalio.common import RetryPolicy
retry_policy = RetryPolicy(
initial_interval=timedelta(seconds=2),
backoff_coefficient=2.0,
maximum_interval=timedelta(minutes=1),
maximum_attempts=5,
)The activity can then use the policy:
result = await workflow.execute_activity(
call_llm,
prompt,
start_to_close_timeout=timedelta(minutes=5),
retry_policy=retry_policy,
)This is considerably safer than implementing retry loops manually inside application code.
However, retries must be designed carefully.
Not every failure should be retried.
For example:
Temporary network failure -> retry
Rate limit -> retry with backoff
Temporary service unavailable -> retry
Invalid authentication -> usually fail
Invalid request -> usually fail
Malformed business input -> usually failA durable system should distinguish transient failures from permanent failures.
Making Agent Operations Idempotent
Retries introduce another important concern: idempotency.
Suppose an execution agent creates a payment, provisions infrastructure, sends an email, or submits a transaction.
If the operation fails after the external system successfully processes it, Temporal may retry the activity.
Without idempotency, the operation could happen twice.
A common solution is an idempotency key.
@activity.defn
async def execute_order(order_id: str, operation_id: str):
existing = await database.find_operation(operation_id)
if existing:
return existing.result
result = await external_api.create_order(
order_id=order_id,
idempotency_key=operation_id,
)
await database.save_operation(
operation_id,
result,
)
return resultThe operation identifier becomes part of the durable contract.
This is particularly important when agents perform real-world actions rather than simply generating text.
Separating Agent Reasoning From Agent Execution
A useful multi-agent architecture separates reasoning from execution.
For example:
Coordinator
|
+--> Research Agent
|
+--> Planning Agent
|
+--> Approval
|
+--> Execution AgentThe planning agent can generate a proposed action:
{
"action": "provision_environment",
"region": "eu-central",
"size": "medium"
}The execution agent then validates the request before acting on it.
This creates a useful safety boundary.
The LLM does not directly control infrastructure.
Instead:
LLM output
|
v
Structured plan
|
v
Validation
|
v
Durable workflow
|
v
External side effectThis pattern helps prevent accidental execution caused by malformed or unexpected model output.
Passing Structured Data Between Agents
Agent-to-agent communication should use explicit schemas whenever possible.
Instead of passing unrestricted text:
await call_agent(
"Please investigate this and do whatever seems appropriate."
)prefer a structured request:
request = {
"task_id": task_id,
"objective": "research_product_market",
"topic": "enterprise AI platforms",
"max_sources": 20,
"output_format": "structured",
}The receiving agent can validate it before starting.
For stronger systems, define request and response objects using typed models.
For example:
from dataclasses import dataclass
@dataclass
class ResearchRequest:
task_id: str
topic: str
max_sources: int
@dataclass
class ResearchResponse:
task_id: str
findings: list[str]
confidence: floatTyped contracts make multi-agent systems easier to evolve.
Handling Long-Running Agent Operations
One of the strongest reasons to use durable workflows is that agents may need to wait.
Consider a workflow requiring human approval:
Research
|
v
Analysis
|
v
Generate Proposal
|
v
WAIT FOR APPROVAL
|
v
ExecuteA normal server process should not remain alive for hours while waiting for approval.
Temporal can suspend the workflow and resume it when the required signal or event arrives.
Conceptually:
@workflow.defn
class ApprovalWorkflow:
def __init__(self):
self.approved = False
@workflow.signal
async def approve(self):
self.approved = True
@workflow.run
async def run(self, proposal):
await workflow.wait_condition(
lambda: self.approved
)
return await workflow.execute_activity(
execute_proposal,
proposal,
start_to_close_timeout=timedelta(minutes=10),
)This is extremely useful for agentic systems because autonomous execution does not always need to mean completely unsupervised execution.
Timeouts Are Part of the Contract
Every remote operation should have sensible timeout behavior.
Consider a coordinator calling an analysis service.
A poorly designed system might wait indefinitely.
A better design defines:
Request timeout
Activity timeout
Workflow timeout
Retry limit
Business deadlineThese are different concepts.
An activity timeout determines how long an individual operation may run.
A workflow deadline determines how long the overall process is allowed to continue.
A business deadline might mean:
The report must be completed by 17:00.Temporal workflows can model these requirements explicitly instead of hiding them inside application servers.
Observability for Agent-to-Agent Systems
Debugging a multi-agent application can become difficult quickly.
A user may report:
“The agent took 20 minutes and produced no result.”
The actual execution path could have been:
Coordinator
|
+-- Research
| +-- Search API
| +-- LLM
| +-- Database
|
+-- Analysis
+-- LLM
+-- ValidationDurable workflow history gives the architecture a much stronger execution trail.
You should also assign a correlation identifier to every business operation:
task_id = "task-8f4a21"Pass that identifier through agent requests:
{
"task_id": "task-8f4a21",
"topic": "market-analysis"
}This makes it easier to connect logs, workflow executions, agent requests, external API calls, and business records.
Avoid Creating One Workflow for Every Thought
An important architectural warning is that not every LLM interaction needs to become a separate durable workflow.
A useful rule is:
Business process -> Workflow
External side effect -> Activity
Reusable remote capability -> Service operation
Agent reasoning -> LLM interactionIf every small model call becomes an independently addressable workflow, the system can become unnecessarily complex.
Instead, identify durable boundaries around meaningful business operations.
For example:
Research market
Analyze research
Prepare proposal
Execute approved proposalThese are meaningful operations.
Individual prompt construction steps usually are not.
Versioning Agent Workflows
Agent systems evolve rapidly.
Prompts change.
Models change.
Tool definitions change.
Business rules change.
That creates a challenge for long-running workflows.
A workflow that started yesterday may still be running after today’s deployment.
Temporal’s workflow versioning mechanisms are therefore important when changing workflow behavior in ways that affect deterministic replay.
For example, rather than abruptly changing behavior:
if new_logic:
...
else:
...use Temporal’s supported versioning mechanisms so existing executions can continue safely while new executions adopt the new implementation.
This becomes particularly important when workflows may run for hours, days, or longer.
A Practical End-to-End Architecture
A production-oriented agent platform might look like this:
User / Application
|
v
+---------------------+
| Coordinator Service |
+----------+----------+
|
Temporal Nexus
+-------------+-------------+
| |
v v
+---------------+ +---------------+
| Research | | Analysis |
| Service | | Service |
+-------+-------+ +-------+-------+
| |
v v
Activities Activities
| |
APIs / Search LLM / DB
| |
+-------------+-------------+
|
v
Execution Service
|
v
External SystemsEach service owns its implementation.
The coordinator knows capabilities.
The services own workflows.
Activities handle side effects.
Nexus provides durable service-to-service interaction.
Temporal provides the execution model underneath.
This architecture allows teams to develop agents independently without turning the entire application into one tightly coupled process.
A Simple Implementation Checklist
When building an agent-to-agent service with Temporal Nexus, consider the following checklist:
[ ] Define clear service capabilities
[ ] Use typed request and response contracts
[ ] Keep workflows deterministic
[ ] Put external operations in activities
[ ] Configure activity timeouts
[ ] Configure appropriate retry policies
[ ] Make side effects idempotent
[ ] Assign correlation/task identifiers
[ ] Define business deadlines
[ ] Handle permanent vs transient failures
[ ] Design long-running operations explicitly
[ ] Use signals or other durable interaction mechanisms when appropriate
[ ] Version workflows safely
[ ] Monitor workflow and activity failures
[ ] Keep agent responsibilities clearly separatedThis checklist captures the core difference between a demo multi-agent application and a production-oriented one.
Testing Durable Agent Systems
Testing should happen at several levels.
First, test individual activities:
async def test_research_activity():
result = await research_topic("AI infrastructure")
assert resultNext, test workflows with mocked activities.
Finally, test service-to-service interactions.
A useful integration test should simulate failures:
Research succeeds
|
v
Analysis fails
|
v
Retry
|
v
Analysis succeeds
|
v
Execution fails
|
v
Resume after worker restartThe goal is not merely to prove that the happy path works.
The goal is to verify that the system behaves correctly when things go wrong.
This is especially important for agentic applications because failure is not an unusual edge case. AI calls, network requests, external APIs, and human interactions all introduce uncertainty.
Security and Authorization
Agent-to-agent communication should also have explicit authorization boundaries.
Not every agent should be allowed to call every operation.
For example:
Research Agent
-> Search Service allowed
-> Execution Service denied
Analysis Agent
-> Research Service allowed
-> Execution Service read-only
Coordinator
-> Research allowed
-> Analysis allowed
-> Execution allowedThis principle reduces the blast radius of an incorrectly configured or compromised component.
Sensitive credentials should remain within the service that needs them rather than being passed through agent messages.
Likewise, an LLM should not receive credentials simply because an execution agent needs them.
The execution service should own the credentials and expose a narrow capability.
The Bigger Architectural Picture
The most important idea behind durable agent-to-agent systems is that agents should be treated as services with responsibilities, not simply as prompts with personalities.
A robust agent service has:
Capability
+
Contract
+
Durable execution
+
Failure handling
+
Observability
+
AuthorizationTemporal Nexus fits naturally into this model because it allows those capabilities to be exposed across service boundaries while Temporal provides durable execution underneath.
This means the architecture can evolve from a simple two-agent experiment into a larger system without requiring every component to communicate through fragile synchronous HTTP calls.
Conclusion
Building agent-to-agent systems is relatively easy when the goal is simply to demonstrate that one AI agent can call another. Building an agent-to-agent system that remains reliable when workers crash, APIs fail, requests take hours, humans need to approve actions, and external systems return unexpected results is a much more demanding engineering problem.
That is where durable execution becomes essential.
Temporal provides the foundation for modeling long-running business processes as durable workflows. Instead of treating an agent request as a single ephemeral function call, developers can represent it as a process that can pause, retry, recover, wait, and continue. Temporal Nexus extends this architectural idea to interactions between independently managed services, making it possible to build agent capabilities that can be consumed across service boundaries without forcing every agent into a tightly coupled synchronous architecture.
The key design principle is to establish clear boundaries.
A coordinator should coordinate. A research service should research. An analysis service should analyze. An execution service should perform controlled external actions. Each service should expose meaningful capabilities rather than leaking internal implementation details.
Workflows should orchestrate durable processes. Activities should handle external side effects. Agent communication should use structured contracts. Retries should be deliberate rather than indiscriminate. Operations that modify external state should be idempotent. Long-running processes should be allowed to wait without requiring a server process to remain alive. Human approval should be modeled as part of the workflow rather than implemented as an improvised polling mechanism.
These principles become particularly important as agentic applications move from prototypes into production. A prototype can tolerate an occasional lost request. A production system processing customer orders, financial operations, infrastructure changes, research pipelines, or other consequential tasks cannot.
Durability also changes how developers think about failure. Instead of designing the system around the assumption that everything will work, the architecture can assume that failures are normal. Workers can restart. Network connections can disappear. AI providers can temporarily become unavailable. APIs can rate-limit clients. A human can take hours to respond. A workflow can still preserve the overall business process through these events.
This is arguably the biggest advantage of combining agent architectures with durable execution: reliability becomes part of the programming model rather than something developers have to bolt onto every individual function.
Temporal Nexus is therefore most useful when viewed not merely as another way to make service calls, but as a way to create durable boundaries between autonomous capabilities. An agent can consume another agent’s capability without needing to know how that capability is implemented internally. The receiving service can manage its own workflows, activities, retries, credentials, data, and operational concerns.
The result is an architecture that looks less like a collection of interconnected AI prompts and more like a distributed system of durable services.
That distinction matters. As the number of agents grows, the primary challenge will not be getting agents to communicate. Modern AI systems can already communicate through APIs, queues, HTTP endpoints, and message buses. The difficult part will be ensuring that those interactions remain correct and observable across failures, retries, delays, deployments, version changes, and real-world side effects.
A durable agent-to-agent architecture addresses that problem directly. By combining well-defined service contracts, deterministic workflows, reliable activities, structured messages, idempotent operations, explicit authorization, observability, and Temporal Nexus, developers can build multi-agent systems that are not only capable but operationally dependable.
The ultimate goal should therefore not be to create the largest possible collection of agents. It should be to create well-defined, independently useful capabilities that can cooperate reliably over time. That is the architectural foundation on which production-grade agent-to-agent systems can be built.