Agent Orchestration on Google Cloud: a System-Level Guide to the Gemini Enterprise Agent Platform

A team ships its first production agent and something interesting happens. The next request from the business is never one more agent that does one more thing. It is "can this agent talk to the agent the data team just stood up," and "the support agent should be able to escalate to the billing agent without dropping the case." The single-agent question becomes a system question almost immediately, and the answers that worked for one agent stop working.

This is the orchestration moment. It is the point at which a successful agent practice either becomes a platform problem solved deliberately, or a glue-code problem solved badly. This guide is for engineering leadership making that call: what orchestration actually is at the substrate level, the patterns that cover the real topologies, the Gemini Enterprise Agent Platform components that carry the load, a worked multi-agent reference in real ADK code, and the honest negative space where a single agent is still the right answer.

The thesis: orchestration is the difference between an agent and a system. A system is what the business needs.

1. Why single agents stop scaling

The failure is not capability. A modern model running inside a well-built single-agent ADK project handles enormous scope. The failure is structural, and it shows up the moment a workflow crosses an ownership boundary, a runtime boundary, or a state boundary. Four failure modes recur, in roughly the order a team meets them.

Workflow glue becomes load-bearing infrastructure no one owns. The first cross-agent integration is a 30-line function that posts JSON from one agent to another. The second one copies the first, with slight differences. By the eighth, the codebase has a shadow integration layer: undocumented, untraced, mutable by whoever is on-call, and a critical workflow depends on it. Glue logic of this kind is not architecture; it is a half-built bus that the team has to maintain instead of building agents.

Case state lives in agent memory and dies at handoff. A claims workflow that starts in the intake agent, gathers evidence in the policy agent, and reaches a decision in the adjudicator carries enough context by the third hop that re-deriving it from the original request is genuinely lossy. Without a shared, durable state primitive, the team writes one: Firestore documents keyed by case ID, Redis hashes, ad-hoc JSON in cloud storage. Each is correct in isolation and wrong in aggregate, because the next multi-agent workflow does it differently.

Governance breaks at the seams. A single agent has one identity, one set of scoped credentials, one audit trail. Five agents wired together with no substrate produce five identities the platform team did not create, five credential rotations that drift apart, and five audit logs the security team has to correlate by timestamp. The compliance question, which agent touched this PHI record, on whose behalf, with what permission, has no clean answer.

Observability stops being useful at exactly the moment you need it most. A trace that ends at the orchestrator's outbound call and begins again, disconnected, at the worker's inbound call is two traces, not one. A production incident is then debugged by stitching log lines across services by hand at 2am, with a customer escalation open. The instrumentation that was sufficient for a single agent is no longer sufficient for the system, and nobody has the budget or the calm to retrofit it.

These four show up together. They are facets of the same underlying problem: the team has agents but no substrate. Orchestration is the substrate.

2. What orchestration actually means

The word gets used loosely, so let us be precise. Orchestration, in the agent context, is the coordination of multiple autonomous agents, each with its own state, tools, and decision loop, toward a single workflow outcome, mediated by a protocol substrate that handles message exchange, state continuity, identity, and audit as platform concerns rather than per-integration code.

Three pieces of that definition do real work.

Autonomous agents. An orchestrated system is not a pipeline of LLM calls, and a chained prompt is not multi-agent orchestration. The constituent agents have their own instructions, their own tools, and their own runtime loops. They make local decisions. The orchestration layer coordinates the decisions; it does not make them.

A single workflow outcome. The agents are coordinating for a purpose: adjudicate this claim, resolve this incident, complete this onboarding. The topology of agents exists to reach it. Agents deployed together that pursue independent goals are a fleet, not an orchestration.

A protocol substrate. This is the part the literature most often elides. The coordination has to run over something: a message protocol, a state store, a registry, a policy enforcement point. Without these, agents communicate through bespoke glue and the failure modes from section 1 are inevitable. With them, coordination is a property of the platform rather than a property of the code you write.

Worth distinguishing from three adjacent things orchestration is not. A prompt chain of the form model_a.generate(...) → model_b.generate(...) is not orchestration. The thing being chained has no autonomy, no tools of its own, no decision loop. A function call from a model is not orchestration: the model invoking a Python function is the agent reaching its tools, and the function is not an agent. A deterministic pipeline running on Airflow or Step Functions is not orchestration in the agent sense, though it can be one of the substrates an orchestrator uses. The defining quality of multi-agent orchestration is that routing and sequencing are partly decided by the agents themselves in response to what they observe. A pipeline that always runs the same three steps is a pipeline. A pipeline where the orchestrator agent decides whether a given case needs the policy agent based on what the intake agent surfaced is orchestration.

3. Patterns for multi-agent orchestration

There is a small library of topologies that cover almost every real multi-agent system. The right move is to learn them as patterns and pick by problem shape, not by defaulting to the most expressive one. Five patterns earn their place.

A few notes on choosing.

The supervisor-worker pattern is the right default for the first multi-agent system a team builds. It is the most legible, the easiest to instrument, and the easiest to extend: a new specialist is a new sub-agent under the existing supervisor. Most production systems that describe themselves as "multi-agent" are this pattern.

The sequential pipeline is right when the sequence is genuinely fixed and the agent boundaries map to stable interfaces. A document processing flow where intake parses, the classifier categorises, the extractor pulls fields, and the validator checks them is naturally a pipeline. Forcing it through a supervisor adds a coordination layer with no benefit.

The parallel fan-out pattern earns its place when the latency win is real. Adjudicating 200 line items in parallel and aggregating beats serialising them, but only if the aggregator is itself well-designed. Many teams write the dispatcher and the workers and underinvest in the aggregator, which is where the actual user-visible synthesis happens.

The peer mesh is rarely the right answer. It is the most flexible topology and the most expensive to govern: there is no single place the whole interaction is visible, which means audit, observability, and incident response all get harder. Choose it deliberately, when a specific requirement forces it, not as a hedge against flexibility you might want.

The hierarchical pattern is for scale. A platform team running 30 agents may have an HR supervisor delegating to onboarding and benefits workers, a finance supervisor delegating to invoicing and close workers, and a top-level router selecting which supervisor a request goes to. Rare, expensive to build well, but the only thing that holds together past a certain scale.

4. The Google Cloud substrate

Orchestration is a substrate problem. The Gemini Enterprise Agent Platform names a primitive for each gap section 1 identified, and the value of the platform is that the substrate is one substrate, not five products glued together. Walk the layers from the ground up.

Agent Runtime: managed execution and scale

Agent Runtime is the managed runtime that hosts an agent in production. The same agent object that runs locally under adk run deploys to Agent Runtime without a rewrite, picks up autoscaling, gets a stable endpoint, and is observable from the first request. For an orchestrated system, Agent Runtime is where every agent in the topology lives, including the supervisor. The supervisor is not infrastructure; it is an agent, with its own runtime, identity, and trace surface, exactly like the workers it coordinates. Every agent has the same deployment shape, so the topology reads as a set of peers in a runtime fabric rather than one orchestrator plus a heterogeneous backplane.

Agent2Agent (A2A) protocol: the message bus

Agent2Agent (A2A) protocol is the open, vendor-neutral wire protocol agents speak to each other. The A2A v1.2 specification carries a formal Agent Card definition, a task state machine, signed-card verification, and gRPC as a first-class transport. The protocol now sits inside the Linux Foundation with AWS, Microsoft, Salesforce, SAP, and ServiceNow as named participants. For multi-agent orchestration, A2A is the message bus: a supervisor invokes a worker over A2A, a worker reports task progress back over A2A, a peer in a mesh discovers another peer's Agent Card and calls it over A2A.

The detail that matters for orchestration design is that A2A is protocol-level interop, not framework-level coupling. A supervisor written in ADK can call a worker written in any A2A-compliant framework, including agents from other vendors. The orchestration boundary is not a Google-product boundary, which is the right design for a real enterprise where the data team's agent, the IT team's agent, and the partner integration are all in the same workflow. Our A2A v1.2 enterprise interoperability guide walks the protocol mechanics in full; this article is the wider orchestration story.

Memory Bank: durable shared state

Memory Bank is the managed cross-session memory service that holds the durable state of a workflow across agent boundaries and across conversations. For orchestration, Memory Bank answers the second failure mode from section 1: case state has a home that is not the agent's local variables.

The distinction worth holding clearly: Memory Bank is not the in-conversation working state (ADK's Session state covers that, scoped to one exchange). Memory Bank is the across-conversation, across-agent durable layer. A claims case opened on Monday and resumed on Wednesday by a different agent in the topology finds its full prior context in Memory Bank; reconstructing it from log scraping would be an architectural failure.

Agent Registry: the control plane

Agent Registry is the platform's catalog of agents. Every agent (supervisor, worker, third-party) is enrolled, and the registry holds its identity, capabilities, observability hooks, and policy bindings. For an orchestrated estate, Agent Registry is what makes "we have a multi-agent system" answerable to the security team. It is also what makes per-agent identity practical: each registered agent has its own IAM principal with credentials scoped to the resources its role requires. Without it, the agent estate is a folklore artifact maintained in someone's head.

Agent Gateway: runtime policy enforcement

Agent Gateway is the runtime policy enforcement point on agent traffic: authentication, authorization, rate limiting, content inspection through Model Armor. For orchestration, the Gateway is where cross-agent calls land before they reach the receiving agent's logic. PHI redaction, prompt-injection screening, and per-agent rate caps are applied uniformly here rather than per-agent in code. A supervisor calling a worker over A2A is screened by the gateway exactly as an external caller would be. There is no privileged in-cluster path that skips the controls.

Vertex AI and Cloud Run: the underlying compute

Vertex AI Platform carries the model layer: Gemini Pro and Flash, MedLM, the Model Garden's third-party models. Cloud Run is the serverless compute backplane several Agent Runtime deployment targets are built on. These are the layers below the agent-aware substrate: necessary, foundational, and uninteresting to topology design once they are in place. An orchestration architect should know they are there and let the agent layer abstract them.

5. A concrete reference architecture: claims adjudication

Patterns are abstract, components are abstract, substrate is abstract. Walk a real multi-agent system end-to-end and the abstractions fall into place.

The worked example is claims processing adjudication: the topology a regional payer uses to clear routine claims without a human and route exceptions to a named adjudicator with the case file already assembled. The full engagement shape lives in the claims processing adjudication blueprint; the engineering view of the topology is what this section covers.

The topology

Three agents, supervisor-worker pattern.

• Intake supervisor receives the inbound claim, validates the payload, looks up the member's policy, and decides whether to delegate to the adjudicator (routine case) or to the exception router (anomalous case).
• Adjudicator worker applies the policy library to the claim, cites the clauses, records the decision. Returns approved, denied, or needs_human_review with a structured rationale.
• Exception router worker assembles the case file for human review (policy snippets, member history, anomaly flags) and hands it off to a named adjudicator with the A2A audit trail attached.

The intake supervisor owns the workflow; the workers do their specialised work and return; Memory Bank holds the case state across the handoffs so any agent in the topology can pick up the case at any point.

The wire

Three substrate decisions wire the topology together.

• A2A v1.2 between agents. The intake supervisor calls the adjudicator over A2A; a needs_human_review outcome triggers an A2A handoff to the exception router. Each call carries signed Agent Card verification, scoped OAuth credentials, and W3C trace context propagation. Even though the agents could be deployed adjacent, the boundary is the protocol.
• Memory Bank for case state. A claim case ID is the durable handle. The intake supervisor writes the validated payload and member policy snapshot to Memory Bank under the case ID; the adjudicator reads them and writes its decision back; the exception router reads everything and writes the assembled case file. No state lives in agent variables across handoffs.
• Agent Registry enrollment for all three. Each agent has its own IAM principal, its own scoped credentials, its own audit attribution. The platform team can answer "which agent touched this case, in what order, with what permissions" from a single registry query.

The code

The ADK shape of the topology, in real Python. This is not the whole system (the full agent code lives in the engagement repo and includes the tool implementations, the Memory Bank client wiring, the policy library connector, and the eval set), but it is the load-bearing skeleton.

# agents/intake_supervisor/agent.py
from google.adk.agents import LlmAgent
 
from .a2a_tools import call_adjudicator, call_exception_router
from .tools import (
    validate_claim_payload,
    fetch_member_policy,
    persist_case_to_memory_bank,
)
from .prompts import INTAKE_SUPERVISOR_INSTRUCTION
 
 
intake_supervisor = LlmAgent(
    name="claims_intake_supervisor",
    model="gemini-2.5-pro",
    description=(
        "Receives an inbound claim, validates it, snapshots the member "
        "policy into Memory Bank, and delegates to the adjudicator or the "
        "exception router. Owns the workflow; does not adjudicate."
    ),
    instruction=INTAKE_SUPERVISOR_INSTRUCTION,
    tools=[
        validate_claim_payload,
        fetch_member_policy,
        persist_case_to_memory_bank,
        call_adjudicator,
        call_exception_router,
    ],
)
 
 
# agents/adjudicator/agent.py
# The worker runs as its own deployed Agent Runtime service, registered
# in Agent Registry, addressable over A2A. The intake supervisor reaches
# it through call_adjudicator above.
from google.adk.agents import LlmAgent
 
from .tools import (
    fetch_case_from_memory_bank,
    fetch_policy_clauses,
    fetch_member_history,
    record_adjudication,
)
from .prompts import ADJUDICATOR_INSTRUCTION
 
 
adjudicator = LlmAgent(
    name="claims_adjudicator",
    model="gemini-2.5-pro",
    description=(
        "Applies the policy library to a claim. Cites the clauses applied, "
        "records the decision in Memory Bank, returns approved/denied/"
        "needs_human_review with structured rationale."
    ),
    instruction=ADJUDICATOR_INSTRUCTION,
    tools=[
        fetch_case_from_memory_bank,
        fetch_policy_clauses,
        fetch_member_history,
        record_adjudication,
    ],
)

Two things are worth noticing about the shape.

First, the intake supervisor's tools include call_adjudicator and call_exception_router. These are A2A client tools. From the model's perspective they look like any other function tool; under the hood they perform Agent Card discovery, signature verification, OAuth token acquisition, and A2A task submission, then track the remote task to a terminal state. The model decides when to delegate; the platform handles how. The detailed mechanics are walked in our Agent2Agent enterprise interoperability guide.

Second, the adjudicator's first tool is fetch_case_from_memory_bank. The adjudicator does not receive the full case payload in its prompt. It receives the case ID and reads the case from Memory Bank. This keeps the prompt small (no PHI in transit beyond an opaque ID), keeps the audit story clean (the Memory Bank read is logged against the adjudicator's identity), and means the adjudicator's behavior is fully reproducible from the case ID alone.

What you get from the topology

A clean reasoning surface. Asking what happened in this claim case yields one answer: a trace across the three agents, a Memory Bank object history, an Agent Registry audit log keyed to the case ID. Asking why this decision yields a structured rationale citing specific policy clauses. Asking who has touched this case yields a per-agent attribution.

Compare to the same workflow built as a single agent: a 3000-token instruction with a sprawling toolset, a brittle in-conversation state model, and a single identity touching everything. Impossible to evolve, too: prompt tweaks on the adjudication path would have to be reasoned about jointly with the routing logic and the intake validation. The multi-agent topology buys evolvability as a side effect of doing the orchestration honestly.

6. Governance and audit at multi-agent scale

The most common objection to multi-agent orchestration is governance: "more agents means more attack surface, more identities to manage, more audit trails to reconcile." On the wrong substrate, the concern is correct. On the Gemini Enterprise Agent Platform it inverts.

Per-agent identity is the default, not the project

The single most important governance property of an orchestrated system is that every agent has its own IAM principal. This makes minimum-necessary access enforceable (an agent cannot retrieve resources outside its scope because its credential does not reach them), makes audit attribution honest (every PHI access is attributed to the agent that performed it, not to a shared service account), and makes lifecycle control clean (decommissioning an agent revokes its access in one step).

On a glue-code orchestration, per-agent identity is a project: you design the credential model, build the rotation pipeline, plumb every cross-agent call through scoped tokens. On the Gemini Enterprise Agent Platform, an agent registered in Agent Registry gets a scoped IAM identity as part of registration. The default is right, and the default is what you ship under deadline pressure.

The Gateway is one seam, not many

Every cross-agent call lands at the Agent Gateway. Model Armor's PHI redaction, prompt-injection screening, sensitive-data inspection, rate limiting: all of it happens at the Gateway, uniformly, regardless of which agent is calling which. A common failure on hand-rolled multi-agent systems is that controls are present on the public-facing agent and absent on the agent-to-agent traffic, because "those are internal." The Gateway closes that. A supervisor calling a worker over A2A is screened by the same Model Armor policy that screens the inbound user request, and so the protection on the worker is real, not assumed.

Unified observability

A trace that follows a single workflow across three agents, four Memory Bank reads, and a model call, all rendered as one connected timeline in Cloud Trace, is what makes a multi-agent system debuggable. The platform's tracing instrumentation handles this natively: A2A calls propagate W3C trace context, Agent Runtime emits OpenTelemetry spans, the platform stitches the picture together. The negative version: a multi-agent incident debugged by correlating log lines across services by hand is a 4-hour incident. The same incident debugged from a single trace is 20 minutes.

7. When NOT to orchestrate

A guide that only argues for orchestration is a sales document. There are real cases where a single agent is the right answer and reaching for multi-agent is over-engineering. Three.

A narrow workflow with one set of tools and one decision loop. If the work is fundamentally one job (classify a ticket, draft a code review comment, summarise a meeting), a single agent does it cleanly. Splitting the work across multiple agents adds a coordination layer with no functional benefit. The Engineering Code Review blueprint is a single-agent system on purpose; the responsibilities do not naturally split.

A workflow that cleanly maps to a deterministic pipeline. If the sequence is fixed, the agents do not need to decide what to do next, and the decision points are rules rather than judgement, an orchestrated multi-agent system is overkill. A deterministic workflow engine running an agent at one or two steps is cheaper, faster, and easier to operate.

A team that has not yet shipped one agent in production. Multi-agent orchestration on top of a team that does not have one production agent is a project that will not finish. Ship the first single-agent system, learn the operational shape (deploys, evals, traces, incidents), then evaluate whether the second workload calls for orchestration. Reaching for multi-agent before the operational muscle exists is a complexity bet a young agent practice cannot afford.

The honest rubric: orchestrate when the coordination need is real, the substrate is in place, and the team has operational maturity. Otherwise, build a better single agent.

8. The implementation path

A team with one production agent and a roadmap that obviously needs more has a concrete path to orchestrated multi-agent. Five steps.

1. Pick the first multi-agent workflow deliberately. Choose the workflow where the second agent is genuinely a different specialist (different instruction, different tools, different model tier), not a refactor of the first. Two agents with substantially the same shape is a code-organisation problem, not an orchestration problem.
2. Enroll the existing agent in Agent Registry. Precondition for every subsequent step. The existing agent gets its scoped IAM identity, Gateway-mediated traffic, and unified audit footprint. The retrofit on a single agent is cheap; the retrofit on five is not.
3. Design the topology before writing the code. Pick the pattern from section 3 explicitly and document why. The pattern decision is the architecture; the code is the implementation. Teams that skip this step end up with whatever shape emerged from the first PR.
4. Wire A2A and Memory Bank from the first interaction. The second agent calls the first over A2A even if they are deployed adjacent in the same cluster. State that crosses the boundary lives in Memory Bank from the first commit. There is no "we will add A2A later"; the substrate is the architecture, not a layer that gets added.
5. Treat governance and observability as table stakes, not Q4 work. Per-agent identity, Gateway-mediated traffic, Model Armor policies, unified trace propagation: all wired in the first sprint. A working-but-ungoverned multi-agent system is the system that gets blocked at security review.

This is the shape of a Zinch Platform engagement on multi-agent orchestration: typically a 10-week build, two or three agents, the substrate properly wired, the eval harness covering the orchestration paths (not just the per-agent behaviors), the runbook for the on-call team. The enterprise approach page covers how the engagement is staffed and how the standards we hold to in the build show up in the artifacts the customer team owns at the end.

The orchestration moment is when an agent practice becomes a platform. The work is real, the substrate is real, and the result, a coordinated system of agents that runs a business workflow rather than answering a workflow question, is the version of agent ROI that compounds. Talk to us about your second agent before it becomes glue code that is harder to remove than to write right the first time.

9. FAQ