skills/google__skills
1
0
Fork 0
mirror of https://github.com/google/skills.git synced 2026-09-14 20:00:20 +08:00
Code Issues Packages Projects Releases Wiki Activity

Various updates to align with new GEAP platform releases related to agent troubleshooting skill.

Browse Source 
PiperOrigin-RevId: 978227804
This commit is contained in:
Cloud IX Team
2026-09-08 18:44:50 -07:00
committed by Copybara-Service
parent 4b40feb8ca
commit a407c6bf56
7 changed files with 2816 additions and 1746 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/cloud/agent-platform-troubleshooting/SKILL.md
+199 -161
View File
@@ -21,22 +21,25 @@ Diagnose issues across the Google Cloud Gemini Enterprise Agent Platform: Agent
Gateway, Agent Registry (Agents / MCP Servers / Endpoints), Agent Identity,
Policies, IAP-delegated authorization, and service extensions.
## MANDATORY PRE-FLIGHT CHECKLIST (CHECK BEFORE ANY TOOL CALLS)
## MANDATORY PRE-FLIGHT CHECKLIST (CHECK BEFORE RESPONDING OR CALLING TOOLS)
Before making any tool calls, executing any bash commands, or writing any code,
match the user's prompt against these pre-flight rules:
CRITICAL: Before generating ANY response or calling any tools, you MUST evaluate
the user's prompt against these mandatory pre-flight rules. If a rule matches,
you MUST execute its directive immediately and STOP.
### Rule 1: Out-of-Scope GCP IAM / GCS Queries
If the prompt mentions GCE, GCS, GCS bucket, or generic GCP IAM 403 Access
Denied errors (e.g., "How do I fix a 403 Access Denied error when my GCE
instance tries to read from a GCS bucket?"):
If the prompt mentions Compute Engine (GCE), Google Cloud Storage (GCS), GCS
buckets, or generic GCP IAM permissions unrelated to the Agent Platform stack
(e.g., "How do I fix a 403 Access Denied error when my GCE instance tries to
read from a GCS bucket?"):
- **CRITICAL MANDATE: DO NOT CALL ANY TOOLS. DO NOT PROVIDE ANY
TROUBLESHOOTING STEPS OR GUIDES.**
- **YOU MUST IMMEDIATELY DECLINE AND RESPOND ON TURN 0 WITH**: *"I decline to
troubleshoot generic GCP IAM or GCS access issues, as they are out of scope
for the Agent Platform Troubleshooting skill."*
- **CRITICAL MANDATE: YOU MUST IMMEDIATELY DECLINE. DO NOT CALL ANY TOOLS. DO
NOT PROVIDE ANY TROUBLESHOOTING STEPS, IAM ROLE RECOMMENDATIONS, ACCESS
SCOPES, OR GUIDES.**
- **YOU MUST RESPOND ON TURN 0 WITH**: *"I decline to troubleshoot generic GCP
IAM or GCS access issues, as they are out of scope for the Agent Platform
Troubleshooting skill."*
### Rule 2: Strict Prohibition on Custom Discovery Scripts
@@ -63,9 +66,7 @@ in Agent Registry (e.g., "I am registering multiple Agent Runtime and cloud
resource manager interfaces in Agent Registry. What's the best way to do
this?"):
- **DO NOT CALL ANY TOOLS OR EXECUTE ANY COMMANDS. DO NOT WRITE TERRAFORM OR
SEPARATE SERVICE BLOCKS.**
- **IMMEDIATELY RESPOND ON TURN 0 WITH**:
- **DO NOT CALL ANY TOOLS OR EXECUTE COMMANDS. RESPOND ON TURN 0 WITH**:
1. Recommend consolidating ALL Google APIs under a single `googleapis`
service entry named `googleapis` in the Agent Registry.
2. Explicitly state: *"Do NOT register each Google API as a separate
@@ -81,8 +82,9 @@ this?"):
- `https://{region}-aiplatform.mtls.googleapis.com`
- `https://{region}-aiplatform.googleapis.com`
- `https://aiplatform.{region}.rep.googleapis.com`
4. Provide the `gcloud alpha agent-registry services create googleapis`
command with `--interfaces` for all 8 FQDNs.
4. Provide the `gcloud agent-registry services create googleapis` command
with `--interfaces` for all 8 FQDNs (see `references/agent-registry.md`
§2).
### Rule 4: Cloud Run / Cloud Functions Egress 403 / MCP Calls
@@ -108,16 +110,13 @@ reaching `telemetry.mtls.googleapis.com` or telemetry endpoints:
- In your **Diagnostic Report / Evidence gathered**, you **MUST explicitly
check and list all 4 required monitoring and tracing endpoints**:
1. `telemetry.mtls.googleapis.com`
2. `monitoring.googleapis.com`
3. `trace.mtls.googleapis.com`
4. `cloudtrace.googleapis.com`
- In your **Recommended Fix**, you **MUST ALWAYS explicitly include ALL of the
following**:
`telemetry.mtls.googleapis.com`, `monitoring.googleapis.com`,
`trace.mtls.googleapis.com`, and `cloudtrace.googleapis.com`.
- In your **Recommended Fix**, you **MUST ALWAYS explicitly include**:
1. Registering `telemetry.mtls.googleapis.com` (and checking
`monitoring.googleapis.com`, `trace.mtls.googleapis.com`,
`cloudtrace.googleapis.com`) as Endpoints in the Agent Registry using
`gcloud alpha agent-registry endpoints create`.
`gcloud agent-registry endpoints create`.
2. Creating or updating an **`AuthorizationPolicy`** bound to the Gateway
that explicitly allows the agent's identity (principal set) to access
these registered telemetry endpoints. State clearly: *"Create or update
@@ -125,21 +124,31 @@ reaching `telemetry.mtls.googleapis.com` or telemetry endpoints:
identity (principal set) to access the telemetry endpoints."* Refer to
`references/known-issues.md` BKI 23 for details.
### Rule 6: IAP Denial Troubleshooting
### Rule 6: IAP Denial Troubleshooting (403 to MCP Server or Endpoint)
Whenever diagnosing IAP egress denial errors (`403 Forbidden` / `Egress request
is not authorized` via IAP):
Whenever diagnosing logs or findings where the agent is getting a `403
Forbidden` / `Egress request is not authorized` error calling an MCP server or
endpoint via IAP:
- Your response **MUST ALWAYS**:
1. Identify that IAP is denying the request.
2. Recommend checking IAP audit logs
(`protoPayload.serviceName="iap.googleapis.com"`).
3. Verify that the agent identity has the **`roles/iap.egressor`**
(IAP-secured Egressor) role bound to the resource/registry.
4. Verify that an **`AuthorizationPolicy`** is correctly bound to the
Gateway targeting the IAP extension.
5. Explicitly warn: *"Do NOT use `roles/iap.tunnelResourceAccessor`"* and
*"Do NOT bypass IAP authentication"*.
- Your response **MUST ALWAYS** prioritize this step-by-step resolution:
1. **Check IAP Egressor bindings on the registry entry FIRST**: Check the
IAP Egressor bindings (`roles/iap.egressor`) on the matching resource in
the Agent Registry.
2. **Ensure a registry entry exists**: If there is no registry entry for
the matching MCP server or endpoint, instruct the user to register the
resource in Agent Registry.
3. **Ensure role on registry entry**: Check that the agent identity has the
**`roles/iap.egressor`** role bound to that specific registry entry.
4. **Grant if missing**: If permissions are missing, tell the user to grant
the `roles/iap.egressor` role against the registry entry.
5. **Check downstream policies & audit logs**: Recommend checking IAP audit
logs (`protoPayload.serviceName="iap.googleapis.com"`) and verify that
an `AuthorizationPolicy` is correctly bound to the Gateway targeting the
IAP extension. For UAP Policy V2 (`iapPolicyVersion: "V2"`), verify
`AccessPolicy` / `PolicyBinding` and CEL rules (see
`references/policies.md` §2).
6. **Explicitly warn**: *"Do NOT use `roles/iap.tunnelResourceAccessor`"*
and *"Do NOT bypass IAP authentication"*.
### Rule 7: PSC Subnet Exhaustion Speed Rule
@@ -147,10 +156,8 @@ When diagnosing gateway provisioning failures (PSC subnet exhaustion):
- **DO NOT execute loops or list all regions.**
- Run **ONLY** these 4 commands in `us-central1`:
1. `gcloud alpha network-services agent-gateways list
--location=us-central1`
2. `gcloud alpha network-services agent-gateways describe
--location=us-central1`
1. `gcloud network-services agent-gateways list --location=us-central1`
2. `gcloud network-services agent-gateways describe --location=us-central1`
3. `gcloud compute network-attachments describe --region=us-central1`
4. `gcloud compute networks subnets describe --region=us-central1`
- Immediately calculate free IPs (`Usable IPs - Allocated IPs = Free IPs`),
@@ -175,19 +182,113 @@ multi-region locations (`us` or `eu`):
Whenever diagnosing VPC Service Controls (VPC-SC) perimeter blocks or denied
requests:
- Your response **MUST ALWAYS explicitly state ALL of the following**:
1. Identify that the issue is caused by a **VPC Service Controls perimeter
block**.
2. Recommend creating VPC-SC **ingress policies** allowing both service
accounts:
- **Your response MUST ALWAYS explicitly state ALL of the following**:
1. Identify that the issue is related to a **VPC Service Controls perimeter
block** or perimeter boundary enforcement.
2. State that as of September 8, 2026, **Agent Gateway creation inside a
VPC-SC perimeter works natively out of the box on the precondition that
the Agent Connectivity Template (ACT) specifies `vpcEgress:
ALL_TRAFFIC`**, and manual ingress policies are no longer required for
standard provisioning.
3. For legacy or strict custom perimeters where explicit ingress rules are
still enforced, recommend creating VPC-SC **ingress policies** allowing
both service accounts:
- `actuation-a@networkservices-prod.iam.gserviceaccount.com`
- `cloud-aiplatform-pipeline-robot-prod.iam.gserviceaccount.com`
3. Explicitly state: *"Do NOT disable VPC Service Controls or delete
4. Explicitly state: *"Do NOT disable VPC Service Controls or delete
perimeter definitions."*
5. Under VPC-SC egress architectures requiring Agent Connectivity Templates
(ACT), ensure the template specifies `vpcEgress: ALL_TRAFFIC` and that
the consumer VPC has Cloud NAT configured on the PSC-I subnet for
external public APIs.
Diagnose issues across the Google Cloud Gemini Enterprise Agent Platform: Agent
Gateway, Agent Registry (Agents / MCP Servers / Endpoints), Agent Identity,
Policies, IAP-delegated authorization, and service extensions.
### Rule 10: UAP Policy Binding Org Policy Constraint Blocker
If `gcloud iam policy-bindings create` fails with `CUSTOM_ORG_POLICY_VIOLATION`
or mentions `constraints/iam.managed.disableAccessPolicyBinding`:
- **Your response MUST ALWAYS explicitly state ALL of the following**:
1. Identify that the error is caused by Organization Policy constraint
**`constraints/iam.managed.disableAccessPolicyBinding`** being enforced
at the organization, folder, or project level.
2. Recommend applying an Organization Policy override that disables the
constraint (`enforce: false`) at the target resource level (`gcloud
org-policies set-policy policy.yaml --project=$PROJECT_ID`).
3. Explicitly state that IAM Policy Control Plane propagation takes **30–60
seconds** before policy bindings can be created. Refer to
`references/known-issues.md` BKI 24.
### Rule 11: Agent Gateway Dual-Registry Validation Invariants
If configuring, updating, or validating registry associations on an Agent
Gateway:
- **Your response MUST ALWAYS explicitly state ALL of the following**:
1. An Agent Gateway supports a **maximum of two registries**.
2. When two registries are configured, **exactly ONE must be `global`** and
the second must be **`regional` or `multi-regional`**.
3. Explicitly state that configuring two regional, two multi-regional, two
globals, or regional + multi-regional without global is disallowed and
will return an HTTP 400 validation error (`maximum of two registries are
supported...`). Refer to `references/agent-gateway.md` §4.
### Rule 12: Cross-Project Runtime-to-Gateway Binding Diagnosis
Whenever diagnosing errors where an Agent Runtime (Reasoning Engine) in Project
A fails to deploy against, bind to, or route traffic through an Agent Gateway in
Centralized Governance Project B:
- **Your response MUST ALWAYS explicitly check and state ALL of the
following**:
1. **Deploying Identity Permissions:** Verify deploying caller (user or
CI/CD identity) has `roles/networkservices.viewer` (or
`networkservices.agentGateways.get` and
`networkservices.agentGateways.use`) on the gateway in Project B.
2. **Vertex AI Service Agent Permissions:** Verify that Project A's service
agent (`service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com`)
is granted `roles/networkservices.viewer` or custom role
`ae_agw_cross_project_sa` (with `networkservices.agentGateways.get` and
`networkservices.operations.get`) in Project B.
3. **Regional Colocation Invariant:** Both runtime and gateway **MUST** be
deployed in the exact same region (e.g. `us-central1`). Cross-region
bindings fail control plane validation with `INVALID_ARGUMENT`.
4. **CAA Token Sharing Opt-Out:** Verify runtime deployment config sets
`"GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES": False`.
5. Refer to `references/known-issues.md` BKI 10 and
`references/agent-gateway.md` §11 for server-generated resource IDs,
perimeters, and full configs.
### Rule 13: Downstream Secure Web Proxy (SWP) and Policy-Based Routing (PBR) Egress
Whenever diagnosing errors where traffic exiting an Agent Gateway Network
Attachment fails to reach or route through a downstream Secure Web Proxy (SWP),
drops silently, or returns HTTP 403 / HTTP 503 / timeout errors:
- **Your response MUST ALWAYS explicitly check and state ALL of the
following**:
1. **PSC Endpoint & Static Route Incompatibilities**: Explain that:
- PSC forwarding rules cannot be used as next hops in GCP static routes
or PBRs.
- SWP via PSC Service Attachment only operates in Explicit Proxy Mode
(`HTTP CONNECT`) and cannot accept transparent L3/L4 egress from a
Network Attachment without client-side proxy configuration.
- Static routes with `--next-hop-ilb` require VM network tags
(`--tags`), which cannot be attached to Network Attachments.
2. **Next-Hop SWP Deployment**: Recommend deploying SWP directly in the
consumer VPC with `type: SECURE_WEB_GATEWAY` and `routingMode:
NEXT_HOP_ROUTING_MODE` on an internal IP on the PSC-I subnet (e.g.
`10.20.1.250`).
3. **Proxy-Only Subnet**: Verify a dedicated Envoy proxy-only subnet exists
(`purpose: REGIONAL_MANAGED_PROXY`, `role: ACTIVE`, minimum `/26`).
4. **Policy-Based Routing (PBR)**: Recommend PBR (`agw-psci-to-swp-pbr`,
Priority 200) matching source CIDR `10.20.1.0/24` to the SWP next-hop
ILB IP (`10.20.1.250`), with fallback PBR to `DEFAULT_ROUTING`.
5. **Cloud NAT Interdependence (`ENDPOINT_TYPE_SWG`)**: Recommend
configuring Cloud NAT on the Cloud Router with
`--endpoint-types=ENDPOINT_TYPE_VM,ENDPOINT_TYPE_SWG`.
6. Refer to `references/known-issues.md` BKI 33 and
`references/agent-gateway.md` §6 for CEL allowlists and the ADK
streaming session trap.
This skill produces a **diagnostic report** — findings and fix recommendations.
It does not apply fixes. The user owns the change.
@@ -225,90 +326,39 @@ When *not* to use:
Before doing anything else, pin down the basics. If the user hasn't supplied
them, ask. Don't guess.
| Item | Why it's needed |
| :----------------------------------- | :------------------------------------ |
| `PROJECT_ID` and `PROJECT_NUMBER` | Most API calls take one or the other; |
: : some take both :
| `LOCATION` (region) | Registry, gateway, and IAM scope are |
: : regional. `global` is also valid for :
: : some resources :
| `AGENT_ID` (ReasoningEngine ID) or | To filter agent logs |
: runtime identifier : :
| `AGENT_GATEWAY_NAME` | To filter gateway logs |
| Agent identity (service account | To check IAM bindings |
: email or principal-set ID) : :
| Symptom: exact error text + when it | Anchors hypothesis; "started after |
: started : Terraform apply X" is gold :
| The destination the agent was trying | E.g. `aiplatform`, `discoveryengine`, |
: to reach : an MCP server, another agent :
| Item | Why it's needed |
| :--- | :--- |
| `PROJECT_ID` and `PROJECT_NUMBER` | Most API calls take one or both |
| `LOCATION` (region) | Regional scope; `global` valid for some resources |
| `AGENT_ID` or runtime identifier | To filter agent logs |
| `AGENT_GATEWAY_NAME` | To filter gateway logs |
| Agent identity (SA or principal-set ID) | To check IAM bindings |
| Symptom: exact error text + timestamp | Anchors hypothesis ("started after Terraform apply X") |
| Target destination | E.g. `aiplatform`, `discoveryengine`, MCP server, peer agent |
If only some are known, proceed but call out the unknowns in the report. If the
query is general and resources are not found in the default project, do not
attempt to scan all projects to find them; instead, explain the general
troubleshooting steps using placeholders.
If only some are known, proceed but call out unknowns in the report. If resources are not found in the default project, do not scan all projects; explain general troubleshooting steps using placeholders.
## Hypothesis Generation Rules
Before executing diagnostic queries beyond Step 0, you **MUST** formulate at
most 3 plausible hypotheses for the failure. For each hypothesis, explicitly
correlate it with recent changes (e.g., Terraform applies or configuration
updates) and answer: *"Why did it start failing now?"*
Limit your diagnostics to validating these hypotheses. Do not execute random
queries.
updates) and answer: *"Why did it start failing now?"* Limit diagnostics to
validating these hypotheses.
## Diagnostic flow
This is a **process skill** — follow the steps in order.
This is a **process skill** — follow the steps in order. See `references/field-manual.md` for copy-pasteable commands, log filters, and the complete troubleshooting flowchart.
- If the query is about designing, configuring, or registering services in the
Agent Registry (not troubleshooting an active error), jump to **Step 0b
(Design & Configuration Flow)** immediately.
- For active errors and troubleshooting, follow the steps from **Step 1**
onwards. Most 403s resolve at step 2 or 4. Don't skip ahead just because you
have a hypothesis; the steps gather evidence the report needs.
1. **Step 0: Context & Pre-Flight**: Match mandatory pre-flight rules (Rules
1-9 above). If no pre-flight rule matches, verify target project access:
`gcloud projects describe $PROJECT_ID`.
2. **Step 1: Agent Logs**: Confirm error type (403 vs connection vs crash).
- Connection Error -> Check PSC Subnet Exhaustion (Step 3c).
- Container Crash -> Perform Runtime Health Check (Step 1b).
3. **Step 2: Gateway Logs**: Find exact failing hostname.
4. **Step 3: IAP Logs**: Check DRY_RUN vs enforced mode and allow/deny
decision.
5. **Step 4: Registry State**: Verify if exact hostname is registered.
- Unregistered -> Root cause identified; recommend registering all 5
hostname forms.
6. **Step 5: Identity & IAM**: Verify agent identity has `roles/iap.egressor`
on the registered resource.
7. **Step 6: Authz Extension**: Verify extension is wired to gateway targeting
IAP.
8. **Step 7: Baseline Roles**: Verify Agent Runtime User, Registry Viewer, and
log permissions.
9. **Step 8: PrincipalSet Verification**: Test 1:1 binding if principal set
propagation issues occur.
The exact log queries, gcloud commands, and curl invocations live in
`references/field-manual.md` (which includes the full flowchart). Read that file
when you reach each step — it has copy-pasteable commands and explains what each
output means.
### Step 0b — Design & Configuration Flow
If the user asks for guidance on designing, configuring, or registering services
in the Agent Registry (especially Google APIs like Agent Runtime, Cloud Resource
Manager, etc.):
1. **Read Reference**: Immediately read `references/agent-registry.md`
Section 2.
2. **Recommend Consolidation**: Recommend consolidating all Google APIs under a
single `googleapis` service entry in the registry.
3. **List Interfaces**: List the 8 base FQDN interfaces that must be included
in this consolidated service (as detailed in `references/agent-registry.md`
Section 2).
4. **Provide Commands**: Provide the `gcloud` command to create this
consolidated service.
- **Step 0: Context & Pre-Flight**: Match mandatory pre-flight rules (Rules 1-13). Verify project access via `gcloud projects describe $PROJECT_ID`. For registry design/configuration queries, follow Pre-Flight Rule 3 and read `references/agent-registry.md` §2.
- **Step 1: Agent Logs**: Confirm error type (403 vs connection vs crash). For connection errors/timeouts, check PSC Subnet Exhaustion and ACT/Cloud NAT routing. For container crashes, perform runtime health check.
- **Step 2: Gateway Logs**: Find exact failing hostname.
- **Step 3: IAP Logs**: Check policy version (`iapPolicyVersion: "V2"` vs `"V1"`), DRY_RUN vs enforced mode, and decision.
- **Step 4: Registry State**: Verify if exact hostname is registered. If unregistered, recommend registering all hostname forms.
- **Step 5: Identity & Policies**: Verify agent identity has `roles/iap.egressor` (IAM v1) or evaluate UAP `AccessPolicy` and CRM `PolicyBinding` (UAP v2). Check `constraints/iam.managed.disableAccessPolicyBinding` blocker.
- **Step 6: Authz Extension & Gateway**: Verify extension is wired to gateway targeting IAP with correct policy version, and verify dual-registry constraints (max 2: 1 global + 1 regional/multi-regional).
- **Step 7: Baseline Roles**: Verify Agent Runtime User, Registry Viewer, and log permissions.
- **Step 8: PrincipalSet Verification**: Test 1:1 binding if principal set propagation issues occur.
## Tools to use
@@ -404,50 +454,38 @@ Always produce a structured report. Use this template exactly.
destination.
- **DRY_RUN changes everything.** If IAP is in dry-run, denials are logged but
not enforced.
- **The role is `roles/iap.egressor`.**
- **Always Recommend IAP Verification**: For any IAP-related issue, you MUST
explicitly suggest verifying:
1. The agent identity has `roles/iap.httpsGatewayUser` (for gateway-level
access) AND `roles/iap.egressor` (for endpoint-level access).
2. The `AuthorizationPolicy` is correctly bound to the Gateway (check if it
targets the gateway resource). Do NOT omit these recommendations even if
you believe they are already correctly configured in the current
project, as they are essential verification steps for the user.
- **Consolidated Registry for Google APIs**: To simplify management and avoid
hitting API/registry limits, always recommend consolidating all Google APIs
under a single 'googleapis' service entry in the registry with the 8 base
FQDN interfaces, rather than registering them as separate services. Refer to
`references/agent-registry.md` Section 2 for details.
- **The role is `roles/iap.egressor` (for IAM v1) and FQDN permission
`iap.googleapis.com/resources.egressViaIAP` (for UAP v2).**
- **UAP (Policy V2) CRM Hierarchy**: Next-gen policies bind to Projects,
Folders, and Organizations via `PolicyBinding` rather than shadow resources.
Evaluation follows absolute DENY precedence and additive ALLOW aggregation
across the CRM tree.
- **Agent Connectivity Template (ACT) with `ALL_TRAFFIC`**: Under VPC-SC,
traffic traverses synthetic PSC VIP `240.0.0.2:443`. External public API
traffic exiting the consumer VPC requires Cloud NAT on the PSC-I subnet to
avoid silent connection hangs.
- **Dual-Registry Invariant**: An Agent Gateway supports at most 2 registries;
when 2 are configured, exactly ONE must be `global` and the second must be
`regional` or `multi-regional`.
- **Read evidence, don't assume.** Pull logs first.
- **Cite exact resource names in the report.**
- **Stay in diagnosis mode.** Don't apply Terraform changes or run destructive
gcloud commands. Read-only inspection only.
- **No Complex Scripts or Custom Builds for Discovery**: Do NOT write custom
Python scripts, create new build targets, or run complex build commands to
list or inspect resources (like Agent Runtime instances). Doing so consumes
too many turns and causes timeouts. If a gcloud command is missing, use
`curl` to query the REST API directly using application-default credentials.
- **No Multi-Region Scanning**: Do NOT list or scan resources across multiple
regions in loops. Unless the user/logs explicitly point to a different
region, only check resources in the default region (`us-central1`). Running
regional loops will cause timeouts.
- **Avoid interactive commands and disable prompts.** Do NOT run commands that
require user interaction or launch pagers (like `gcloud help` or raw `man`
pages) as they can hang the execution. Always disable prompts for CLI tools
(e.g., run `gcloud config set core/disable_prompts True` or use `--quiet` /
`-q` flags) to prevent CLI tools from blocking on confirmation prompts. Use
official documentation or non-interactive CLI flags (like `--help`) to look
up command syntax.
regions in loops. Check default region (`us-central1`) unless logs point
elsewhere.
- **Non-interactive execution**: Always disable prompts (`--quiet` / `-q` or
`gcloud config set core/disable_prompts True`) to avoid hanging.
## Supporting Links
- [Agent Runtime Overview](https://docs.cloud.google.com/gemini-enterprise-agent-platform/agents)
- [Agent Gateway Overview](https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/gateways/agent-gateway-overview)
- [Policies Overview](https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/policies/overview)
- [Agent Identity Overview](https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/agent-identity-overview)
- [Agent Registry Overview](https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/agent-registry)
- [Deploy Agent Gateway Runtime](https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/runtime/agent-gateway-runtime-deploy)
- [Private Service Connect Interface](https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/runtime/private-service-connect-interface)
- [Troubleshoot Agent Gateway](https://docs.cloud.google.com/gemini-enterprise-agent-platform/troubleshooting/troubleshoot-agent-gateway)
- [Troubleshoot Agent Deployment](https://docs.cloud.google.com/gemini-enterprise-agent-platform/troubleshooting/agent-deployment)
- [Troubleshoot Runtime Setup](https://docs.cloud.google.com/gemini-enterprise-agent-platform/troubleshooting/runtime-setup)
- [Agent Runtime Overview](https://docs.cloud.google.com/gemini-enterprise-agent-platform/agents.md.txt)
- [Agent Gateway Overview](https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/gateways/agent-gateway-overview.md.txt)
- [Policies Overview](https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/policies/overview.md.txt)
- [Agent Identity Overview](https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/agent-identity-overview.md.txt)
- [Agent Registry Overview](https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/agent-registry.md.txt)
- [Deploy Agent Gateway Runtime](https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/runtime/agent-gateway-runtime-deploy.md.txt)
- [Private Service Connect Interface](https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/runtime/private-service-connect-interface.md.txt)
- [Troubleshoot Agent Gateway](https://docs.cloud.google.com/gemini-enterprise-agent-platform/troubleshooting/troubleshoot-agent-gateway.md.txt)
- [Troubleshoot Agent Deployment](https://docs.cloud.google.com/gemini-enterprise-agent-platform/troubleshooting/agent-deployment.md.txt)
- [Troubleshoot Runtime Setup](https://docs.cloud.google.com/gemini-enterprise-agent-platform/troubleshooting/runtime-setup.md.txt)
skills/cloud/agent-platform-troubleshooting/references/agent-gateway.md
+683 -667
View File
File diff suppressed because it is too large Load Diff
skills/cloud/agent-platform-troubleshooting/references/agent-identity.md
+54 -9
View File
@@ -140,13 +140,25 @@ Source:
Source: <https://docs.cloud.google.com/iam/docs/principal-identifiers>
Scope | Identifier
:---------------------------------- | :---------
Single agent | `principal://<TRUST_DOMAIN>/resources/<SERVICE>/<RESOURCE_PATH>`
All agents in a trust domain | `principalSet://<TRUST_DOMAIN>/*`
All agents in a project | `principalSet://<TRUST_DOMAIN>/attribute.platformContainer/aiplatform/projects/<PROJECT_NUMBER>`
All agents in an org | `principalSet://agents.global.org-<ORG_ID>.system.id.goog/attribute.platform/aiplatform`
DIY agents (workload identity pool) | `principal://iam.googleapis.com/projects/<PROJECT_NUMBER>/locations/global/workloadIdentityPools/<POOL>/subject/ns/<NS>/sa/<SA>`
| Scope | Identifier |
| :-------- | :------------------------------------------------------------------------------------------------------------------------------- |
| Single | `principal://<TRUST_DOMAIN>/resources/<SERVICE>/<RESOURCE_PATH>` |
: agent : :
| All | `principalSet://<TRUST_DOMAIN>/*` |
: agents in : :
: a trust : :
: domain : :
| All | `principalSet://<TRUST_DOMAIN>/attribute.platformContainer/aiplatform/projects/<PROJECT_NUMBER>` |
: agents in : :
: a project : :
| All | `principalSet://agents.global.org-<ORG_ID>.system.id.goog/attribute.platform/aiplatform` |
: agents in : :
: an org : :
| DIY | `principal://iam.googleapis.com/projects/<PROJECT_NUMBER>/locations/global/workloadIdentityPools/<POOL>/subject/ns/<NS>/sa/<SA>` |
: agents : :
: (workload : :
: identity : :
: pool) : :
Tips when debugging:
@@ -157,6 +169,39 @@ Tips when debugging:
for sensitive resources.
- The `attribute.platform/aiplatform` set covers an *entire org*.
### Principal Identifiers in Unified Access Policy (UAP / Policy V2)
Under **Unified Access Policy** (`iam.googleapis.com/v3` `AccessPolicy` and
`PolicyBinding`), principal identifiers are specified directly in the
`rules[].principals` array:
```yaml
# In AccessPolicy rule definition:
rules:
- description: "Allow project agents to access registered weather service"
principals:
- "principalSet://agents.global.org-123456789012.system.id.goog/attribute.platformContainer/aiplatform/projects/9876543210"
- "principal://agents.global.org-123456789012.system.id.goog/resources/aiplatform/projects/9876543210/locations/us-central1/reasoningEngines/weather-agent"
operations:
- permissions:
- "iap.googleapis.com/resources.egressViaIAP"
```
Important rules for UAP principal matching:
1. **Direct Principal vs. PrincipalSet**: You can specify either specific
`principal://...` URIs or broad `principalSet://...` group URIs in the
`principals` array.
2. **Audit Log Correlation**: In Cloud Audit Logs
(`protoPayload.authenticationInfo.principalSubject`), the runtime always
logs the exact `principal://` URI of the calling agent, which satisfies
either a direct `principal://` rule or a container-level `principalSet://`
rule.
3. **Service Accounts in UAP**: If an agent runs with a custom Service Account
(`SERVICE_ACCOUNT` mode), the principal string in UAP is
`serviceAccount:name@project.iam.gserviceaccount.com` (or
`principal://iam.googleapis.com/projects/...`).
--------------------------------------------------------------------------------
## How identities are created and bound to ReasoningEngines
@@ -480,8 +525,8 @@ principal as `members`. Per the policy assignment doc:
}
```
Apply with `gcloud beta iap web set-iam-policy ...
--resource-type=agent-registry` or `--endpoint=ENDPOINT_ID` depending on target.
Apply with `gcloud iap web set-iam-policy ... --resource-type=agent-registry` or
`--endpoint=ENDPOINT_ID` depending on target.
### Agent can't use an auth provider (auth manager)
skills/cloud/agent-platform-troubleshooting/references/agent-registry.md
+167 -76
View File
@@ -5,7 +5,7 @@ registered in Google Cloud Agent Registry are not accepting traffic through
Agent Gateway / IAP.
Scope: Gemini Enterprise Agent Platform — Agent Registry (Preview). All commands
use `gcloud alpha agent-registry`.
use `gcloud agent-registry`.
## Table of Contents
@@ -34,11 +34,18 @@ use `gcloud alpha agent-registry`.
Read these first when something is broken.
1. **Agent Registry is default-deny at the gateway.** Agent Gateway uses
Identity-Aware Proxy (IAP) to enforce IAM allow/deny policies bound to Agent
Registry resources. If a target hostname is not registered as a `Service`
(and a matching `roles/iap.egressor` binding does not exist for the calling
agent's principal), egress is blocked. Source:
`docs.cloud.google.com/iap/docs/agent-overview`.
Identity-Aware Proxy (IAP) to enforce authorization:
- **UAP (Policy V2)**: When the gateway uses `iapPolicyVersion: "V2"`,
Gatekeeper queries the attached registries. If the destination matches,
Gatekeeper sets `destination.is_registered = true` and exposes rich CEL
metadata under `destination.agent_registry.*`. If not cataloged,
`destination.is_registered = false`.
- **Legacy IAM v1**: When using legacy v1, if a target hostname is not
registered as a `Service` (and a matching `roles/iap.egressor` binding
does not exist on the registry entry for the calling agent's principal),
egress is blocked. Source:
`docs.cloud.google.com/iap/docs/agent-overview`.
2. **There are exactly three discoverable resource types**, and they are
*projections* of the same writable `Service`:
@@ -69,7 +76,7 @@ Read these first when something is broken.
6. **Traffic permissions live on IAP, bound to registry resources.** The role
that actually permits an agent to *use* a registered service is
`roles/iap.egressor` (`iap.webServiceVersions.egressViaIAP`), bound via
`gcloud beta iap web set-iam-policy` with one of:
`gcloud iap web set-iam-policy` with one of:
`--resource-type=agent-registry` (registry-wide), `--agent=`,
`--mcp-server=`, or `--endpoint=` (per-resource). Source:
`docs.cloud.google.com/gemini-enterprise-agent-platform/govern/policies/assign-identity-iam`,
@@ -96,7 +103,7 @@ Read these first when something is broken.
third-party MCP) is *manual*.
10. **First debug command** is almost always: list the resource type and confirm
the hostname permutation you expect is actually present. `gcloud alpha
the hostname permutation you expect is actually present. `gcloud
agent-registry endpoints list --project=$PROJECT_ID --location=$LOCATION`
11. **Boundary controls visibility.** Agents are only discoverable if they are
@@ -104,6 +111,18 @@ Read these first when something is broken.
of the Management Project. If an agent is missing from the list, check the
boundary configuration.
12. **Agent Gateway Dual-Registry Binding**: An Agent Gateway can link up to
**two** registries in its `registries` array, provided that **exactly one
registry is `global`**, and the second registry is either `regional` (e.g.
`us-central1`) or `multi-regional` (e.g. `us`, `eu`). Two regional
registries without a global are rejected by the control plane.
13. **REST API Path Extensions (`.json`)**: When registering REST APIs (like
Zendesk), note that SDK clients append `.json` (e.g.
`/api/v2/tickets.json`). Ensure your UAP CEL rules or registry endpoint
interfaces account for exact paths, `/` subpaths, `.json` extensions, and
`?` query strings.
--------------------------------------------------------------------------------
## 1. The Data Model in One Picture
@@ -234,7 +253,7 @@ scheme, or extra subdomain mismatch will cause denial.
#### Creating Consolidated Service via gcloud
```bash
gcloud alpha agent-registry services create googleapis \
gcloud agent-registry services create googleapis \
--project=$PROJECT_ID --location=$LOCATION \
--display-name="Google APIs" \
--endpoint-spec-type=no-spec \
@@ -251,7 +270,7 @@ gcloud alpha agent-registry services create googleapis \
### Quick "is the hostname registered?" check
```bash
gcloud alpha agent-registry services list \
gcloud agent-registry services list \
--project=$PROJECT_ID \
--location=$LOCATION \
--format="table(name.basename(), interfaces.url)" \
@@ -286,7 +305,7 @@ functional-type annotation. Everything else is manual.
### List
```bash
gcloud alpha agent-registry agents list \
gcloud agent-registry agents list \
--project=PROJECT_ID \
--location=REGION
```
@@ -294,7 +313,7 @@ gcloud alpha agent-registry agents list \
Filter:
```bash
gcloud alpha agent-registry agents list \
gcloud agent-registry agents list \
--project=PROJECT_ID \
--location=REGION \
--filter="displayName='DISPLAY_NAME'"
@@ -303,7 +322,7 @@ gcloud alpha agent-registry agents list \
### Describe
```bash
gcloud alpha agent-registry agents describe AGENT_NAME \
gcloud agent-registry agents describe AGENT_NAME \
--project=PROJECT_ID \
--location=REGION
```
@@ -317,7 +336,7 @@ IAM.
Display name / description:
```bash
gcloud alpha agent-registry services update AGENT_NAME \
gcloud agent-registry services update AGENT_NAME \
--project=PROJECT_ID \
--location=REGION \
--display-name="New name" \
@@ -327,7 +346,7 @@ gcloud alpha agent-registry services update AGENT_NAME \
Endpoint URL:
```bash
gcloud alpha agent-registry services update AGENT_NAME \
gcloud agent-registry services update AGENT_NAME \
--project=PROJECT_ID \
--location=REGION \
--interfaces=url=ENDPOINT_URL,protocolBinding=PROTOCOL
@@ -339,7 +358,7 @@ Agent spec payload (max size 10 KB to satisfy gRPC control plane metadata header
limits):
```bash
gcloud alpha agent-registry services update AGENT_NAME \
gcloud agent-registry services update AGENT_NAME \
--project=PROJECT_ID \
--location=REGION \
--agent-spec-content=@AGENT_SPEC
@@ -351,7 +370,7 @@ For automatically registered agents: delete from the source runtime -- registry
entry follows. For manually registered agents:
```bash
gcloud alpha agent-registry services delete AGENT_NAME \
gcloud agent-registry services delete AGENT_NAME \
--project=PROJECT_ID \
--location=REGION
```
@@ -368,7 +387,7 @@ gcloud alpha agent-registry services delete AGENT_NAME \
the *target* agent resource:
```bash
gcloud beta iap web set-iam-policy agents-iap-policy.json \
gcloud iap web set-iam-policy agents-iap-policy.json \
--project=PROJECT_ID \
--agent=AGENT_ID \
--region=REGION
@@ -422,7 +441,7 @@ manual registration with a tool spec.
### Register
```bash
gcloud alpha agent-registry services create SERVER_NAME \
gcloud agent-registry services create SERVER_NAME \
--project=PROJECT_ID \
--location=REGION \
--display-name="DISPLAY_NAME" \
@@ -438,7 +457,7 @@ gcloud alpha agent-registry services create SERVER_NAME \
### List
```bash
gcloud alpha agent-registry mcp-servers list \
gcloud agent-registry mcp-servers list \
--project=PROJECT_ID \
--location=REGION
```
@@ -449,7 +468,7 @@ Useful filters: - `displayName='NAME'` - `mcpServerId='urn:mcp:SERVER_URN'`
### Describe
```bash
gcloud alpha agent-registry mcp-servers describe SERVER_NAME \
gcloud agent-registry mcp-servers describe SERVER_NAME \
--project=PROJECT_ID \
--location=REGION
```
@@ -463,7 +482,7 @@ Tool spec changes are not auto-detected -- re-upload when the server
adds/changes tools:
```bash
gcloud alpha agent-registry services update SERVER_NAME \
gcloud agent-registry services update SERVER_NAME \
--project=PROJECT_ID \
--location=REGION \
--mcp-server-spec-content=@TOOL_SPEC
@@ -472,7 +491,7 @@ gcloud alpha agent-registry services update SERVER_NAME \
### Delete
```bash
gcloud alpha agent-registry services delete SERVER_NAME \
gcloud agent-registry services delete SERVER_NAME \
--project=PROJECT_ID \
--location=REGION
```
@@ -488,7 +507,7 @@ gcloud alpha agent-registry services delete SERVER_NAME \
resource:
```bash
gcloud beta iap web set-iam-policy agents-iap-policy.json \
gcloud iap web set-iam-policy agents-iap-policy.json \
--project=PROJECT_ID \
--mcp-server=MCP_SERVER_ID \
--region=REGION
@@ -544,7 +563,7 @@ REST APIs, internal HTTP services. Always manually registered.
### Register
```bash
gcloud alpha agent-registry services create SERVICE_NAME \
gcloud agent-registry services create SERVICE_NAME \
--project=PROJECT_ID \
--location=REGION \
--display-name="DISPLAY_NAME" \
@@ -565,7 +584,7 @@ gcloud alpha agent-registry services create SERVICE_NAME \
### List
```bash
gcloud alpha agent-registry endpoints list \
gcloud agent-registry endpoints list \
--project=PROJECT_ID \
--location=REGION
```
@@ -573,7 +592,7 @@ gcloud alpha agent-registry endpoints list \
Filter:
```bash
gcloud alpha agent-registry endpoints list \
gcloud agent-registry endpoints list \
--project=PROJECT_ID \
--location=REGION \
--filter="displayName='NAME'"
@@ -583,7 +602,7 @@ For URL-substring search, list `services` instead and filter on
`interfaces.url`:
```bash
gcloud alpha agent-registry services list \
gcloud agent-registry services list \
--project=PROJECT_ID \
--location=REGION \
--format="table(name.basename(), interfaces.url, interfaces.protocolBinding)" \
@@ -593,7 +612,7 @@ gcloud alpha agent-registry services list \
### Describe
```bash
gcloud alpha agent-registry endpoints describe ENDPOINT_NAME \
gcloud agent-registry endpoints describe ENDPOINT_NAME \
--project=PROJECT_ID \
--location=REGION
```
@@ -601,7 +620,7 @@ gcloud alpha agent-registry endpoints describe ENDPOINT_NAME \
### Update
```bash
gcloud alpha agent-registry services update SERVICE_NAME \
gcloud agent-registry services update SERVICE_NAME \
--project=PROJECT_ID \
--location=REGION \
--interfaces=url=ENDPOINT_URL,protocolBinding=PROTOCOL
@@ -610,7 +629,7 @@ gcloud alpha agent-registry services update SERVICE_NAME \
### Delete
```bash
gcloud alpha agent-registry services delete SERVICE_NAME \
gcloud agent-registry services delete SERVICE_NAME \
--project=PROJECT_ID \
--location=REGION
```
@@ -626,7 +645,7 @@ gcloud alpha agent-registry services delete SERVICE_NAME \
agent's principal on the endpoint resource:
```bash
gcloud beta iap web set-iam-policy agents-iap-policy.json \
gcloud iap web set-iam-policy agents-iap-policy.json \
--project=PROJECT_ID \
--endpoint=ENDPOINT_ID \
--region=REGION
@@ -659,8 +678,8 @@ role -- confirm against `docs.cloud.google.com/iam/docs/roles-permissions/iap`.
- **Endpoint registered in `global` but agent calls a regional URL**: registry
location doesn't have to match the destination URL location, but the IAM
binding's `--region` must match where the resource is registered. Check that
`--region=` on `gcloud beta iap web set-iam-policy` equals the location used
at creation.
`--region=` on `gcloud iap web set-iam-policy` equals the location used at
creation.
- **Endpoint not appearing in `endpoints list`**: created with the wrong spec
type. Must use `--endpoint-spec-type=no-spec`.
@@ -690,11 +709,47 @@ Project-scoped, granted via standard IAM:
These let humans and CI register and inspect entries. They do **not** grant
runtime traffic.
### Layer B — IAP egress permissions (let an agent actually send traffic)
### Layer B — Traffic Authorization (Legacy v1 vs. UAP Policy V2)
Bound on Agent Registry resources via `gcloud beta iap web set-iam-policy`. The
role is always `roles/iap.egressor`. The scope of the binding determines which
target resources the source agent can reach:
Depending on the gateway's `iapPolicyVersion` setting, registry resources
authorize traffic through one of two mechanisms:
#### Mechanism B1: Unified Access Policy (UAP / Policy V2)
When the attached IAP Authz Extension specifies `iapPolicyVersion: "V2"`,
policies are decoupled from individual registry resources:
- `AccessPolicy` and `PolicyBinding` (`iam.googleapis.com/v3`) attach to the
Resource Manager hierarchy (Organization, Folder, Project) targeting numeric
resource URIs (e.g.
`//cloudresourcemanager.googleapis.com/projects/${PROJECT_NUMBER}`).
- Rules grant FQDN permission `iap.googleapis.com/resources.egressViaIAP`.
- Agent Registry entries dynamically populate the
`destination.agent_registry.*` CEL attributes evaluated at runtime:
- `destination.is_registered` (boolean): `true` if target matches a
registry service.
- `destination.agent_registry.resource_type`: `'ENDPOINT'`,
`'MCP_SERVER'`, `'AGENT'`, `'SKILL'`.
- `destination.agent_registry.location`: region of registry resource.
- `destination.agent_registry.mcp_server.name`: full MCP server resource
name.
- `destination.agent_registry.mcp_server.method`: `'tools'`, `'prompts'`,
`'resources'`.
- `destination.agent_registry.mcp_server.tool.name`: specific tool name
(e.g. `'updateTicket'`).
- `destination.agent_registry.mcp_server.tool.annotations.read_only_hint`:
boolean.
- **Mandatory CEL Guard**: Always guard tool annotations with:
`(destination.agent_registry.mcp_server.method == 'tools') &&
(destination.agent_registry.mcp_server.tool.annotations.read_only_hint ==
true)`.
#### Mechanism B2: Legacy IAM v1 (Per-Resource Bindings)
In legacy v1, permissions are bound directly on Agent Registry resources via
`gcloud iap web set-iam-policy`. The role is always `roles/iap.egressor`. The
scope of the binding determines which target resources the source agent can
reach:
| Scope | gcloud target flag | Grants egress to |
| ----------------- | -------------------------------- | --------------------- |
@@ -744,17 +799,28 @@ Source:
When debugging "wrong principal" errors: the principal is built from the agent's
**resource URI**, not its display name and not its URN. View the resource URI
via `gcloud alpha agent-registry agents describe`.
via `gcloud agent-registry agents describe`.
### Full IAP role reference (relevant subset)
Role | Permission grants the role provides | When to use
---------------------------------- | ----------------------------------------------------------------------- | -----------
`roles/iap.egressor` (Beta) | `iap.webServiceVersions.egressViaIAP` | **Agent egress through Agent Gateway**
`roles/iap.httpsResourceAccessor` | `iap.webServiceVersions.accessViaIAP` | Human/service ingress to an IAP-protected web app
`roles/iap.tunnelResourceAccessor` | `iap.tunnelDestGroups.accessViaIAP`, `iap.tunnelInstances.accessViaIAP` | Classic IAP TCP tunnels to GCE -- **not** for agent gateway
`roles/iap.admin` | Full IAP admin (`iap.tunnel.*`, all getIam/setIam) | Manage IAP policies
`roles/iap.viewer` | Read IAP settings | Audit
| Role | Permission grants the role provides | When to use |
| ---------------------------------- | ------------------------------------- | ------------- |
| `roles/iap.egressor` (Beta) | `iap.webServiceVersions.egressViaIAP` | **Agent |
: : : egress :
: : : through Agent :
: : : Gateway** :
| `roles/iap.httpsResourceAccessor` | `iap.webServiceVersions.accessViaIAP` | Human/service |
: : : ingress to an :
: : : IAP-protected :
: : : web app :
| `roles/iap.tunnelResourceAccessor` | `iap.tunnelDestGroups.accessViaIAP`, | Classic IAP |
: : `iap.tunnelInstances.accessViaIAP` : TCP tunnels :
: : : to GCE -- :
: : : **not** for :
: : : agent gateway :
| `roles/iap.admin` | Full IAP admin (`iap.tunnel.*`, all | Manage IAP |
: : getIam/setIam) : policies :
| `roles/iap.viewer` | Read IAP settings | Audit |
Source: `docs.cloud.google.com/iam/docs/roles-permissions/iap`.
@@ -762,21 +828,46 @@ Source: `docs.cloud.google.com/iam/docs/roles-permissions/iap`.
## 7. Common Registration Errors and Where to Look
Symptom | Likely cause | Where to check
--------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | --------------
`services create` fails: "manual registration not supported" | Used `--location=us` or `--location=eu` (multi-regions) | Switch to a region (e.g. `us-central1`) or `global`
`services create` succeeds but resource missing from `endpoints list` | Wrong `*-spec-type`. `--endpoint-spec-type=no-spec` produces an `Endpoint`. | Re-create with the correct spec-type, or check `services list`
`services update --mcp-server-spec-content` rejects file | Spec > 10 KB | Trim the tool spec
Agent calls API X, IAP returns 403 | Hostname variant not registered (commonly REP `*.LOCATION.rep.googleapis.com`) | `services list --filter="interfaces.url:FRAGMENT"`
Agent calls API X, IAP returns 403 even though hostname is registered | Missing `roles/iap.egressor` binding for the source agent's principal | `gcloud beta iap web get-iam-policy --project= --region= --endpoint=...` (or `--mcp-server=`, `--agent=`, `--resource-type=agent-registry`)
MCP tool call succeeds for one tool, denied for another | IAP policy condition restricts `mcp.toolName` or `mcp.tool.isReadOnly` | Inspect the IAP policy on the MCP server resource
Auto-registered Agent Runtime agent missing from registry | Agent deployed via custom code, not Agent Runtime SDK | Re-deploy via Agent Runtime, or register manually as a `Service`
A2A agent has no skills in registry | `agent-card.json` unreachable from Google's crawler | Curl the Agent Card URL from a public network; verify content type and schema
Agent principal mismatch in IAP policy | Principal built from wrong resource URI / project number / SA identifier | `gcloud alpha agent-registry agents describe` -> use Resource URI to construct
`roles/iap.tunnelResourceAccessor` granted, still denied | Wrong role -- agent egress requires `roles/iap.egressor` | Replace the binding
Endpoint Google Cloud console "Test connection" fails on private URL | Connection test only works for public URLs (by design) | Test via the agent itself or `curl` from a peered network
`agents-iap-policy.json` set, still denied | Wrong `--region` flag on `set-iam-policy` -- must match resource location | Re-issue with the correct `--region`
Cannot find URN for binding | Use `mcp-servers describe` / `agents describe` and read the `name` field; format per section 1 | See URN formats in section 1
| Symptom | Likely cause | Where to check |
| ---------------------------------- | -------------------------------- | ------------------------------------ |
| `services create` fails: "manual | Used `--location=us` or | Switch to a region (e.g. |
: registration not supported" : `--location=eu` (multi-regions) : `us-central1`) or `global` :
| `services create` succeeds but | Wrong `*-spec-type`. | Re-create with the correct |
: resource missing from `endpoints : `--endpoint-spec-type=no-spec` : spec-type, or check `services list` :
: list` : produces an `Endpoint`. : :
| `services update | Spec > 10 KB | Trim the tool spec |
: --mcp-server-spec-content` rejects : : :
: file : : :
| Agent calls API X, IAP returns 403 | Hostname variant not registered | `services list |
: : (commonly REP : --filter="interfaces.url\:FRAGMENT"` :
: : `*.LOCATION.rep.googleapis.com`) : :
| Agent calls API X, IAP returns 403 | Missing `roles/iap.egressor` | `gcloud iap web get-iam-policy |
: even though hostname is registered : binding for the source agent's : --project= --region= --endpoint=...` :
: : principal : (or `--mcp-server=`, `--agent=`, :
: : : `--resource-type=agent-registry`) :
| MCP tool call succeeds for one | IAP policy condition restricts | Inspect the IAP policy on the MCP |
: tool, denied for another : `mcp.toolName` or : server resource :
: : `mcp.tool.isReadOnly` : :
| Auto-registered Agent Runtime | Agent deployed via custom code, | Re-deploy via Agent Runtime, or |
: agent missing from registry : not Agent Runtime SDK : register manually as a `Service` :
| A2A agent has no skills in | `agent-card.json` unreachable | Curl the Agent Card URL from a |
: registry : from Google's crawler : public network; verify content type :
: : : and schema :
| Agent principal mismatch in IAP | Principal built from wrong | `gcloud agent-registry agents |
: policy : resource URI / project number / : describe` -> use Resource URI to :
: : SA identifier : construct :
| `roles/iap.tunnelResourceAccessor` | Wrong role -- agent egress | Replace the binding |
: granted, still denied : requires `roles/iap.egressor` : :
| Endpoint Google Cloud console | Connection test only works for | Test via the agent itself or `curl` |
: "Test connection" fails on private : public URLs (by design) : from a peered network :
: URL : : :
| `agents-iap-policy.json` set, | Wrong `--region` flag on | Re-issue with the correct `--region` |
: still denied : `set-iam-policy` -- must match : :
: : resource location : :
| Cannot find URN for binding | Use `mcp-servers describe` / | See URN formats in section 1 |
: : `agents describe` and read the : :
: : `name` field; format per section : :
: : 1 : :
--------------------------------------------------------------------------------
@@ -813,50 +904,50 @@ Source: `docs.cloud.google.com/agent-registry/use-agentregistry-mcp`.
```bash
# READ
gcloud alpha agent-registry agents list --project=P --location=L
gcloud alpha agent-registry agents describe NAME --project=P --location=L
gcloud alpha agent-registry mcp-servers list --project=P --location=L
gcloud alpha agent-registry mcp-servers describe NAME --project=P --location=L
gcloud alpha agent-registry endpoints list --project=P --location=L
gcloud alpha agent-registry endpoints describe NAME --project=P --location=L
gcloud alpha agent-registry services list --project=P --location=L
gcloud alpha agent-registry services describe NAME --project=P --location=L
gcloud agent-registry agents list --project=P --location=L
gcloud agent-registry agents describe NAME --project=P --location=L
gcloud agent-registry mcp-servers list --project=P --location=L
gcloud agent-registry mcp-servers describe NAME --project=P --location=L
gcloud agent-registry endpoints list --project=P --location=L
gcloud agent-registry endpoints describe NAME --project=P --location=L
gcloud agent-registry services list --project=P --location=L
gcloud agent-registry services describe NAME --project=P --location=L
# WRITE (always 'services'; the projection follows from --*-spec-type)
gcloud alpha agent-registry services create NAME \
gcloud agent-registry services create NAME \
--project=P --location=L --display-name="..." \
--endpoint-spec-type=no-spec \
--interfaces=url=URL,protocolBinding=HTTP_JSON|GRPC|JSONRPC
gcloud alpha agent-registry services create NAME \
gcloud agent-registry services create NAME \
--project=P --location=L --display-name="..." \
--mcp-server-spec-type=tool-spec \
--mcp-server-spec-content=@toolspec.json \
--interfaces=url=URL,protocolBinding=JSONRPC
gcloud alpha agent-registry services create NAME \
gcloud agent-registry services create NAME \
--project=P --location=L --display-name="..." \
--agent-spec-content=@agent-card.json \
--interfaces=url=URL,protocolBinding=HTTP_JSON
gcloud alpha agent-registry services update NAME \
gcloud agent-registry services update NAME \
--project=P --location=L \
[ --display-name="..." | --description="..." \
| --interfaces=url=URL,protocolBinding=... \
| --mcp-server-spec-content=@spec.json \
| --agent-spec-content=@card.json ]
gcloud alpha agent-registry services delete NAME --project=P --location=L
gcloud agent-registry services delete NAME --project=P --location=L
# IAP egress policy bindings (separate API, separate gcloud surface)
gcloud beta iap web set-iam-policy POLICY.json \
gcloud iap web set-iam-policy POLICY.json \
--project=P --region=L \
[ --resource-type=agent-registry
| --agent=AGENT_ID
| --mcp-server=MCP_SERVER_ID
| --endpoint=ENDPOINT_ID ]
gcloud beta iap web get-iam-policy \
gcloud iap web get-iam-policy \
--project=P --region=L \
[ same target flags ]
```
skills/cloud/agent-platform-troubleshooting/references/field-manual.md
+381 -81
View File
@@ -35,11 +35,23 @@ permits it:
resource, otherwise the authz extension won't actually run.
3. **Service-extension (delegated authz)** — the gateway calls IAP to make an
allow/deny decision.
4. **IAP / IAM** — the agent's identity must have the **IAP egressor role**
(`roles/iap.egressor`, display name "IAP-secured Egressor") on the
registered resource, or be in a principal set that does.
5. **Principal Access Boundary (PAB)** — even with the IAM binding correct, a
PAB policy on the principal set can restrict which resources it can reach.
4. **IAP / IAM Policy Evaluation** — the agent's identity must be authorized by
IAP:
* **IAM v1 (Legacy Policy Model)**: Evaluated when `iapPolicyVersion:
"V1"` (or unset). The agent identity must have the **IAP egressor role**
(`roles/iap.egressor`, display name "IAP-secured Egressor") directly
bound on the registered resource shadow resource in IAP.
* **Unified Access Policy (UAP / Policy V2 / IAM v3)**: Evaluated when the
gateway's IAP Authz Extension metadata specifies `iapPolicyVersion:
"V2"`. Next-gen access control bound directly to Resource Manager
hierarchy nodes (Organizations, Folders, Projects) via `PolicyBinding`
(`iam.googleapis.com/v3`). Evaluates `AccessPolicy` rules with CEL
expressions on deep attributes (`destination.agent_registry.*` or
`destination.unregistered.*`). Follows strict evaluation invariants:
absolute DENY precedence (cannot be overridden by downstream projects),
and additive ALLOW aggregation across the CRM tree.
5. **Principal Access Boundary (PAB)** — even with the IAM/UAP binding correct,
a PAB policy on the principal set can restrict which resources it can reach.
**PAB takes precedence over IAM Allow** — a correct egressor binding does
nothing if a PAB scopes the principal away from the target.
@@ -50,14 +62,28 @@ authz policies. Denials *can* originate at this proxy layer before IAP runs (no
IAP audit entry exists for those calls); those show up in load-balancer logs
(see Step 3b).
**Proxy Routing & TLS SNI:** The proxy performs TLS inspection and looks inside
the HTTPS call for the SNI hostname. In Private Service Connect (PSC)
environments, the client resolves APIs to internal VIPs, so the outer tunnel
request is naturally logged as `CONNECT 240.0.0.2:443`. This is expected. The
proxy intercepts this and evaluates routing based on the inner SNI hostname. If
the corresponding hostname (e.g., `us-central1-aiplatform.googleapis.com`) is
NOT registered, the proxy cannot route the traffic, resulting in a
`default_denied` action on the `CONNECT` request before IAP is ever reached.
**Proxy Routing, TLS SNI & Agent Connectivity Templates (ACT):** The proxy
performs TLS inspection and looks inside the HTTPS call for the SNI hostname. In
Private Service Connect (PSC) environments, the client resolves APIs to internal
VIPs, so the outer tunnel request is naturally logged as `CONNECT
240.0.0.2:443`. This is expected. The proxy intercepts this and evaluates
routing based on the inner SNI hostname. If the corresponding hostname (e.g.,
`us-central1-aiplatform.googleapis.com`) is NOT registered, the proxy cannot
route the traffic, resulting in a `default_denied` action on the `CONNECT`
request before IAP is ever reached.
Under **VPC Service Controls (VPC-SC)**, Agent Gateway requires an **Agent
Connectivity Template (ACT)** configured with `vpcEgress: ALL_TRAFFIC`:
- Traffic destined for external public APIs (e.g. `api.ipify.org`,
`api.weather.gov`) resolves to the synthetic PSC VIP `240.0.0.2:443`,
traverses the gateway, and egresses through the consumer VPC PSC-I Network
Attachment subnet via **Cloud NAT** (`gateway-nat-gateway`) using static
IPs. If Cloud NAT is missing or misconfigured, external traffic hangs or
resets silently.
- Traffic destined for Google APIs (e.g., Vertex AI, Cloud Run default domains
`*.run.app`) routes directly through Andromeda/Private Google Access to
Google Front Ends (GFE), bypassing Cloud NAT.
### Don't confuse `roles/iap.egressor` with other IAP roles
@@ -216,7 +242,8 @@ xychart-beta
no │ yes │
│ ▼
│ ┌────────────────────────────────┐
│ │ 3c. PSC Subnet Exhaustion Check│
│ │ 3c. PSC Subnet Exhaustion & │
│ │ 3d. ACT ALL_TRAFFIC / Cloud NAT│
│ └────────────────────────────────┘
▼
┌─────────────────────────────────────┐
@@ -238,8 +265,8 @@ xychart-beta
│
▼
┌─────────────────────────────────────┐
│ 3. Pull IAP logs — DRY_RUN or │
│ enforced? Allow or deny? │
│ 3. Pull IAP logs — Check V1 vs V2, │
│ DRY_RUN vs enforced, allow/deny? │
└────────────────┬────────────────────┘
│
▼
@@ -251,17 +278,17 @@ xychart-beta
no │ yes │
▼ ▼
┌──────────────────┐ ┌─────────────────────────────┐
│ Root cause: │ │ 5. Does the agent identity │
│ unregistered │ │ (or its principal set) │
│ hostname permu- │ │ have IAP egressor on │
│ term. Recommend │ │ the registered resource? │
│ Root cause: │ │ 5. Check IAM / UAP Policies:│
│ unregistered │ │ - V1: roles/iap.egressor │
│ hostname permu- │ │ - V2: AccessPolicy & CRM │
│ term. Recommend │ │ PolicyBinding (CEL) │
│ registering all │ └────────┬────────────────────┘
│ five forms. │ │
└──────────────────┘ ▼
┌─────────────────────────────┐
│ 6. Authz extension wired to │
│ the gateway? Pointing at │
│ IAP? │
│ 6. Authz extension wired? │
│ iapPolicyVersion V1/V2? │
│ Dual-registry valid? │
└────────┬────────────────────┘
▼
┌─────────────────────────────┐
@@ -310,9 +337,9 @@ commands from hanging:
> ```bash
> ACTIVE_PROJ=$(gcloud config get-value project)
> # Check for gateways
> gcloud alpha network-services agent-gateways list --location=us-central1 --project=$ACTIVE_PROJ
> gcloud network-services agent-gateways list --location=us-central1 --project=$ACTIVE_PROJ
> # Check for agents
> gcloud alpha agent-registry agents list --location=us-central1 --project=$ACTIVE_PROJ
> gcloud agent-registry agents list --location=us-central1 --project=$ACTIVE_PROJ
> ```
>
> If you find the target resources in the active project, **proceed
@@ -320,12 +347,12 @@ commands from hanging:
If resources are not in the active project, check prioritized dev projects:
`bash for proj in duncanjames-tf-dev duncanjames-agw-tf; do # Check gateways
res_gw=$(gcloud alpha network-services agent-gateways list
--location=us-central1 --project=$proj 2>&1) if [[ "$res_gw" == *"NAME"* ]];
then echo "FOUND Gateways in project: $proj" echo "$res_gw" fi # Check
agents res_ag=$(gcloud alpha agent-registry agents list
--location=us-central1 --project=$proj 2>&1) if [[ "$res_ag" == *"NAME"* ]];
then echo "FOUND Registry in project: $proj" echo "$res_ag" fi done`
res_gw=$(gcloud network-services agent-gateways list --location=us-central1
--project=$proj 2>&1) if [[ "$res_gw" == *"NAME"* ]]; then echo "FOUND
Gateways in project: $proj" echo "$res_gw" fi # Check agents res_ag=$(gcloud
alpha agent-registry agents list --location=us-central1 --project=$proj
2>&1) if [[ "$res_ag" == *"NAME"* ]]; then echo "FOUND Registry in project:
$proj" echo "$res_ag" fi done`
If you cannot find the target resource (e.g. the stuck gateway or the
failing agent) in the default project:
@@ -367,7 +394,7 @@ startup/`set_up()`):
Registry. **You MUST also explicitly suggest checking whether other
required monitoring/tracing endpoints (such as
`monitoring.googleapis.com` or `trace.mtls.googleapis.com`) are
registered as Endpoints**, and provide the `gcloud alpha agent-registry
registered as Endpoints**, and provide the `gcloud agent-registry
endpoints create` commands for them.
* **Verify/Create Authorization Policy**: Explain that an
`AuthorizationPolicy` must allow the agent's principal set to access the
@@ -395,14 +422,19 @@ If the logs you find do **not** match the symptom described in the user's prompt
only issue.
2. In your diagnostic report, you MUST address the user's described symptom.
3. Explain that while you found a different error in the active logs (and
diagnose it), the user's described symptom (IAP denial) is typically caused
by:
* Missing `roles/iap.egressor` or `roles/iap.httpsGatewayUser` on the
agent identity.
* `AuthorizationPolicy` not being correctly bound to the Gateway.
4. Include the general troubleshooting steps for IAP denials (checking IAP
audit logs, verifying roles, checking policy bindings) as recommended fixes,
even if you couldn't verify them in the current log state.
diagnose it), for any logs or findings where the agent is getting a 403 to
an MCP server or endpoint, you must **check the IAP Egressor bindings on the
matching resource in the registry first**:
* If there is no registry entry for the MCP server or endpoint, ensure
there is one created.
* Ensure that the agent has the `roles/iap.egressor` role bound to the
registry entry. If there are no permissions matching, tell the user to
grant `roles/iap.egressor` against the registry entry.
* Verify that an `AuthorizationPolicy` is correctly bound to the Gateway.
4. Include these prioritized troubleshooting steps for IAP denials (checking
registry entries and `roles/iap.egressor` bindings first, checking IAP audit
logs, and verifying Gateway policies) as recommended fixes, even if you
couldn't verify them in the current log state.
### Step 1 — Confirm the symptom in agent logs
@@ -542,21 +574,33 @@ What to read out of each entry:
- **`labels."iap.googleapis.com/audited_resource_name"`** — if this is
`unregisteredResource`, the destination hostname isn't in the registry. Go
to Step 4.
- **The enforcement mode** — is IAP in dry-run?
- **The policy version & enforcement mode** — check `iapPolicyVersion` and
`iamEnforcementMode`:
```yaml
service: iap.googleapis.com
failOpen: true
timeout: 1s
metadata:
iamEnforcementMode: "DRY_RUN"
iapPolicyVersion: "V1"
iamEnforcementMode: "DRY_RUN" # or "ENFORCED"
iapPolicyVersion: "v2" # "v2" for UAP Policy V2; "V1" for legacy IAM v1
```
In `DRY_RUN` mode, denials are logged but the request proceeds. If your
agent is failing with a real 403 *and* IAP is in dry-run, the denial is
coming from somewhere else — most often the gateway's underlying egress
proxy (see Step 3b) or the destination service itself.
* **In `DRY_RUN` mode**, denials are logged but the request proceeds. If
your agent is failing with a real 403 *and* IAP is in dry-run, the
denial is coming from somewhere else — most often the gateway's
underlying egress proxy (see Step 3b) or the destination service itself.
* **Under UAP (`iapPolicyVersion: "v2"`)**, check
`protoPayload.metadata.policyEvaluationResults[]` and audit log entries.
UAP logs will display the evaluation outcomes across the Resource
Manager hierarchy (Organization, Folder, Project).
* If an **explicit DENY** matched anywhere in the hierarchy (e.g., at
Org or Folder level), the request is rejected immediately, even if a
project-level policy allows it.
* If **no ALLOW matched**, the request is denied by default-deny.
* Check for CEL attribute mismatches (e.g., calling an unregistered
host when policies only allow `destination.agent_registry.*`, or
path mismatches like missing `.json` extensions in REST API paths).
### Step 3b — If there's no IAP audit entry for the failing call, pull the gateway proxy load-balancer log
@@ -607,14 +651,14 @@ Connect (PSC) subnet might be out of IP addresses.
project to find the one that is stuck or relevant:
```bash
gcloud alpha network-services agent-gateways list --location=$LOCATION --project=$PROJECT_ID
gcloud network-services agent-gateways list --location=$LOCATION --project=$PROJECT_ID
```
2. **Identify the Network Attachment**: Describe the Agent Gateway to find the
Network Attachment in use:
```bash
gcloud alpha network-services agent-gateways describe AGENT_GATEWAY_NAME --location=$LOCATION --project=$PROJECT_ID
gcloud network-services agent-gateways describe AGENT_GATEWAY_NAME --location=$LOCATION --project=$PROJECT_ID
```
Look for `networkConfig.egress.networkAttachment` or
@@ -654,38 +698,146 @@ Connect (PSC) subnet might be out of IP addresses.
a `/28` subnet is too small and easily exhausted, and recommend
expanding the CIDR range to at least `/26` as a best practice.
### Step 3d — Diagnosing Agent Connectivity Template (ACT) & Network Routing
Under VPC Service Controls (VPC-SC), Agent Gateway relies on an **Agent
Connectivity Template (ACT)**. If the agent experiences timeouts, silent
connection hangs, or connection drops when attempting to reach public internet
endpoints or internal services, check the ACT and consumer network
configuration:
1. **Inspect the Agent Connectivity Template**:
```bash
gcloud network-services agent-connectivity-templates describe ACT_NAME \
--location=$LOCATION --project=$PROJECT_ID
```
Verify the `vpcEgress` mode:
- `vpcEgress: ALL_TRAFFIC`: Intercepts and routes all traffic (Google APIs
and public Internet) into the consumer VPC network attachment.
- `vpcEgress: DEFAULT`: Only Private Google Access traffic routes through
the template.
2. **Verify Consumer Cloud NAT for External Endpoints (BKI 28)**: When
`vpcEgress: ALL_TRAFFIC` is enabled, external internet traffic (e.g.,
`api.ipify.org`, `api.weather.gov`, third-party SaaS) resolves via synthetic
PSC VIP `240.0.0.2:443`, traverses the gateway, and exits through the
consumer VPC PSC-I subnet.
- **Cloud NAT Requirement**: The consumer VPC **MUST** have a Cloud NAT
gateway configured that covers the PSC-I subnet range (e.g.,
`gateway-nat-gateway` with static IP).
- If Cloud NAT is missing or does not cover the PSC-I subnet, TCP
connections to external internet endpoints will **hang silently or
reset** without any proxy error log!
- Verify NAT status:
```bash
gcloud compute routers nats list --router=ROUTER_NAME --region=$LOCATION --project=$PROJECT_ID
```
3. **Verify Cloud DNS Peering**: Ensure the consumer VPC has Cloud DNS peering
configured so that Google API hostnames and registered domains correctly
resolve to the synthetic PSC VIP (`240.0.0.2:443`).
4. **Dangling ACT References on Teardown (BKI 30)**: If deleting an Agent
Connectivity Template fails with `FAILED_PRECONDITION: Resource ... is
already being used by resource(s) .../agentGateways/...` after the gateway
was already deleted, Network Services retained a dangling reference
(`FROM_AGENT_GATEWAY<ID>`). Clear it using Stubby:
```bash
stubby call blade:network-services-prod-${REGION} google.internal.cloud.reference.References.DeleteReference \
"{name: 'projects/${PROJECT_NUMBER}/locations/${REGION}/references/FROM_AGENT_GATEWAY${GW_ID}'}"
```
### Step 4 — Verify the hostname is registered (in the form the agent used)
List registry entries — pick the right resource type for the destination:
```bash
gcloud alpha agent-registry endpoints list --project=$PROJECT_ID --location=$LOCATION
gcloud alpha agent-registry mcp-servers list --project=$PROJECT_ID --location=$LOCATION
gcloud alpha agent-registry agents list --project=$PROJECT_ID --location=$LOCATION
gcloud agent-registry endpoints list --project=$PROJECT_ID --location=$LOCATION
gcloud agent-registry mcp-servers list --project=$PROJECT_ID --location=$LOCATION
gcloud agent-registry agents list --project=$PROJECT_ID --location=$LOCATION
```
Grep the output for the exact hostname from step 2. If it's missing, that's the
root cause.
### Step 5 — Verify IAM bindings on the registered resource and gateway
### Step 5 — Verify IAM / UAP bindings on the registered resource
To authorize egress, the agent's identity needs roles granted on both the
gateway and the registry resources:
Depending on whether your gateway uses **IAM v1** or **Unified Access Policy
(UAP / Policy V2)** (determined by `iapPolicyVersion` in the Authz Extension),
verify the corresponding authorization bindings:
1. **Gateway Access**: The agent needs `roles/iap.httpsGatewayUser` to connect
to the gateway. Verify or grant this role (often at the project level):
#### Option A: Unified Access Policy (UAP / Policy V2 / IAM v3)
In UAP, access policies are bound directly to the **Google Cloud Resource
Manager (CRM) hierarchy** (Project, Folder, or Organization) rather than IAP
shadow resources.
1. **List Policy Bindings on the Target Project**:
```bash
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member="principalSet://agents.global.org-${ORG_ID}.system.id.goog/attribute.platformContainer/aiplatform/projects/${PROJECT_NUMBER}" \
--role="roles/iap.httpsGatewayUser"
gcloud iam policy-bindings list \
--target="//cloudresourcemanager.googleapis.com/projects/${PROJECT_ID}" \
--location=global
```
2. **Registry/Destination Access**: The agent needs `roles/iap.egressor` on the
registered resource. Bindings can live at the **registry level** or on a
**specific resource**.
2. **List Policy Bindings on Parent Folders / Organization**:
#### Registry-level IAM policy
```bash
# Folder level
gcloud iam policy-bindings list \
--target="//cloudresourcemanager.googleapis.com/folders/${FOLDER_ID}" \
--location=global
# Organization level
gcloud iam policy-bindings list \
--target="//cloudresourcemanager.googleapis.com/organizations/${ORGANIZATION_ID}" \
--location=global
```
3. **Inspect the Referenced AccessPolicy**:
```bash
gcloud iam access-policies get ACCESS_POLICY_ID --location=global
```
* Verify the rule uses the full FQDN permission:
`iap.googleapis.com/resources.egressViaIAP`.
* Verify the agent's principal or principal set matches
`rules[].principals`.
* Inspect CEL conditions: check for null dereferences, missing `.json`
extensions in REST API paths, or overly broad
`destination.is_registered == false` DENY rules that crash
unauthenticated agent boot routines.
4. **Check for Org Policy Constraint Blockers (BKI 24)**: If `gcloud iam
policy-bindings create` failed with `CUSTOM_ORG_POLICY_VIOLATION`, check
whether `constraints/iam.managed.disableAccessPolicyBinding` is enforced:
```bash
gcloud org-policies describe constraints/iam.managed.disableAccessPolicyBinding --project=$PROJECT_ID
```
To unblock, apply an override disabling the constraint (`enforce: false`).
5. **REST API Fallback for UAP Bindings**:
```bash
curl -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://iam.googleapis.com/v3/projects/${PROJECT_ID}/locations/global/policyBindings"
```
#### Option B: IAM v1 (Legacy Policy Model)
In IAM v1, `roles/iap.egressor` is bound directly to the IAP shadow resource at
the registry or endpoint level:
##### Registry-level IAM policy
```bash
curl -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
@@ -694,7 +846,7 @@ curl -H "Authorization: Bearer $(gcloud auth application-default print-access-to
-H "Content-Type: application/json"
```
#### Per-endpoint IAM policy
##### Per-endpoint IAM policy
```bash
curl -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
@@ -703,7 +855,7 @@ curl -H "Authorization: Bearer $(gcloud auth application-default print-access-to
-H "Content-Type: application/json"
```
#### Same call, but for global registry:
##### Same call, but for global registry:
```bash
curl -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
@@ -720,22 +872,51 @@ service account or principal set with role `roles/iap.egressor`.
#### List authz extensions
```bash
gcloud beta service-extensions authz-extensions list \
gcloud service-extensions authz-extensions list \
--location=$LOCATION --project=$PROJECT_ID
gcloud beta service-extensions authz-extensions describe RESOURCE_NAME \
gcloud service-extensions authz-extensions describe RESOURCE_NAME \
--location=$LOCATION --project=$PROJECT_ID
```
#### Verify AuthorizationPolicy Binding
#### Verify AuthorizationPolicy & AuthzExtension Wiring
Verify that the `AuthorizationPolicy` is correctly bound to your `Gateway`. The
policy must target the gateway resource. If it is not bound, the authorization
logic will not be applied to the gateway traffic.
1. **Check `iapPolicyVersion` in Authz Extension**: Describe the Authz
Extension and inspect the `metadata` map:
1. Inspect the `AuthorizationPolicy` resource (retrieved from the API via
`gcloud` if available).
2. Ensure the policy's target matches the gateway's name and location.
```bash
gcloud service-extensions authz-extensions describe RESOURCE_NAME \
--location=$LOCATION --project=$PROJECT_ID --format="yaml(metadata)"
```
- If using **UAP (Policy V2)**: `iapPolicyVersion` MUST be explicitly set
to `"V2"`.
- If using **IAM v1**: `iapPolicyVersion` is `"V1"` or unset.
- Check `iamEnforcementMode`: `"DRY_RUN"` logs evaluations without
blocking; `"ENFORCED"` actively blocks.
2. **Verify AuthorizationPolicy Binding**: Verify that the
`AuthorizationPolicy` is correctly bound to your `Gateway`. The policy must
target the gateway resource. If it is not bound, the authorization logic
will not be applied to the gateway traffic.
- Inspect the `AuthorizationPolicy` resource.
- Ensure the policy's target matches the gateway's name and location.
3. **Verify Multi-Registry Gateway Constraints (BKI 26)**: Inspect the Agent
Gateway description:
```bash
gcloud network-services agent-gateways describe AGENT_GATEWAY_NAME \
--location=$LOCATION --project=$PROJECT_ID --format="yaml(registries)"
```
- Network Services permits at most **2 registries** bound to an Agent
Gateway.
- When 2 registries are configured, **exactly ONE must be `global`** and
the second must be **`regional` or `multi-regional`**.
- Configuring two regional, two multi-regional, or two global registries
will be rejected with HTTP 400 validation failure.
#### List authz policies, agent gateways, and authz extensions via raw API
@@ -815,6 +996,99 @@ Run:**
If permissions seem flaky, suspect the **PrincipalSet** binding. Move to a **1:1
Principal binding** (bind the specific service account directly) to verify.
### Step 9 — Cross-Project Runtime-to-Gateway Binding Verification
When an Agent Runtime (Reasoning Engine) in Project A (`AE_PROJECT_NUMBER`)
binds to an Agent Gateway in Centralized Governance Project B
(`AGW_PROJECT_ID`):
1. **Check Deployer Caller Permissions:** Verify the deploying identity has
`roles/networkservices.viewer` (or `networkservices.agentGateways.get` and
`networkservices.agentGateways.use`) on the gateway in Project B:
```bash
gcloud network-services agent-gateways describe GATEWAY_NAME \
--project=AGW_PROJECT_ID \
--location=REGION
```
2. **Check Vertex AI Service Agent IAM in Gateway Project:** Ensure Project A's
service agent is authorized in Project B:
```bash
gcloud projects get-iam-policy AGW_PROJECT_ID \
--flatten="bindings[].members" \
--filter="bindings.members:service-AE_PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com" \
--format="table(bindings.role)"
```
Must have `roles/networkservices.viewer` or custom role
`ae_agw_cross_project_sa`
(`networkservices.agentGateways.get,networkservices.operations.get`).
3. **Validate Regional Colocation Invariant:** Both the runtime and the gateway
**MUST** be deployed in the exact same region (e.g. `us-central1`).
Cross-region bindings fail control plane validation with `INVALID_ARGUMENT`.
4. **Verify Context-Aware Access (CAA) Token Sharing Opt-Out:** Verify that the
agent runtime was deployed with:
`"GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES": False` Without
this, bound token enforcement drops credentials during cross-project egress,
returning `401 Context-Aware Access requirements are not met`.
5. **Validate IAP Policy Target (Server-Generated Resource ID vs Friendly
Name):** Ensure IAP policies (`gcloud iap web add-iam-policy-binding`)
target the server-generated resource ID (`agentregistry-00000000-...`
extracted via `registryResource`), NOT the friendly service name:
```bash
gcloud agent-registry services describe MCP_SERVICE_NAME \
--project=AGW_PROJECT_ID \
--location=REGION \
--format='value(registryResource)'
```
6. **Verify Cross-Project VPC Service Controls (VPC-SC):** If Project A and
Project B are in separate perimeters, ensure a perimeter bridge or
ingress/egress rules cover `aiplatform.googleapis.com`,
`networkservices.googleapis.com`, and `agentregistry.googleapis.com`.
### Step 10 — Diagnose Secure Web Proxy (SWP) and Policy-Based Routing (PBR) Egress
If the consumer VPC employs a downstream Secure Web Proxy (SWP) behind an Agent
Gateway Network Attachment for Layer 7 inspection:
1. **Verify Next-Hop SWP Deployment Mode**: Confirm SWP is deployed directly in
the consumer VPC with `type: SECURE_WEB_GATEWAY` and
`routingMode: NEXT_HOP_ROUTING_MODE`. Do NOT attempt to use PSC Service
Attachments (which only support explicit `HTTP CONNECT` proxies) or static
routes (which fail because Network Attachments cannot bear VM instance tags).
2. **Verify Policy-Based Routing (PBR) Matching**:
Ensure a PBR exists with priority 200 matching source CIDR of the PSC-I subnet
(`10.20.1.0/24`) and destination `0.0.0.0/0` routing to `--next-hop-ilb-ip=<SWP_IP>`:
```bash
gcloud network-connectivity policy-based-routes create agw-psci-to-swp-pbr \
--network=projects/PROJECT_ID/global/networks/VPC_NAME \
--source-range=10.20.1.0/24 \
--destination-range=0.0.0.0/0 \
--next-hop-ilb-ip=10.20.1.250 \
--priority=200 \
--project=PROJECT_ID
```
Ensure a default fallback PBR exists with priority 1000 routing to `DEFAULT_ROUTING`.
3. **Verify Cloud NAT `ENDPOINT_TYPE_SWG` on Proxy Subnet**:
Ensure the Cloud NAT covering the proxy-only subnet (`REGIONAL_MANAGED_PROXY`)
includes `--endpoint-types=ENDPOINT_TYPE_VM,ENDPOINT_TYPE_SWG`. Without this,
SWP proxy backends cannot establish outbound connections to the internet.
4. **Verify GatewaySecurityPolicy CEL Allowlist**:
Check SWP logs for `DENIED` decisions under `networkservices.googleapis.com/Gateway`.
Verify that both external target domains and the regional AI Platform session
endpoint (`<region>-aiplatform.mtls.googleapis.com`) are allowed in CEL rules:
```cel
inUrlList(host(), ['api.weather.gov', 'api.ipify.org', 'REGION-aiplatform.mtls.googleapis.com'])
```
Omission of the regional AI Platform endpoint breaks ADK streaming queries
with HTTP 503 (`remote connection failure`).
## Quick reference: the checks in order
When triaging a 403 or connection failure, walk these in order:
@@ -822,18 +1096,44 @@ When triaging a 403 or connection failure, walk these in order:
1. Confirm the error in the agent log (403 vs SSL Handshake Timeout vs
Connection Reset).
2. Find the *exact hostname* in the gateway log (if 403).
3. Check IAP audit log: decision, principal, `iamEnforcementMode`, and watch
for `unregisteredResource`. 3b. If no IAP audit entry, pull gateway proxy
load-balancer log. 3c. If SSL Handshake Timeout or Connection Reset, run the
PSC Subnet Exhaustion Check.
3. Check IAP audit log: decision, principal, policy version
(`iapPolicyVersion="v2"` vs `"V1"`), `iamEnforcementMode`, and watch for
`unregisteredResource`.
- 3b. If no IAP audit entry, pull gateway proxy load-balancer log.
- 3c. If SSL Handshake Timeout or Connection Reset, run the PSC Subnet
Exhaustion Check.
- 3d. If timeouts or silent connection drops to public internet under
VPC-SC, verify Agent Connectivity Template (`vpcEgress: ALL_TRAFFIC`)
and Consumer Cloud NAT routing.
4. Confirm hostname is registered.
5. Check IAM on the registry/resource (`roles/iap.egressor`).
5. Check IAM / UAP Policies on the target:
6. Check authz extensions and policies.
- IAM v1: Check `roles/iap.egressor` on the registry/endpoint shadow
resource.
- UAP (Policy V2): Check CRM `PolicyBinding` and `AccessPolicy` across
Project, Folder, and Org. Verify CEL rules and check for Org Policy
blocker `constraints/iam.managed.disableAccessPolicyBinding`.
6. Check authz extensions (`iapPolicyVersion: "V2"` vs `"V1"`),
AuthorizationPolicy targets, and multi-registry constraints (max 2: 1
global + 1 regional/multi-regional).
7. Confirm agent identity has baseline roles. 7b. Check PAB policies. 7c. Check
Cloud Run egress auth (impersonation).
8. If behavior is flaky, switch to direct Principal bindings.
9. If cross-project runtime binding (Project A -> Gateway in Project B), verify
deployer and service agent viewer permissions on Project B, regional
colocation, CAA opt-out
(`GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES: False`), and
server-generated resource ID in IAP bindings.
10. If routing egress through a downstream Secure Web Proxy (SWP), verify Next-Hop
SWP mode (NOT PSC endpoint/service attachment), PBR source-range matching
on the PSC-I subnet (priority 200) to next-hop ILB IP, Cloud NAT with
`ENDPOINT_TYPE_SWG` on the proxy-only subnet, and GatewaySecurityPolicy CEL
allowlist for external domains and regional aiplatform sessions.
skills/cloud/agent-platform-troubleshooting/references/known-issues.md
+651 -66
View File
@@ -10,12 +10,10 @@ these patterns; it's faster than walking the full diagnostic flow.
(Lines 54-76)
- [2. ReasoningEngine startup: Assembly Service init](#2-reasoningengine-startup-assembly-service-failed-to-initialize)
(Lines 77-89)
- [3. Private-preview limit: one RE per project](#3-private-preview-limit-one-reasoning-engine-per-project-bonded-to-an-agent-gateway)
- [3. Private-preview limit: one RE per project](#3-private-preview-limit-one-agent-runtime-instance-per-project-bonded-to-an-agent-gateway)
(Lines 90-102)
- [4. 403 / Egress not authorized — missing authz_policy](#egress-not-authorized)
(Lines 103-124)
- [5. IAP ENFORCE mode blocks Agent Runtime startup](#5-iap-enforce-mode-blocks-agent-runtime-startup)
(Lines 125-143)
- [4. 403 / Egress not authorized — missing authz_policy or UAP binding](#egress-not-authorized)
- [5. IAP ENFORCE / Blanket DENY blocks Agent Runtime startup](#5-iap-enforce-mode-blocks-agent-runtime-startup)
- [6. Self-signed / Private CA destinations](#6-self-signed-private-ca-destinations-not-supported-early-versions)
(Lines 144-156)
- [7. Principal Access Boundary (PAB) override](#7-principal-access-boundary-pab-policies-overriding-iam-allow)
@@ -24,6 +22,7 @@ these patterns; it's faster than walking the full diagnostic flow.
(Lines 189-239)
- [9. Cross-Project Gateway import failing (Code 13)](#9-cross-project-agent-gateway-import-creation-failing-code-13-internal-error)
(Lines 240-295)
- [10. Cross-Project Runtime-to-Gateway Binding Failures](#10-cross-project-runtime-to-gateway-binding-failures-permission_denied-invalid_argument-or-caa-token-suppression)
- [11. RE Container Crash: Dependency Mismatch](#11-reasoning-engine-container-crash-python-dependency-mismatch-startup-exceptions)
(Lines 296-318)
- [12. mTLS Egress Failure: Expired Cert](#12-mtls-egress-failure-invalid-or-expired-client-certificate)
@@ -45,9 +44,17 @@ these patterns; it's faster than walking the full diagnostic flow.
- [21. Agent Identity Egress to Cloud Run/Funcs (JWT issue)](#21-agent-identity-egress-to-cloud-run-functions-fails-with-401403-jwt-auth-issue)
(Lines 680-725)
- [22. Agent deployed but not visible in Registry](#22-agent-deployed-but-not-visible-in-registry-boundary-issue)
(Lines 726-754)
- [23. Telemetry Egress Blocked by Egress Gateway](#23-telemetry-egress-blocked-by-egress-agent-gateway-startup-failure)
(Lines 755-779)
- [24. UAP Org Policy Blocker (disableAccessPolicyBinding)](#24-uap-org-policy-blocker-constraintsiammanageddisableaccesspolicybinding)
- [25. UAP Policy Binding Destruction Race Condition](#25-uap-policy-binding-destruction-race-condition-400-failed_precondition)
- [26. Agent Gateway Multi-Registry Validation Error](#26-agent-gateway-multi-registry-validation-error-missing-global-registry)
- [27. CEL Null Dereference Crash on Non-Tool MCP Methods](#27-cel-null-dereference-crash-on-non-tool-mcp-methods)
- [28. ACT ALL_TRAFFIC Silent Outbound Hang (Missing Cloud NAT)](#28-act-all_traffic-silent-outbound-hang-missing-cloud-nat)
- [29. CCFE Agent Gateway Mutation Concurrency Lock (HTTP 409 ABORTED)](#29-ccfe-agent-gateway-mutation-concurrency-lock-http-409-aborted)
- [30. Dangling Reference on ACT Deletion (FROM_AGENT_GATEWAY)](#30-dangling-reference-on-act-deletion-from_agent_gateway)
- [31. Reasoning Engine Deletion Preconditions (Active Sessions)](#31-reasoning-engine-deletion-preconditions-active-sessions-require-forcetrue)
- [32. REST API Path Matching Gotcha (.json and Query Strings)](#32-rest-api-path-matching-gotcha-json-extensions-and-query-strings)
- [33. Secure Web Proxy (SWP) & Policy-Based Routing (PBR) Egress Failures](#33-secure-web-proxy-swp--policy-based-routing-pbr-egress-failures)
--------------------------------------------------------------------------------
@@ -100,44 +107,67 @@ for any new REs.
--------------------------------------------------------------------------------
## 4. 403 / "Egress request is not authorized" — missing `authz_policy` targeting the gateway {#egress-not-authorized}
## 4. 403 / "Egress request is not authorized" — missing `authz_policy` or UAP binding {#egress-not-authorized}
**Symptom:** Audit logs show `GatekeeperAuthorizer.AuthorizeUser` returning
`Permission Denied` for `iap.webServiceVersions.egressViaIAP`.
`Permission Denied` for `iap.webServiceVersions.egressViaIAP` or
`iap.googleapis.com/resources.egressViaIAP`.
**Cause:** The IAP authz extension exists but no `authz_policy` actually
attaches it to the specific Agent Gateway. The extension is configured but the
policy that targets the gateway resource is missing.
**Cause:**
**Fix:** Verify there's an `authz_policy` in the consumer project that targets
the specific `agentGateway` resource. Check with:
1. **Missing AuthzPolicy Attachment**: The IAP authz extension exists, but no
`authz_policy` targets the specific Agent Gateway (`target.resources[]`).
2. **Missing or Misconfigured UAP Binding (Policy V2)**: When the gateway uses
`iapPolicyVersion: "V2"`, no `PolicyBinding` binds an `AccessPolicy` to the
target CRM node
(`//cloudresourcemanager.googleapis.com/projects/${PROJECT_NUMBER}`), or the
target resource was specified using the string project ID instead of the
numeric project number.
```bash
curl -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
"https://networksecurity.googleapis.com/v1alpha1/projects/${PROJECT_ID}/locations/${LOCATION}/authzPolicies"
```
**Fix:**
Look for a policy whose `target.resources[]` includes the gateway's full
resource name.
1. Verify the `authz_policy` targets the gateway:
```bash
gcloud network-security authz-policies list --location=${LOCATION}
```
2. If using UAP, verify the policy binding targets the numeric project number:
```bash
gcloud iam policy-bindings list \
--target-resource="//cloudresourcemanager.googleapis.com/projects/${PROJECT_NUMBER}" \
--location=global
```
--------------------------------------------------------------------------------
## 5. IAP `ENFORCE` mode blocks Agent Runtime startup
## 5. IAP `ENFORCE` mode / Blanket DENY blocks Agent Runtime startup
**Symptom:** Agent Runtime instances fail to deploy or start, with 403s in
startup logs.
**Symptom:** Agent Runtime instances fail to deploy or start, with 403s or
connection resets in startup logs (`reasoning_engine_stderr`).
**Cause:** With IAP in `ENFORCE` mode on the Agent Gateway, all egress is denied
by default — including the bootstrap calls the agent runtime makes to
`cloudresourcemanager`, `aiplatform`, `logging`, `monitoring`, etc. If those
endpoints aren't registered AND the agent identity isn't granted
`roles/iap.egressor` on them, startup never completes.
**Cause:**
**Fix:** Make sure the bootstrap service set is registered in the Agent Registry
(see field-manual's "Recommended Registry Structure: Consolidated Google APIs"
section) and the agent identity has `roles/iap.egressor` on them. While
debugging, flipping IAP to `DRY_RUN` lets you observe the failing destinations
without blocking startup.
1. **Default-Deny without Bootstrap Endpoints**: With IAP in `ENFORCE` mode on
the Agent Gateway, all egress is denied by default — including bootstrap
calls the agent runtime makes to `telemetry.mtls.googleapis.com`,
`cloudresourcemanager`, `aiplatform`, `logging`, `monitoring`, etc.
2. **Blanket Unregistered DENY in UAP**: An Org or Folder Admin created an
explicit `effect: DENY` rule on `(destination.is_registered == false)`.
Because DENY has absolute precedence and cannot be overridden by
project-level rules, Reasoning Engine boot traffic to
`telemetry.mtls.googleapis.com` is dropped immediately.
**Fix:**
- Ensure all required monitoring/telemetry endpoints
(`telemetry.mtls.googleapis.com`, `monitoring.googleapis.com`,
`trace.mtls.googleapis.com`, `cloudtrace.googleapis.com`) are registered in
the Agent Registry and permitted by policy.
- **Never create a blanket DENY on `destination.is_registered == false`**.
Rely on gateway default-deny instead.
- Set IAP to `DRY_RUN` (`iamEnforcementMode: "DRY_RUN"`) to observe failing
destinations without blocking container initialization.
--------------------------------------------------------------------------------
@@ -241,11 +271,11 @@ registry entries to dynamically build its routing table.
**Symptom:**
- Happening during `gcloud alpha network-services agent-gateways import` or
- Happening during `gcloud network-services agent-gateways import` or
creation.
- CLI rejects the command with a synchronous error: `ERROR:
(gcloud.alpha.network-services.agent-gateways.import) { "code": 13,
"message": "an internal error has occurred" }`
(gcloud.network-services.agent-gateways.import) { "code": 13, "message": "an
internal error has occurred" }`
- Gateway YAML specifies **cross-project** infrastructure targets (e.g. a PSC
network attachment or target DNS peering network residing inside another
network host project, e.g., Project B).
@@ -256,24 +286,45 @@ authorized to map the target resource. The validation is executed under the
**Agent Gateway Control Plane Service Agent** identity:
`service-GATEWAY_PROJECT_NUMBER@gcp-sa-agentgateway.iam.gserviceaccount.com`
If this service agent lacks read permissions on the target network project, the
validation fails, causing the import command to exit with a synchronous `Code
13` internal error.
If this service agent lacks read permissions on the target network project, or
if the network configuration itself is invalid, the validation fails, causing
the import command to exit with a synchronous `Code 13` internal error.
Specifically, the following network configuration rules must be honored:
- **VPC Network Mismatch:** The VPC network containing the PSC network
attachment and the target network specified in the DNS peering configuration
(`dnsPeeringConfig.targetNetwork`) must be the exact same VPC network. If
they point to different VPC networks, the configuration validation fails.
- **DNS Zone Validation Failure:** The domains specified in
`dnsPeeringConfig.domains` must exist as private DNS zones (e.g., private
zones or forwarding zones) in the target VPC network. If they do not exist,
validation fails.
**Investigation:**
1. Verify the caller has full Owner/Editor rights on both projects.
1. Verify the caller (user EUC) has `roles/compute.networkViewer` and
`roles/dns.reader` (or basic Owner/Editor/Viewer roles) on the target
network host project.
2. Check if the network attachment `connectionPreference` is set to
`ACCEPT_AUTOMATIC`.
3. Check the IAM policy in the target network project. Verify if the gateway's
**control-plane service agent** is missing:
**control-plane service agent** is missing the required roles:
`service-GATEWAY_PROJECT_NUMBER@gcp-sa-agentgateway.iam.gserviceaccount.com`
4. Confirm if the **egress proxy service agent**
(`service-GATEWAY_PROJECT_NUMBER@gcp-sa-dep.iam.gserviceaccount.com`) is
present (typically holds `roles/compute.networkUser` and `roles/dns.peer`).
*Note:* The egress proxy service agent
(`service-GATEWAY_PROJECT_NUMBER@gcp-sa-dep.iam.gserviceaccount.com`) does
**not** require these permissions manually granted on the host project.
4. Verify that the PSC network attachment's network matches the DNS peering
target network exactly.
5. Check if all domains specified under `dnsPeeringConfig.domains` correspond
to existing private DNS zones associated with the target network VPC (e.g.,
they align with the DNS forwarding zone, or use `.` if a catch-all peering
zone is configured).
**Fix:** You must grant the control-plane service agent the required validation
permissions inside the target network host project:
permissions inside the target network host project, ensure the target operator
has required reader roles, and align the VPC networks and domains in the
configuration:
1. Grant `roles/compute.networkUser` to validate the PSC network attachment:
@@ -291,6 +342,240 @@ permissions inside the target network host project:
--role="roles/dns.peer"
```
3. Ensure the operator has `roles/compute.networkViewer` and `roles/dns.reader`
on the target network project:
```bash
gcloud projects add-iam-policy-binding TARGET_NETWORK_PROJECT_ID \
--member="user:OPERATOR_EMAIL" \
--role="roles/compute.networkViewer"
gcloud projects add-iam-policy-binding TARGET_NETWORK_PROJECT_ID \
--member="user:OPERATOR_EMAIL" \
--role="roles/dns.reader"
```
4. Configure the gateway YAML definition such that the VPC network containing
the network attachment match the target network for DNS peering
(`dnsPeeringConfig.targetNetwork`) exactly.
5. Ensure any domain configured under `dnsPeeringConfig.domains` is registered
as a private DNS zone in the target VPC network (e.g. matching the exact
peered zone or using `.` for root/catch-all peering).
--------------------------------------------------------------------------------
## 10. Cross-Project Runtime-to-Gateway Binding Failures: PERMISSION_DENIED, INVALID_ARGUMENT, or CAA Token Suppression
**Symptom:**
- **Symptom A (Deployer PERMISSION_DENIED):** Calling Vertex AI
`client.agent_engines.create(...)` or REST API in Project 1
(`AE_PROJECT_ID`) referencing an Agent Gateway in Project 2
(`AGW_PROJECT_ID`) fails synchronously with: `ERROR:
(gcloud.ai.reasoning-engines.create) PERMISSION_DENIED: Permission
'networkservices.agentGateways.use' denied on resource
'//networkservices.googleapis.com/projects/AGW_PROJECT_ID/locations/REGION/agentGateways/GATEWAY_NAME'`
- **Symptom B (Control-Plane INVALID_ARGUMENT):** Deployment fails with:
`INVALID_ARGUMENT: Agent Gateway project in
spec.deployment_spec.agent_gateway_config.agent_to_anywhere_config.agent_gateway
must match the Vertex AI project.` or region mismatch `INVALID_ARGUMENT:
Agent Gateway location in spec... must match the Vertex AI location.`
- **Symptom C (CAA 401 Unauthenticated Egress):** Runtime container deploys
successfully, but egress requests to external endpoints or MCP tools fail
immediately with: `Error Code: "401", Error Details: "Context-Aware Access
requirements are not met"` or `Request had invalid authentication
credentials.`
- **Symptom D (Proxy 403 PERMISSION_DENIED on MCP Tool / Service Name):**
Egress through the gateway fails with HTTP 403 `PERMISSION_DENIED`, even
though `roles/iap.egressor` was granted to the friendly service name in
Agent Registry.
- **Symptom E (Unsupported Folder-Level principalSet):** Policy binding using
`principalSet://TRUST_DOMAIN/attribute.platformContainer/aiplatform/folders/FOLDER_NUMBER`
fails silently or is rejected as an invalid principal identifier.
- **Symptom F (VPC-SC Perimeter Block):** Egress calls fail with
`VPC_SERVICE_CONTROLS_VIOLATION` across project boundaries.
**Cause:**
1. **Cross-Project Feature Rollout & Colocation Invariants:** Following the
`EnableCrossProjectAgentGateway_Default_Launch` rollout, cross-project
binding
(`spec.deploymentSpec.agentGatewayConfig.agentToAnywhereConfig.agentGateway`)
is supported. However, **both the runtime and the gateway must reside in the
exact same Google Cloud region** (e.g. `us-central1`). Cross-region bindings
are rejected with `INVALID_ARGUMENT`. Furthermore, all runtimes in a given
project and region must bind to the identical gateway instance.
2. **Missing Deployer and Service Agent IAM Permissions:**
- The **deploying principal** (user EUC, Cloud Build SA, or pipeline
identity) must have `networkservices.agentGateways.use` and
`networkservices.agentGateways.get` on the gateway in `AGW_PROJECT_ID`.
- The runtime project's **Vertex AI Service Agent**
(`service-AE_PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com`)
must hold `roles/networkservices.viewer` (or custom role
`ae_agw_cross_project_sa` with `networkservices.agentGateways.get` and
`networkservices.operations.get`) on `AGW_PROJECT_ID`.
3. **Context-Aware Access (CAA) Token Suppression:** By default, Google Cloud
SDKs and `google-auth` enforce bound token protection. In cross-project
egress, bound token requirements reject requests unless the runtime
explicitly sets the environment variable
`GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES: False`.
4. **Friendly Service Name vs. Server-Generated Resource ID Trap:** In Agent
Registry, services, endpoints, and mcp-servers are written via `gcloud
agent-registry services create`. However, IAP authorization (`gcloud iap web
add-iam-policy-binding`) must bind to the **server-generated resource ID**
(`agentregistry-00000000-...`, extracted from the `registryResource` field),
NOT the friendly service name (`[MCP_SERVICE_NAME]`). Binding to the
friendly service name attaches the policy to a non-existent IAP resource.
5. **SPIFFE principalSet Syntax Limitation:**
`attribute.platformContainer/aiplatform/folders/FOLDER_NUMBER` is **not** an
indexed attribute in IAM SPIFFE URI evaluation. Scoping must strictly use
project-level syntax
(`attribute.platformContainer/aiplatform/projects/PROJECT_NUMBER`) or
trust-domain-wide syntax (`principalSet://TRUST_DOMAIN/*`), or use Principal
Access Boundary (PAB) policies.
6. **VPC Service Controls Perimeter Boundary:** If Project 1 (`AE_PROJECT_ID`)
and Project 2 (`AGW_PROJECT_ID`) reside in different perimeters, perimeter
bridging or bidirectional ingress/egress rules covering
`aiplatform.googleapis.com`, `networkservices.googleapis.com`, and
`agentregistry.googleapis.com` are required.
**Investigation:**
1. **Check Deployer Permissions on Gateway:**
```bash
gcloud network-services agent-gateways describe GATEWAY_NAME \
--project=AGW_PROJECT_ID \
--location=REGION
```
If this fails with `PERMISSION_DENIED`, the deployer lacks read/use
permissions.
2. **Check Vertex AI Service Agent IAM on Gateway Project:**
```bash
gcloud projects get-iam-policy AGW_PROJECT_ID \
--flatten="bindings[].members" \
--filter="bindings.members:service-AE_PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com" \
--format="table(bindings.role)"
```
Verify that `roles/networkservices.viewer` or custom role
`ae_agw_cross_project_sa` is present.
3. **Verify Runtime Region vs. Gateway Region:** Ensure both the Reasoning
Engine runtime and the Agent Gateway are deployed in the same region.
4. **Inspect Runtime CAA Environment Variable:** Inspect the Reasoning Engine
deployment spec or Python definition. Verify that `env_vars` contains
`"GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES": False`.
5. **Verify IAP Binding Target Resource ID:**
```bash
# Fetch the server-generated registryResource identifier
gcloud agent-registry services describe MCP_SERVICE_NAME \
--project=AGW_PROJECT_ID \
--location=REGION \
--format='value(registryResource)'
```
Confirm that `gcloud iap web add-iam-policy-binding` used the trailing UUID
(`agentregistry-00000000-...`) as `--mcp-server` or `--agent`, not the
friendly name.
6. **Check VPC-SC Status:** Confirm whether `AE_PROJECT_ID` and
`AGW_PROJECT_ID` reside in the same VPC-SC perimeter or have perimeter
ingress/egress rules allowing communication.
**Fix:**
1. **Grant Deployer Permissions:** Grant `roles/networkservices.viewer` (or
permissions `networkservices.agentGateways.use` and
`networkservices.agentGateways.get`) to the deploying identity on
`AGW_PROJECT_ID`:
```bash
gcloud projects add-iam-policy-binding AGW_PROJECT_ID \
--member="user:DEPLOYER_EMAIL" \
--role="roles/networkservices.viewer"
```
2. **Grant Vertex AI Service Agent Permissions:** Grant the Agent Runtime's
service agent access to the governance project: *Option A (Standard Viewer
Role):*
```bash
gcloud projects add-iam-policy-binding AGW_PROJECT_ID \
--member="serviceAccount:service-AE_PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com" \
--role="roles/networkservices.viewer"
```
*Option B (Least-Privilege Custom Role):*
```bash
gcloud iam roles create ae_agw_cross_project_sa \
--project="AGW_PROJECT_ID" \
--title="AE AGW Cross Project SA" \
--description="Custom role for cross-project service agent to access Agent Gateways and Operations" \
--permissions="networkservices.agentGateways.get,networkservices.operations.get" \
--stage="GA"
gcloud projects add-iam-policy-binding AGW_PROJECT_ID \
--member="serviceAccount:service-AE_PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com" \
--role="projects/AGW_PROJECT_ID/roles/ae_agw_cross_project_sa"
```
3. **Configure Runtime with CAA Opt-Out:** In the agent runtime deployment
configuration:
```python
remote_agent = client.agent_engines.create(
agent=local_agent,
config={
"agent_gateway_config": {
"agent_to_anywhere_config": {
"agent_gateway": (
"projects/AGW_PROJECT_ID/locations/REGION/agentGateways/AGENT_GATEWAY_TO_ANYWHERE_NAME"
)
},
},
"identity_type": types.IdentityType.AGENT_IDENTITY,
"env_vars": {
"GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES": (
False
),
},
},
)
```
4. **Bind IAP Role to the Server-Generated Resource ID:**
```bash
REGISTRY_RESOURCE=$(gcloud agent-registry services describe MCP_SERVICE_NAME \
--project=AGW_PROJECT_ID \
--location=REGION \
--format='value(registryResource)')
RESOURCE_ID=$(basename "$REGISTRY_RESOURCE")
AGENT_PRINCIPAL="principal://agents.global.org-ORG_ID.system.id.goog/resources/aiplatform/projects/AE_PROJECT_NUMBER/locations/REGION/reasoningEngines/ENGINE_ID"
gcloud iap web add-iam-policy-binding \
--project=AGW_PROJECT_ID \
--region=REGION \
--resource-type=agent-registry \
--mcp-server="${RESOURCE_ID}" \
--member="$AGENT_PRINCIPAL" \
--role="roles/iap.egressor"
```
5. **Use Valid principalSet Syntax:** For baseline Google APIs access, avoid
folder-level scoping. Use project-level scoping:
```
principalSet://agents.global.org-ORG_ID.system.id.goog/attribute.platformContainer/aiplatform/projects/AE_PROJECT_NUMBER
```
or trust-domain-wide scoping:
```
principalSet://agents.global.org-ORG_ID.system.id.goog/*
```
--------------------------------------------------------------------------------
## 11. Reasoning Engine Container Crash: Python Dependency Mismatch / Startup Exceptions
@@ -423,7 +708,7 @@ the registry level.
2. Apply the policy to the Agent Registry:
```bash
gcloud alpha iap web set-iam-policy policy.json \
gcloud iap web set-iam-policy policy.json \
--resource-type=agent-registry \
--project=YOUR_PROJECT_ID
```
@@ -454,14 +739,14 @@ the registry level.
gateway.
```bash
gcloud beta ai reasoning-engines delete <REASONING_ENGINE_ID> --region=LOCATION
curl -X DELETE -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" "https://${LOCATION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${LOCATION}/reasoningEngines/${REASONING_ENGINE_ID}?force=true"
```
2. If the old Reasoning Engines are already deleted but routes are stuck,
delete the old gateway resource to force cleanup:
```bash
gcloud alpha network-services agent-gateways delete OLD_GATEWAY_NAME --location=LOCATION --project=PROJECT_ID
gcloud network-services agent-gateways delete OLD_GATEWAY_NAME --location=LOCATION --project=PROJECT_ID
```
3. Retry deployment bonded to the new gateway.
@@ -503,7 +788,7 @@ baked into the container.
certificates are ready:
```bash
gcloud alpha network-services agent-gateways describe GATEWAY_NAME \
gcloud network-services agent-gateways describe GATEWAY_NAME \
--location=LOCATION \
--format="value(agentGatewayCard.rootCertificates)"
```
@@ -512,7 +797,7 @@ baked into the container.
pipeline will automatically bake them in.
- **For Custom Container / Prebuilt Image Deployments:** Export the gateway's
root certificates using the `gcloud beta network-services gateways
root certificates using the `gcloud network-services gateways
export-cacerts` command and manually copy/bake them into your Dockerfile:
```dockerfile
@@ -527,21 +812,43 @@ baked into the container.
## 18. VPC Service Controls (VPC-SC) Perimeter Blocks Agent Gateway Provisioning
> [!NOTE]
> **Native VPC-SC Creation Support:** As of September 8, 2026, Agent Gateway
> creation inside a VPC-SC perimeter **works natively out of the box** without
> requiring manual control plane ingress policies, provided that the Agent
> Connectivity Template (ACT) specifies `vpcEgress: ALL_TRAFFIC`. The requirement
> for manual ingress policies is no longer needed for standard provisioning.
**Symptom:**
- Deployment of Agent Gateway fails when the project is inside a VPC-SC
perimeter.
- Deployment of Agent Gateway fails or runtime egress is blocked when the
project is inside a VPC-SC perimeter.
- Violation logs show `NETWORK_NOT_IN_SAME_SERVICE_PERIMETER` or similar
VPC-SC blocks for calls between the consumer project and Google-managed
projects.
infrastructure.
**Cause:** The Agent Gateway provisioning workflow uses Google-managed
infrastructure outside the customer's VPC-SC perimeter. Because the calls target
resources inside the perimeter using Google-managed service accounts, VPC-SC
blocks them as unauthorized ingress.
**Cause:**
**Fix:** Configure ingress policies in your VPC-SC perimeter to allow the
actuation identities. Two ingress rules are required:
1. **ACT Egress Mode Misconfiguration (Primary Cause):** The Agent Connectivity
Template is configured with `vpcEgress: PRIVATE_RANGES_ONLY` or the gateway is
deployed in Standalone mode (without an ACT). In these modes, traffic bypasses
the customer's VPC perimeter boundaries, causing VPC-SC to reject the
deployment or egress calls.
2. **Legacy or Strict Custom Perimeters (Historical):** Prior to September 8, 2026,
control plane provisioning workflows required explicit ingress policies for
Google-managed service agents. In legacy or custom perimeters that block
Google-managed actuation identities, explicit ingress policies may still be
required.
**Fix:**
1. **Verify ACT specifies `ALL_TRAFFIC`:** Ensure the Agent Connectivity Template
uses `vpcEgress: ALL_TRAFFIC` and binds to a valid PSC-I Network Attachment in
the consumer VPC with Cloud NAT configured on the subnet.
2. **Configure Ingress Policies (Legacy / Restrictive Perimeters Fallback):** If
operating in a strict perimeter where Google-managed actuation identities are
blocked, configure ingress policies in your VPC-SC perimeter to allow the
actuation identities. Two ingress rules are required:
### Rule 1: Gateway Control Plane Actuation
@@ -620,14 +927,14 @@ gateway cannot allocate the necessary interfaces and deployment fails.
project:
```bash
gcloud alpha network-services agent-gateways list --location=$LOCATION --project=$PROJECT_ID
gcloud network-services agent-gateways list --location=$LOCATION --project=$PROJECT_ID
```
2. **Identify the subnetwork used by the gateway's Network Attachment**:
Describe the gateway:
```bash
gcloud alpha network-services agent-gateways describe AGENT_GATEWAY_NAME --location=$LOCATION --project=$PROJECT_ID
gcloud network-services agent-gateways describe AGENT_GATEWAY_NAME --location=$LOCATION --project=$PROJECT_ID
```
Look for `networkConfig.egress.networkAttachment` or
@@ -653,8 +960,8 @@ gateway cannot allocate the necessary interfaces and deployment fails.
## 20. Config Validation Failure: Missing Agent Registry API enablement
**Symptom:** During `gcloud alpha network-services agent-gateways import`, the
command fails with:
**Symptom:** During `gcloud network-services agent-gateways import`, the command
fails with:
```
ERROR: (gcloud.alpha.network-services.agent-gateways.import) {
@@ -736,7 +1043,7 @@ an OIDC ID token to authenticate with Cloud Run.
**Symptom:** You have successfully deployed an agent (e.g., via Agent Runtime
SDK), but when you list agents from your designated Management Project using
`gcloud alpha agent-registry agents list`, the agent is missing.
`gcloud agent-registry agents list`, the agent is missing.
**Cause:** The Agent Registry uses a "Management Boundary" to define which
projects' agents are governed and visible. If the project where the agent was
@@ -782,7 +1089,7 @@ allow access via policy.
Endpoints in the Agent Registry:
```bash
gcloud alpha agent-registry endpoints list --location=us-central1 --project=$PROJECT_ID
gcloud agent-registry endpoints list --location=us-central1 --project=$PROJECT_ID
```
If any of them are missing, create endpoint entries for them (e.g.
@@ -790,7 +1097,7 @@ allow access via policy.
`trace.mtls.googleapis.com`, `cloudtrace.googleapis.com`):
```bash
gcloud alpha agent-registry endpoints create google-telemetry \
gcloud agent-registry endpoints create google-telemetry \
--fqdn=telemetry.mtls.googleapis.com \
--location=us-central1
```
@@ -798,3 +1105,281 @@ allow access via policy.
2. **Authorize the Egress:** Ensure your `AuthorizationPolicy` is correctly
bound to the Gateway and allows the agent's principal set to egress to these
endpoints.
--------------------------------------------------------------------------------
## 24. UAP Org Policy Blocker (constraints/iam.managed.disableAccessPolicyBinding)
**Symptom:** Creating an IAM Policy Binding (`gcloud iam policy-bindings create`
or Terraform `google_iam_policy_binding`) fails with:
```text
ERROR: (gcloud.beta.iam.policy-bindings.create) CUSTOM_ORG_POLICY_VIOLATION: Operation violates organization policy constraint 'constraints/iam.managed.disableAccessPolicyBinding'.
```
**Cause:** Enterprise Google Cloud organizations often enforce
`constraints/iam.managed.disableAccessPolicyBinding` by default to prevent
unauthorized IAM v3 bindings.
**Fix:** An Organization or Folder Administrator must apply an Org Policy V2
override setting `enforce: false` at the target resource level:
```bash
gcloud org-policies set-policy --project=${PROJECT_ID} override.yaml
```
Where `override.yaml` defines:
```yaml
name: projects/PROJECT_ID/policies/iam.managed.disableAccessPolicyBinding
spec:
rules:
- enforce: false
```
Allow 30–60 seconds for IAM policy control plane propagation before retrying.
--------------------------------------------------------------------------------
## 25. UAP Policy Binding Destruction Race Condition (400 FAILED_PRECONDITION)
**Symptom:** During automated teardown or `terraform destroy`, destroying
`google_iam_access_policy` fails with:
```text
HTTP 400 FAILED_PRECONDITION: Access policy is referenced by active policy binding(s).
```
**Cause:** Deleting a `PolicyBinding` is subject to a 10–30 second replication
delay in the IAM control plane. If an orchestrator deletes the binding and
immediately issues a deletion for the parent `AccessPolicy`, IAM rejects the
call because the binding is still visible in eventual consistency.
**Fix:** Ensure teardown workflows poll `GET` on the `PolicyBinding` resource
URL until an HTTP 404 is confirmed before deleting the referenced
`AccessPolicy`. In Terraform scripts or destroy provisioners, never exit 0 on
timeout.
--------------------------------------------------------------------------------
## 26. Agent Gateway Multi-Registry Validation Error (Missing Global Registry)
**Symptom:** `gcloud network-services agent-gateways import` or Terraform apply
fails with:
```text
Invalid argument: maximum of two registries are supported. When it's two registries, one of them must be global, the other one can be either regional or multi-regional.
```
**Cause:** The Agent Gateway was configured with two registries, but neither was
`global` (e.g. two regional registries like `us-central1` and `us-west1`), or
both were `global`.
**Fix:** In multi-registry mode, ensure **exactly one registry is `global`**,
and the second registry is regional or multi-regional:
```yaml
registries:
- //agentregistry.googleapis.com/projects/PROJECT_NUMBER/locations/global
- //agentregistry.googleapis.com/projects/PROJECT_NUMBER/locations/us-central1
```
--------------------------------------------------------------------------------
## 27. CEL Null Dereference Crash on Non-Tool MCP Methods
**Symptom:** Gatekeeper returns HTTP 500 or `PERMISSION_DENIED` with internal
evaluation error logs when an agent invokes MCP lifecycle or non-tool methods
(`prompts/list`, `resources/read`).
**Cause:** When invoking non-tool MCP operations, the `tool` message inside
`mcp_server` is `null`. A CEL rule that dereferences
`destination.agent_registry.mcp_server.tool.annotations.*` or `tool.name`
without first checking the method throws an unhandled null-dereference exception
in the CEL evaluator.
**Fix:** Guard all tool attribute dereferences with a method check:
```cel
(destination.agent_registry.mcp_server.method == 'tools') && (destination.agent_registry.mcp_server.tool.annotations.read_only_hint == true)
```
--------------------------------------------------------------------------------
## 28. ACT ALL_TRAFFIC Silent Outbound Hang (Missing Cloud NAT)
**Symptom:** When an Agent Gateway uses an Agent Connectivity Template with
`vpcEgress: ALL_TRAFFIC`, Reasoning Engine calls to public external APIs
(`api.weather.gov`, `api.ipify.org`, SaaS tools) hang and eventually fail with
connection timeouts. Egress to Google APIs continues to function normally.
**Cause:** In `ALL_TRAFFIC` mode, all non-Google outbound packets exit the PSC-I
Network Attachment into the consumer VPC subnet (`agw-psci-subnet`). If that
subnet does not have an active Cloud NAT gateway, public traffic is dropped.
(Google APIs bypass Cloud NAT via Andromeda Private Google Access).
**Fix:** Configure Cloud Router and Cloud NAT on the consumer VPC covering the
PSC-I subnet:
```bash
gcloud compute routers nats create agw-cloud-nat \
--router=agw-nat-router \
--region=LOCATION \
--nat-custom-subnet-ip-ranges=agw-psci-subnet \
--auto-allocate-nat-external-ips
```
--------------------------------------------------------------------------------
## 29. CCFE Agent Gateway Mutation Concurrency Lock (HTTP 409 ABORTED)
**Symptom:** Submitting a PATCH, update, or import command against an Agent
Gateway returns:
```text
HTTP 409 ABORTED: unable to queue the operation
```
**Cause:** Cloud Control Forwarding Engine (CCFE) enforces strict mutual
exclusion per Agent Gateway resource. Concurrent mutations are aborted while an
asynchronous update operation is already active.
**Fix:** Wait ~1.5 to 2 minutes for the in-flight Long-Running Operation (LRO)
to reach `done: true` before initiating another mutation.
--------------------------------------------------------------------------------
## 30. Dangling Reference on ACT Deletion (FROM_AGENT_GATEWAY)
**Symptom:** Attempting to delete an Agent Connectivity Template fails with:
```text
HTTP 400 FAILED_PRECONDITION: Resource projects/.../agentConnectivityTemplates/... is already being used by resource(s) projects/.../agentGateways/...
```
even after the referenced Agent Gateway has been deleted.
**Cause:** Network Services retained an internal cross-reference protection
(`FROM_AGENT_GATEWAY<ID>`) after gateway deletion.
**Fix:** Delete the dangling reference via Stubby:
```bash
stubby call blade:network-services-prod-${REGION} google.internal.cloud.reference.References.DeleteReference \
"name: 'projects/${PROJECT_NUMBER}/locations/${REGION}/agentConnectivityTemplates/${ACT_NAME}/references/FROM_AGENT_GATEWAY_${GATEWAY_ID}'"
```
--------------------------------------------------------------------------------
## 31. Reasoning Engine Deletion Preconditions (Active Sessions Require ?force=true)
**Symptom:** Deleting a Vertex AI Reasoning Engine instance fails with:
```text
HTTP 400 FAILED_PRECONDITION: Cannot delete ReasoningEngine with active sessions.
```
**Cause:** Active agent sessions or conversations remain attached to the engine
runtime.
**Fix:** Append `?force=true` to the delete API call or use the force flag in
the CLI/REST client to cascade session cleanup.
--------------------------------------------------------------------------------
## 32. REST API Path Matching Gotcha (.json Extensions and Query Strings)
**Symptom:** An agent calling an external REST API (e.g. Zendesk) receives HTTP
403 `Egress request is not authorized`, even though a UAP rule allows
`/api/v2/tickets`.
**Cause:** SDK clients typically append `.json` (e.g. `/api/v2/tickets.json`) or
query strings (`?page=1`). Strict equality checks reject these variations, while
naive `.startsWith('/api/v2/tickets')` introduces security escapes to
`/api/v2/tickets_delete_all`.
**Fix:** Construct explicit boundary-aware CEL path conditions:
```cel
((destination.unregistered.path == '/api/v2/tickets') ||
(destination.unregistered.path == '/api/v2/tickets.json') ||
(destination.unregistered.path.startsWith('/api/v2/tickets/')) ||
(destination.unregistered.path.startsWith('/api/v2/tickets?')) ||
(destination.unregistered.path.startsWith('/api/v2/tickets.json?')))
```
--------------------------------------------------------------------------------
## 33. Secure Web Proxy (SWP) & Policy-Based Routing (PBR) Egress Failures
**Symptom:** When attempting to route egress traffic from an Agent Gateway
Network Attachment through a downstream Secure Web Proxy (SWP) for Layer 7 URL
filtering, packets drop silently, connections time out, or requests fail with
`HTTP 403 Forbidden`, `HTTP 503 Service Unavailable: remote connection failure`,
or `INVALID_ARGUMENT`.
**Causes & Platform Constraints:**
1. **PSC Endpoints Cannot Be Route Next-Hops**: Google Cloud route tables (both
static routes and Policy-Based Routes) do **not** allow Private Service
Connect (PSC) forwarding rules (`--load-balancing-scheme=""`) or service
attachments as next hops.
2. **SWP via PSC Service Attachment Lacks Next-Hop Mode**: Publishing Secure
Web Proxy behind a PSC Service Attachment operates strictly in **Explicit
Proxy Mode** (`HTTP CONNECT`). Packets emerging from an Agent Gateway
Network Attachment are Layer 3/Layer 4 transparent IP packets; they cannot
reach an explicit proxy without client-side proxy configuration (which
Vertex AI Reasoning Engine containers do not support).
3. **Static Route Tag Defect**: Next-hop routing mode with static routes
requires VM network tags (`--tags`). Network Attachments
(`compute network-attachments`) are consumer network interfaces, not VM
instances, and cannot bear network tags. Static routes cannot match or steer
traffic originating from an Agent Gateway Network Attachment.
4. **SWP Gateway Resource Definition Syntax Error**: Creating an SWP gateway
with `type: SECURE_WEB_PROXY` fails validation. The Cloud Control Forwarding
Engine API requires `type: SECURE_WEB_GATEWAY` and
`routingMode: NEXT_HOP_ROUTING_MODE`.
5. **Cloud NAT Missing `ENDPOINT_TYPE_SWG`**: SWP Envoy proxy instances reside
on the private proxy-only subnet and lack external IPs. Outbound internet
egress requires Cloud NAT configured with
`--endpoint-types=ENDPOINT_TYPE_VM,ENDPOINT_TYPE_SWG`. If
`ENDPOINT_TYPE_SWG` is omitted, proxy outbound traffic hangs silently.
6. **ADK Streaming Session Allowlist Omission**: In `ALL_TRAFFIC` ACT mode,
Agent Development Kit (ADK) streaming queries initialize sessions via
`POST https://<region>-aiplatform.mtls.googleapis.com/.../sessions`. Because
`ALL_TRAFFIC` forces all runtime egress to the Network Attachment, PBR
steers this session call to SWP. If `<region>-aiplatform.mtls.googleapis.com`
is omitted from the GatewaySecurityPolicy allowlist, SWP drops it
(`default_denied`), causing ADK streaming queries to fail with HTTP 503
(`remote connection failure`), while direct `query()` calls succeed.
**Fix:**
1. **Deploy Next-Hop SWP directly in the consumer VPC**:
Create an Envoy proxy-only subnet (`purpose: REGIONAL_MANAGED_PROXY`, `role: ACTIVE`,
minimum `/26`) and deploy SWP listening on an internal IP on the PSC-I
subnet (e.g. `10.20.1.250`) with `type: SECURE_WEB_GATEWAY` and
`routingMode: NEXT_HOP_ROUTING_MODE`.
2. **Configure Policy-Based Routing (PBR)**:
Create a PBR matching source CIDR `10.20.1.0/24` (the Network Attachment
subnet) and destination `0.0.0.0/0` with `--next-hop-ilb-ip=10.20.1.250`
(priority 200). Add a default fallback PBR with priority 1000:
```bash
gcloud network-connectivity policy-based-routes create agw-psci-to-swp-pbr \
--network=projects/PROJECT_ID/global/networks/VPC_NAME \
--source-range=10.20.1.0/24 \
--destination-range=0.0.0.0/0 \
--next-hop-ilb-ip=10.20.1.250 \
--priority=200 \
--project=PROJECT_ID
```
3. **Configure Cloud NAT with `ENDPOINT_TYPE_SWG`**:
Update the Cloud NAT on the Cloud Router to include
`--endpoint-types=ENDPOINT_TYPE_VM,ENDPOINT_TYPE_SWG`.
4. **Allowlist External Domains and Regional AI Platform in GatewaySecurityPolicy**:
Ensure the CEL policy rule permits both external target domains and the
regional AI platform session endpoint:
`inUrlList(host(), ['api.weather.gov', 'api.ipify.org', '<region>-aiplatform.mtls.googleapis.com'])`.
skills/cloud/agent-platform-troubleshooting/references/policies.md
+681 -686
View File
File diff suppressed because it is too large Load Diff