<!-- Generated by docs/build.py from gcp.html. Edit the HTML, not this file. -->

# Google Cloud

Streams every request hitting a Cloud Load Balancing backend service to Obsero.

```
viewer -> Cloud CDN + external Application LB -> Cloud Logging
       -> Log Router sink -> Pub/Sub -> Cloud Run adapter -> POST /v1/events
                          \-> dead-letter topic (replayable)
```

Everything runs in your own project. The adapter strips `authorization`, `cookie` and `set-cookie` before anything leaves it.

## Read this first

> **Warning: Backend buckets produce no request logs**
>
> `BackendBucket` has **no `logConfig` field at all** in the Compute v1 API, so a static site served straight from Cloud Storage through Cloud CDN emits nothing to ship. Your origin has to be a backend **service**.
>
> Apps on Cloud Run, GKE or a MIG already are one and are unaffected. A pure GCS origin has to change before it can be onboarded — which is why the demo site here serves its static pages from Cloud Run rather than a bucket.

## Try it without a GCP project

```
cd gcp
make test                                        # adapter payload vs the contract
make event TOKEN=<tracking-id> DOMAIN=acme.com    # POST one event, check the ID
```

Then, to watch it work end to end, [the quickstart](quickstart.md) — or the make targets directly:

```
make site              # site + LB + header logging only
make sample            # confirm the headers reach Cloud Logging
make deploy            # the rest: sink, Pub/Sub, adapter
make traffic N=40      # mock browser / AI-agent / crawler traffic
make check             # score what actually reached the endpoint
make destroy           # the load balancer bills at idle -- do this
```

The two-stage start is deliberate: `make sample` confirms the header allow-list behaves before you build anything downstream of it.

## Wiring it into your own load balancer

*the whole integration*

```
module "obsero_ingestion" {
  source = "./ingestion"

  project_id  = var.project_id
  name_prefix = "acme-prod"
  site_token  = var.obsero_site_token

  backend_service_name = google_compute_backend_service.site.name
}
```

Unlike the AWS module there is no ARN to attach — this one configures logging on the named backend service directly. Better DX, at the cost of mutating a resource you own. It is reversible; `terraform destroy` takes the allow-list back off.

## The headline win: named request headers on the cheap path

CloudFront forces a choice — pay ~$11/month for real-time logs to get the full header set, or take standard logging and lose everything but User-Agent. **GCP has no such trade.** `BackendServiceLogConfig.loggingHttpRequestHeaders` names the headers you want and they arrive through the ordinary, cheap path:

```
from,                                            # GPTBot, Googlebot
signature-agent, signature-input, signature,     # Web Bot Auth
accept, accept-language,                         # browsers vs */* bots
sec-ch-ua, sec-ch-ua-platform, sec-ch-ua-mobile, # client hints
sec-fetch-dest                                   # presence alone is the signal
```

> **Caution: The API caps the list at 10**
>
> An eleventh header is rejected outright. Ten is enough only because `user-agent`, `referer` and the client IP arrive as first-class `httpRequest` fields whether or not you name them — spending slots on those would waste them. The adapter merges both sources.
>
> So the question is never “can I add one more?” It is which one you are dropping. `sec-fetch-mode` and `sec-fetch-site` were the two cut from the default list.

It is an allow-list, not “all headers”, which is arguably the better default: the payload carries only what classification needs, so nothing incidental leaves your project.

### What the logged headers actually look like

The allow-list lands under `jsonPayload.loggingHttpRequestHeaders`, as an **array of pairs**, with every value **base64-encoded**:

```
"loggingHttpRequestHeaders": [
  { "headerKey": "from",            "headerValue": "Ym90QG9wZW5haS5jb20=" },
  { "headerKey": "signature-agent", "headerValue": "Imh0dHBzOi8vY2hhdGdwdC5jb20i" }
]
```

- **The base64 is a feature.** It is why Web Bot Auth signatures survive byte-for-byte — quotes, commas and embedded colons included. CloudFront's percent-encoded `cs-headers` needed careful splitting on `%0A` *before* decoding, or a header value could forge an extra header. Nothing here can.
- **An absent header is simply missing from the array.** There is no empty-string placeholder, so “not sent” and “sent empty” are indistinguishable.
- **`httpRequest.userAgent` and `httpRequest.remoteIp` are populated regardless**, confirming those slots are free.

## Permissions — roles/editor is not enough

`roles/editor` deliberately excludes log-routing configuration and resource-level `setIamPolicy`, which are the core of this pipeline. Check these before anyone runs apply.

| Role | For |
|---|---|
| `roles/editor` | load balancer, CDN, backend services, NEGs, Pub/Sub topics and subscriptions, Artifact Registry |
| `roles/logging.configWriter` | the Log Router sink, and the `_Default` exclusion that stops you paying to store logs you already routed. **Not implied by Editor** — apply fails at the sink. |
| `roles/pubsub.admin` | `pubsub.topics.setIamPolicy`, to grant the sink's writer identity publisher on the topic. **`roles/pubsub.editor` does not cover it** — without it the sink is created but silently publishes nothing. |
| `roles/run.admin` | deploying the adapter and setting its invoker policy |
| `roles/iam.serviceAccountAdmin` | the two service accounts the module creates |

```
PROJECT=your-project
MEMBER=user:you@example.com

for ROLE in logging.configWriter pubsub.admin run.admin iam.serviceAccountAdmin; do
  gcloud projects add-iam-policy-binding "$PROJECT" \
    --member="$MEMBER" --role="roles/$ROLE"
done
```

Check it directly rather than inferring from role names. There is no `gcloud projects test-iam-permissions` subcommand, so the REST call is the way — an empty `permissions` array back means none are granted:

```
curl -s -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  -d '{"permissions":[
        "logging.sinks.create","logging.sinks.update",
        "logging.exclusions.create","logging.buckets.update",
        "pubsub.topics.setIamPolicy"
      ]}' \
  "https://cloudresourcemanager.googleapis.com/v1/projects/$PROJECT:testIamPermissions"
```

### APIs

`setup.sh` checks these and offers to enable them:

```
gcloud services enable \
  compute.googleapis.com run.googleapis.com pubsub.googleapis.com \
  logging.googleapis.com artifactregistry.googleapis.com iam.googleapis.com \
  --project="$PROJECT"
```

Only project-level IAM is visible locally; org-level constraints surface at apply time. The two most likely to bite are a constraint blocking external HTTP(S) load balancers, and `constraints/iam.disableServiceAccountKeyCreation`.

## The Terraform trap that fails silently

Neither `hashicorp/google` nor `google-beta` (v7) models `loggingHttpRequestHeaders` — `log_config` exposes only `enable`, `optional_fields`, `optional_mode` and `sample_rate`. The module therefore shells out to `gcloud` from a `terraform_data` + `local-exec`.

> **Warning: The allow-list survives Terraform only if it is re-asserted**
>
> The first version keyed the shell-out on a hash of the inputs and paired it with a destroy-time provisioner that removed the list again. On the next full `terraform apply` that destroy provisioner fired during a replacement and wrote back a `logConfig` with no `loggingHttpRequestHeaders`.
>
> Logs kept flowing. They simply arrived with nothing to classify on, and `terraform plan` said “No changes” because Terraform cannot see a field it does not model.
>
> The fix is two resources rather than one. `terraform_data.request_logging` triggers on `timestamp()`, so every apply re-installs the list and any drift heals itself. Removal moved to `terraform_data.request_logging_cleanup`, whose triggers never change, so it fires only on a genuine `terraform destroy`.
>
> **The cost is one unavoidable change in every plan. Do not “clean that up”** — the trade is an invisible failure of the pipeline's whole purpose against a noisy plan.

## Before you trust the numbers

> **Caution: Config changes take minutes to reach the edge**
>
> After re-installing the allow-list, requests sent ~60 seconds later were still logged with no headers. The same probe at ~10 minutes carried all of them. Budget several minutes before trusting a run taken straight after an apply, and treat a sudden run of headerless events after a deploy as propagation rather than breakage.

> **Note: Excluding from _Default really does hide the logs**
>
> Once `exclude_from_default_bucket` takes effect, `gcloud logging read` returns nothing for the load balancer — the entries are routed but never stored. That is the intended saving, and it means **Pub/Sub is the only place to read raw entries**. `make sample` reads storage, so it goes quiet once the exclusion lands; pull the verify subscription instead.

> **Note: Delivery is at-least-once**
>
> A push that fails after the adapter already forwarded some of its events is retried whole. After `max_delivery_attempts` the message parks on the dead-letter topic, where it stays queued and replayable.

> **Note: The adapter image is built locally, not in Cloud Build**
>
> `terraform apply` builds and pushes a container unless you set `adapter_image`. That needs `docker` and an authenticated `gcloud` on the machine running apply. In a default-configured project Cloud Build runs as the default compute service account, which cannot read its own staging bucket — a local build needs only a configured credential helper, and is faster besides.

## Cost

Per 1M requests/month, ~1 KB per log entry:

| Component | Cost |
|---|---|
| Pub/Sub | ~$0.04 |
| Cloud Run adapter | $0 — free tier, scales to zero |
| Cloud Logging routing | $0 — routing to a sink is not charged |
| Cloud Logging storage | $0 with `exclude_from_default_bucket` |
| **Pipeline total** | **under $1/month** |

The load balancer itself is **not** free — roughly $18/month for the forwarding rule plus per-GB processing, whatever the traffic. That does not matter to a customer who already runs one; it matters a lot for a demo rig that exists only to make traffic.

> **Caution: Confirm before quoting**
>
> Check current forwarding-rule and data-processing pricing before quoting any of these numbers to anyone.

## Tearing it down

```
make destroy      # terraform destroy, then check nothing is left standing
make leftovers    # just tell me what is still deployed
```

Do this between runs. The forwarding rule bills at idle whether or not traffic is flowing, and it is the single most expensive thing in the repo.

`terraform destroy` needs `project_id`, which has no default. Keep it in `site/terraform/terraform.tfvars`; the script checks for it up front rather than letting Terraform stall on a prompt. `destroy.sh --sweep` deletes leftovers rather than only reporting them — useful if a destroy half-finishes, since a Cloud Run service or backend service left behind will not show up in any later plan.

## Still unverified

Do not assert these:

- per-header value truncation in `loggingHttpRequestHeaders` — the *count* limit is settled at 10, and nothing in a 40-request run came back truncated
- whether request logging is sampled independently of `sample_rate` under load, the way CloudFront real-time logs turned out to be best-effort. Three runs of 40 delivered 39, 40 and 40 — no evidence of loss, far too small a sample to call it exact
- end-to-end latency beyond the observed ~30–60s

## Code on this page

- [`gcp/ingestion/main.tf`](https://github.com/Obsero-ai/obsero-public/blob/main/gcp/ingestion/main.tf) -- the pipeline: sink, Pub/Sub, Cloud Run adapter
- [`gcp/ingestion/variables.tf`](https://github.com/Obsero-ai/obsero-public/blob/main/gcp/ingestion/variables.tf) -- every module input, with its validation
- [`gcp/ingestion/adapter/index.mjs`](https://github.com/Obsero-ai/obsero-public/blob/main/gcp/ingestion/adapter/index.mjs) -- the Cloud Run adapter
- [`gcp/site/terraform/main.tf`](https://github.com/Obsero-ai/obsero-public/blob/main/gcp/site/terraform/main.tf) -- the demo stack: mock site plus one module block
- [`gcp/PERMISSIONS.md`](https://github.com/Obsero-ai/obsero-public/blob/main/gcp/PERMISSIONS.md) -- the IAM roles, and why Editor is not enough
- [`gcp/NOTES.md`](https://github.com/Obsero-ai/obsero-public/blob/main/gcp/NOTES.md) -- behaviour measured on live deployments