Google Cloud

Streams every request hitting a Cloud Load Balancing backend service to Obsero.

Everything runs in your own project. The adapter strips authorization, cookie and set-cookie before anything leaves it.

#Read this first

#Try it without a GCP project

cd gcp
make test                                        # adapter payload vs the contract
make event TOKEN=<tracking-id> DOMAIN=acme.com    # POST one event, check the ID

Then, to watch it work end to end, the quickstart — or the make targets directly:

make site              # site + LB + header logging only
make sample            # confirm the headers reach Cloud Logging
make deploy            # the rest: sink, Pub/Sub, adapter
make traffic N=40      # mock browser / AI-agent / crawler traffic
make check             # score what actually reached the endpoint
make destroy           # the load balancer bills at idle -- do this

The two-stage start is deliberate: make sample confirms the header allow-list behaves before you build anything downstream of it.

#Wiring it into your own load balancer

module "obsero_ingestion" {
  source = "./ingestion"

  project_id  = var.project_id
  name_prefix = "acme-prod"
  site_token  = var.obsero_site_token

  backend_service_name = google_compute_backend_service.site.name
}

Unlike the AWS module there is no ARN to attach — this one configures logging on the named backend service directly. Better DX, at the cost of mutating a resource you own. It is reversible; terraform destroy takes the allow-list back off.

#The headline win: named request headers on the cheap path

CloudFront forces a choice — pay ~$11/month for real-time logs to get the full header set, or take standard logging and lose everything but User-Agent. GCP has no such trade. BackendServiceLogConfig.loggingHttpRequestHeaders names the headers you want and they arrive through the ordinary, cheap path:

from,                                            # GPTBot, Googlebot
signature-agent, signature-input, signature,     # Web Bot Auth
accept, accept-language,                         # browsers vs */* bots
sec-ch-ua, sec-ch-ua-platform, sec-ch-ua-mobile, # client hints
sec-fetch-dest                                   # presence alone is the signal

It is an allow-list, not “all headers”, which is arguably the better default: the payload carries only what classification needs, so nothing incidental leaves your project.

#What the logged headers actually look like

The allow-list lands under jsonPayload.loggingHttpRequestHeaders, as an array of pairs, with every value base64-encoded:

"loggingHttpRequestHeaders": [
  { "headerKey": "from",            "headerValue": "Ym90QG9wZW5haS5jb20=" },
  { "headerKey": "signature-agent", "headerValue": "Imh0dHBzOi8vY2hhdGdwdC5jb20i" }
]

• The base64 is a feature. It is why Web Bot Auth signatures survive byte-for-byte — quotes, commas and embedded colons included. CloudFront's percent-encoded cs-headers needed careful splitting on %0A before decoding, or a header value could forge an extra header. Nothing here can.
• An absent header is simply missing from the array. There is no empty-string placeholder, so “not sent” and “sent empty” are indistinguishable.
• httpRequest.userAgent and httpRequest.remoteIp are populated regardless, confirming those slots are free.

#Permissions — roles/editor is not enough

roles/editor deliberately excludes log-routing configuration and resource-level setIamPolicy, which are the core of this pipeline. Check these before anyone runs apply.

PROJECT=your-project
MEMBER=user:you@example.com

for ROLE in logging.configWriter pubsub.admin run.admin iam.serviceAccountAdmin; do
  gcloud projects add-iam-policy-binding "$PROJECT" \
    --member="$MEMBER" --role="roles/$ROLE"
done

Check it directly rather than inferring from role names. There is no gcloud projects test-iam-permissions subcommand, so the REST call is the way — an empty permissions array back means none are granted:

curl -s -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  -d '{"permissions":[
        "logging.sinks.create","logging.sinks.update",
        "logging.exclusions.create","logging.buckets.update",
        "pubsub.topics.setIamPolicy"
      ]}' \
  "https://cloudresourcemanager.googleapis.com/v1/projects/$PROJECT:testIamPermissions"

#APIs

setup.sh checks these and offers to enable them:

gcloud services enable \
  compute.googleapis.com run.googleapis.com pubsub.googleapis.com \
  logging.googleapis.com artifactregistry.googleapis.com iam.googleapis.com \
  --project="$PROJECT"

Only project-level IAM is visible locally; org-level constraints surface at apply time. The two most likely to bite are a constraint blocking external HTTP(S) load balancers, and constraints/iam.disableServiceAccountKeyCreation.

#The Terraform trap that fails silently

Neither hashicorp/google nor google-beta (v7) models loggingHttpRequestHeaders — log_config exposes only enable, optional_fields, optional_mode and sample_rate. The module therefore shells out to gcloud from a terraform_data + local-exec.

#Before you trust the numbers

#Cost

Per 1M requests/month, ~1 KB per log entry:

The load balancer itself is not free — roughly $18/month for the forwarding rule plus per-GB processing, whatever the traffic. That does not matter to a customer who already runs one; it matters a lot for a demo rig that exists only to make traffic.

#Tearing it down

make destroy      # terraform destroy, then check nothing is left standing
make leftovers    # just tell me what is still deployed

Do this between runs. The forwarding rule bills at idle whether or not traffic is flowing, and it is the single most expensive thing in the repo.

terraform destroy needs project_id, which has no default. Keep it in site/terraform/terraform.tfvars; the script checks for it up front rather than letting Terraform stall on a prompt. destroy.sh --sweep deletes leftovers rather than only reporting them — useful if a destroy half-finishes, since a Cloud Run service or backend service left behind will not show up in any later plan.

#Still unverified

Do not assert these:

• per-header value truncation in loggingHttpRequestHeaders — the count limit is settled at 10, and nothing in a 40-request run came back truncated
• whether request logging is sampled independently of sample_rate under load, the way CloudFront real-time logs turned out to be best-effort. Three runs of 40 delivered 39, 40 and 40 — no evidence of loss, far too small a sample to call it exact
• end-to-end latency beyond the observed ~30–60s