<!-- Generated by docs/build.py from index.html. Edit the HTML, not this file. -->

# Stream CDN request logs to Obsero

Every HTTP request hitting your CDN becomes one POST to Obsero, so AI agent traffic can be classified on the request headers it actually sent.

Two working end-to-end implementations — AWS CloudFront and Google Cloud — plus a demo site and a traffic harness for each. This is a starting point, not a product. Fork it, keep the parts that match your stack, throw away the rest.

- [Quickstart](quickstart.md) -- One command, four steps: a mock site, the pipeline, connect it, tear it down.
- [The contract](contract.md) -- The URL and the payload. The only thing you have to get right.
- [AWS](aws.md) -- CloudFront → Firehose → Lambda adapter. One real decision to make.
- [Google Cloud](gcp.md) -- LB request logs → Pub/Sub → Cloud Run adapter. Named headers on the cheap path.

## Pick your path

The same repo serves three kinds of reader. Start from the row that sounds like you.

| You are | Do this | Time |
|---|---|---|
| **New to this** — you want to see it work before you touch anything real | Follow [the quickstart checklist](quickstart.md#checklist). Every step says what to run and what you should see. `setup.sh` does the work; nothing needs editing. | ~20 min, including the CDN deploy |
| **Already running a site** on CloudFront — you want your real traffic | [Skip the mock site](quickstart.md#your-own-site): deploy the pipeline, then pick your distribution from the list. Your distribution is never modified. | ~5 min |
| **Experienced with Terraform** — you want the module in your own stack | Read [one module block](aws.md#wiring-it-in) and [what it creates](aws.md#what-it-creates), then the code itself: `aws/ingestion/main.tf` and the adapter, `aws/ingestion/lambda/adapter/index.mjs`. Or skip the infrastructure entirely and call [`sendEvent`](contract.md#obsero-mjs) from wherever you already see requests. | an afternoon to review |
| **A coding agent**, or working with one | Read `AGENTS.md` in the repo, or [`llms.txt`](llms.txt) for these docs as Markdown. [More below](#agents). | — |

Every file path on these pages links to that file on GitHub, and each page ends with the code it describes — so you can check any claim here against the source.

## The shape of it

Obsero cares about exactly two things: the URL you POST to, and the payload. Everything else — which cloud, which log source, which language, whether you batch, whether you use Terraform at all — is your call.

*one event per HTTP request*

```
POST https://analytics-staging.obsero.ai/v1/events

content-type:     application/json
x-obsero-domain:  acme.com          # which site this event belongs to
x-obsero-key:     <tracking ID>     # proves you own it

{
  "path": "/pricing",
  "method": "GET",
  "statusCode": 200,
  "headers": {
    "host": "acme.com",
    "user-agent": "Mozilla/5.0 ...; compatible; ChatGPT-User/1.0; +https://openai.com/bot",
    "accept": "*/*",
    "signature-agent": "\"https://chatgpt.com\""
  }
}
```

That is the whole integration. [Read the contract in full →](contract.md)

## Which one do I want?

| Your setup | Start here | Why |
|---|---|---|
| CloudFront in front of anything | [AWS](aws.md) | Two log sources: cheap (User-Agent only) or full headers for ~$11/mo. |
| Google Cloud load balancer + Cloud CDN | [GCP](gcp.md) | Full headers on the cheap path — no trade to make. |
| A **plain GCS bucket** behind Cloud CDN | [GCP, caveat first](gcp.md#backend-buckets) | Backend *buckets* emit no request logs at all. Your origin has to be a backend *service*. |
| Cloudflare, Fastly, Akamai, a bare origin | [none of them](contract.md#obsero-mjs) | Import `sendEvent` and call it from wherever you already see requests. That is the entire integration. |

## What's in the repo

```
obsero.mjs      the contract: build an event, validate it, POST it
setup.sh        the interactive installer
destroy.sh      teardown, plus a sweep for resources Terraform lost track of
aws/            CloudFront -> Firehose -> Lambda adapter -> Obsero
gcp/            LB request logs -> Pub/Sub -> Cloud Run adapter -> Obsero
docs/           this site
```

Each cloud directory has the same three parts:

```
ingestion/      the Terraform module you would actually ship
site/           a demo site to point it at, so you can see traffic flow
test/           a traffic harness: 19 client personas, then score what arrived
```

Both `ingestion/` modules deploy on their own. `site/` and `test/` exist to prove the thing works and are the first things to delete once it does.

## If you work with a coding agent

There are `AGENTS.md` files at the repo root and in each cloud directory. They carry the contract, the setup order, the failure modes that look like bugs but are not, and the handful of things in the Terraform that must not be “cleaned up” because they fail silently when removed. Point your agent at them before it touches anything.

| File | Read by | What is in it |
|---|---|---|
| `AGENTS.md` | any agent working in the repo | the contract, the menu and its flags, the order to debug in, the rules |
| `CLAUDE.md` | Claude Code, which looks for this name | imports `AGENTS.md`, so there is one source of truth |
| `aws/AGENTS.md`, `gcp/AGENTS.md` | an agent working on one cloud | symptom → cause → fix, and what each step of `setup.sh` applies |
| [`llms.txt`](llms.txt) | an agent reading these docs over the web | an index of every page as Markdown, and the key source files |
| [`llms-full.txt`](llms-full.txt) | an agent that wants it all in one fetch | every page of these docs in one Markdown file |

Every page is also published as Markdown next to its HTML — `quickstart.md` beside `quickstart.html` — and linked from the *View as Markdown* link at the bottom of the page. Agents that do the driving themselves can use the non-interactive forms:

```
./setup.sh aws pipeline --token <tracking-id>
./setup.sh aws connect --dist <distribution-id>
./setup.sh aws status
```

The rule in `AGENTS.md` that matters most: an agent must never run an apply or a destroy without the developer's go-ahead on that specific command. These stacks cost real money.

## Code on this page

- [`README.md`](https://github.com/Obsero-ai/obsero-public/blob/main/README.md) -- the same overview, as it reads on GitHub
- [`AGENTS.md`](https://github.com/Obsero-ai/obsero-public/blob/main/AGENTS.md) -- instructions for a coding agent working in the repo
- [`obsero.mjs`](https://github.com/Obsero-ai/obsero-public/blob/main/obsero.mjs) -- the contract as code: build, validate and POST an event


---

<!-- Generated by docs/build.py from quickstart.html. Edit the HTML, not this file. -->

# Quickstart

One command. Deploy a mock site, build a real ingestion pipeline, connect the two, watch classified traffic arrive, then tear it all down. Or skip the mock site and connect a site you already run.

> **Warning: This costs real money**
>
> The demo rigs bill while they are up. The GCP forwarding rule is about **$18/month whether or not any traffic hits it**. Run step 4 when you are done — not tomorrow.

## Before you start

| You need | Both | AWS | GCP |
|---|---|---|---|
| Terraform ≥ 1.5 | ✓ |  |  |
| Node ≥ 18 | ✓ |  |  |
| Authenticated CLI |  | `aws configure` | `gcloud auth login` |
| Docker |  | — | ✓ images are built locally |
| A tracking ID from Obsero | ✓ |  |  |

`setup.sh` checks all of this for you and tells you exactly what is missing before it creates anything. On GCP it also checks the required APIs and offers to enable them — but **not** the IAM roles, which are the thing most likely to bite. [Check those first →](gcp.md#permissions)

## Your checklist

Everything you do, in order, for AWS. Each item says what to run and what you should see; if you see something else, stop there — the item after it will not work either. [The steps in detail](#the-steps) explains what each one builds.

1. ### Install the tools

   ```
   terraform -version     # v1.5 or newer
   node --version         # v18 or newer
   aws --version          # any v2
   ```

   Missing one? `setup.sh` names it and links the installer, so you can also just run it and see.
2. ### Log the AWS CLI in

   ```
   aws sts get-caller-identity
   ```

   **You should see** your account number and an ARN. An error means the CLI has no credentials: run `aws configure`, or `export AWS_PROFILE=<name>` if you use named profiles. Everything below runs as this identity, in this account.
3. ### Get the code

   ```
   git clone https://github.com/Obsero-ai/obsero-public.git
   cd obsero-public
   ```
4. ### Check your tracking ID

   The site token Obsero issued for your domain. It is sent as `x-obsero-key` on every event and is what proves the domain is yours. Check it before building anything — a rejected ID makes the pipeline forward into nothing, silently:

   ```
   node obsero.mjs --token <tracking-id> --domain <your-domain>
   ```

   **You should see** the event accepted. Then open the menu — everything from here is a number typed into it:

   ```
   ./setup.sh aws
   ```
5. ### Press `1` — deploy the mock site

   Takes 3–5 minutes. **You should see** `Site is up` and a `https://….cloudfront.net` URL. Open it: a small site with a few pages. Skip this item if you are [connecting your own site](#your-own-site).
6. ### Press `2` — deploy the pipeline

   Paste your tracking ID and press Enter; each character shows as `*`, then it confirms the first and last four. Answer `y` to the test event. **You should see** `Obsero accepted it`, then, 2–3 minutes later, `Pipeline is live` with `connected: nothing yet`. That is expected — the next item connects it.
7. ### Press `3` — connect

   A numbered list of every CloudFront distribution in the account. Type the number next to `the mock site`, press Enter to see it ticked, press Enter on an empty line, then `y`. **You should see** `Connected` and the distribution ID.
8. ### Press `5` — send traffic and score it

   Sends requests as 19 client personas, waits about 90 seconds for delivery, then scores how many reached Obsero with their headers intact. **You should see** every request correlated and no header loss. Some missing right after connecting is the 60–90s latency floor: run it again.
9. ### Press `4` when you are finished

   Type `destroy` to confirm. It removes everything `setup.sh` created and then checks nothing is left billing. Distributions it did not create are only disconnected, never deleted.

Press `6` at any point to see what is deployed and what is connected.

## You're done when

- `./setup.sh aws status` shows the pipeline **deployed** and at least one distribution **connected**.
- `make -C aws logs` shows the adapter forwarding events within 90 seconds of a request to a connected site.
- The traffic score correlates every request — the measured baseline is 78 of 78 on AWS, with zero header loss.
- Events show up in Obsero under your domain, with AI agents and crawlers told apart from browsers.

Stuck on one of these? [Troubleshooting](troubleshooting.md#order) has the order to check things in.

## Use your own site instead of the mock

The mock site is only there so you have traffic to look at. If you already run a site on CloudFront, connect that instead — the pipeline is the same either way.

```
./setup.sh aws   # then:
2                # the pipeline, with your tracking ID
3                # tick your distribution, listed by domain
```

Or, without the menu:

```
./setup.sh aws pipeline --token <tracking-id>
./setup.sh aws connect --dist <distribution-id>
```

> **Note: What happens to your distribution: nothing**
>
> Connecting never edits the distribution's configuration. It adds a CloudFront *standard logging v2* delivery that copies each request's log record into the pipeline's Firehose. Traffic, caching and latency are untouched. If the distribution already uses standard logging v2 — to S3, say — that keeps working: its existing logging source is reused, and left in place when you disconnect.

- **Which headers arrive.** Standard logging carries User-Agent, Referer, Host and X-Forwarded-For — enough to tell most agents apart, not enough to *verify* Web Bot Auth signatures. [The one real decision](aws.md#log-source) explains the trade.
- **What it costs.** No idle cost; Firehose bills per GB, which is cents at typical volumes. [Cost →](aws.md#cost)
- **Permissions.** Your AWS identity needs to list CloudFront distributions and create CloudWatch Logs deliveries, in `us-east-1` — where CloudFront logging always lives.
- **Stopping.** Run `3` again and untick it, or `./setup.sh aws connect --dist none`. Destroying the pipeline disconnects everything too.

## Two checks that cost nothing

Both of these pass or fail in seconds and rule out most of what goes wrong later. Run them before you create any infrastructure.

```
cd aws            # or gcp
make test         # adapter payload vs the contract — no cloud account needed
make event TOKEN=<tracking-id> DOMAIN=acme.com
```

`make test` feeds sample log lines through the real adapter parser and asserts the result against `validateEvent`. `make event` POSTs one hand-made event, so you know your tracking ID is accepted before anything depends on it.

## The steps in detail

```
./setup.sh
```

It asks which cloud, then shows this:

```
Obsero ingestion -- aws

  1) Deploy the mock site          a site worth collecting logs from
  2) Deploy the ingestion pipeline asks for your tracking ID
  3) Connect                       pick distributions already in your account
  4) Destroy everything            the rig bills while it is up

  5) Send mock traffic and score what arrived
  6) Status -- what is deployed right now
  7) Switch cloud
  q) Quit
```

1. ### Deploy the mock site

   A static site with real pages, behind a real CDN, so there is traffic worth logging. On AWS that is a private S3 bucket behind CloudFront; on GCP it is the same pages served from Cloud Run behind a global external Application Load Balancer with Cloud CDN in front.

   Nothing is sent to Obsero at this point, and no tracking ID is needed. The step prints the site URL when it finishes — open it.

   ```
   ==> Site is up
      https://d1234abcd.cloudfront.net

      Open it. Nothing is being sent to Obsero yet -- step 2 builds the pipeline,
      step 3 connects this site (or any other distribution) to it.
   ```

   Budget 3–5 minutes on AWS (CloudFront is the slow part) and 5–8 on GCP, which also builds a container image on the way.
2. ### Deploy the ingestion pipeline

   This is the step that needs your tracking ID — the site token Obsero issued you, sent as the `x-obsero-key` header on every forwarded event. Each character shows as `*` so you can see a paste landed, and the first and last four are echoed back to check. It is saved to `<cloud>/site/terraform/terraform.tfvars`, which is gitignored and `chmod 600`, because `terraform destroy` and every later apply need it too.

   ```
   ==> Your Obsero tracking ID
      The site token Obsero issued you. It is sent as the x-obsero-key header
      on every forwarded event, and is what proves the domain is yours.

      Tracking ID (paste, then Enter): ********************************
      got 32 characters: wuJ7...oF_E
    v using wuJ7...oF_E

   ==> Collecting from
      nothing yet -- the pipeline is built idle, and step 3 connects it
      to any CloudFront distribution in your account, the mock site included

      Send one test event first, to check the ID is accepted? [y/N]
   ```

   Say yes. A rejected tracking ID means the pipeline forwards into nothing, silently — better to find out in one second than after a ten-minute apply. Then the pipeline goes up: on AWS, CloudFront standard logs into Firehose into an adapter Lambda; on GCP, load balancer request logs through a Log Router sink into Pub/Sub into a Cloud Run adapter.

   On AWS the pipeline does not need the mock site and is built collecting from nothing; step 3 decides where it collects from. On GCP it attaches to the mock site's backend service here, and builds the site too if step 1 has not run.

   > **Note: Give it a minute before you judge it**
   >
   > **AWS:** first events land in 60–90 seconds. CloudFront flushes continuously but Firehose buffers for at least 60s, and that is a floor, not a setting.
   >
   > **GCP:** the header allow-list takes several minutes to reach the edge. Events that arrive headerless right after an apply are propagation, not breakage.
3. ### Connect (AWS)

   Lists every CloudFront distribution in your account — the mock site and anything you already run — and lets you tick the ones the pipeline collects from. Untick one to disconnect it.

   ```
   ==> CloudFront distributions in this account

      1) [x] E1A2B3C4D5E6F7  d1234abcd.cloudfront.net
                the mock site
      2) [ ] E9Z8Y7X6W5V4U3  acme.com,www.acme.com
                reuses its existing logging source "CreatedByCloudFront-E9Z8Y7X6W5V4U3"

      numbers toggle (e.g. "1 3"), a = all, n = none, Enter = done, q = cancel
   Connect: 2
   ```

   Your distributions are never modified. Connecting adds a CloudFront standard logging v2 delivery from the distribution into the pipeline's Firehose. A distribution can have only one standard logging v2 source, so if it already has one — the CloudFront console creates it the first time anyone turns that logging on — it is reused and left in place on disconnect.

   The picks are saved as `connected_distribution_arns` in `terraform.tfvars`, so every later apply keeps them.
4. ### Destroy everything

   Terraform does the work; the wrapper handles what it cannot — emptying S3 buckets whose contents Terraform does not own, passing each stack the credentials it needs, and then **checking the cloud afterwards** for anything still standing. A lost state file or a half-finished destroy leaves resources that no `terraform plan` will ever mention.

   ```
   ./destroy.sh --check      # just tell me what is still deployed
   ./destroy.sh --dry-run    # show the plan, touch nothing
   ./destroy.sh aws          # one cloud
   ./destroy.sh --sweep      # delete leftovers too, not just report them
   ```

   CloudFront is the one exception: a distribution has to be disabled, left to propagate, and deleted minutes later, so the script reports it rather than pretending it can be done in one pass.

## Watch it work

Option 5 sends a reproducible mix of traffic as 19 client personas — each carrying the headers that client really sends — waits out the delivery latency, then scores what actually arrived.

```
./setup.sh aws traffic -n 40
```

Every request is tagged `?mt=<runId>-<seq>`. The scorer reads the records back, replays them through the **real** adapter parser, correlates each tag, and diffs headers sent against headers received. Measured baselines: 78/78 correlated on AWS, 40/40 on GCP, zero header loss on either.

| kind | personas |
|---|---|
| `ai-agent` | ChatGPT-User, Claude-User, Perplexity-User, OAI-SearchBot, DuckAssistBot |
| `ai-crawler` | GPTBot, ClaudeBot, PerplexityBot, Meta-ExternalAgent, Bytespider, CCBot, Amazonbot |
| `human` | Chrome/Windows, Safari/iPhone, Firefox/macOS, Chrome/Android |
| `search-bot` | Googlebot, Bingbot, Applebot |

```
cd aws
make traffic N=40                    # weighted mix
make traffic N=20 KIND=ai-agent      # one class
make traffic N=10 ONLY=gptbot SEED=7 # specific, reproducible
make personas                        # list them
```

## Without the menu

Every menu item has a flag form, for CI or for driving it from a script:

```
./setup.sh aws site
./setup.sh aws pipeline --token <tracking-id>
./setup.sh aws connect --dist E1A2B3C4D5E6F7   # exactly these; --dist none clears
./setup.sh aws connect --dist +E9Z8Y7X6W5V4U3  # add one to what is connected
./setup.sh aws traffic -n 40
./setup.sh aws status
./setup.sh aws destroy --yes
```

Or skip `setup.sh` entirely and use the make targets directly — `make` with no argument in either cloud directory lists them all. `make deploy` is steps 1–3 for the mock site in one go; `make e2e` is deploy, traffic, wait, score.

## What setup.sh is actually doing

Worth knowing, because it explains the one thing that looks odd: the site, the pipeline and its connections are **one Terraform stack**, not three. Step 1 is a targeted apply of the site resources only. On AWS, step 2 is a targeted apply of the `module.ingestion` block, which does not reference the site, and step 3 re-applies that same module with a new `connected_distribution_arns` list. On GCP, step 2 is the plain untargeted apply.

Because both stacks declare `obsero_site_token` with no default, step 1 writes the placeholder `set-in-step-2` into `terraform.tfvars` so it can plan at all. Step 2 refuses to apply while that is still the value.

## Code on this page

- [`setup.sh`](https://github.com/Obsero-ai/obsero-public/blob/main/setup.sh) -- the interactive installer this page walks through
- [`destroy.sh`](https://github.com/Obsero-ai/obsero-public/blob/main/destroy.sh) -- teardown, and the sweep for leftovers
- [`aws/Makefile`](https://github.com/Obsero-ai/obsero-public/blob/main/aws/Makefile) -- every AWS task, one make target each
- [`gcp/Makefile`](https://github.com/Obsero-ai/obsero-public/blob/main/gcp/Makefile) -- every GCP task, one make target each


---

<!-- Generated by docs/build.py from contract.html. Edit the HTML, not this file. -->

# The contract

The URL you POST to, and the payload. Everything else is yours to decide.

Which cloud, which log source, which language, whether you buffer or send per request, whether you use Terraform at all — none of that is part of the contract. This page is.

## The request

*one event per HTTP request*

```
POST https://analytics-staging.obsero.ai/v1/events

content-type:     application/json
x-obsero-domain:  acme.com          # which site this event belongs to
x-obsero-key:     <tracking ID>     # proves you own it

{
  "path": "/pricing",
  "method": "GET",
  "statusCode": 200,
  "headers": {
    "host": "acme.com",
    "user-agent": "Mozilla/5.0 ...; compatible; ChatGPT-User/1.0; +https://openai.com/bot",
    "accept": "*/*",
    "signature-agent": "\"https://chatgpt.com\"",
    "signature-input": "sig1=(\"@authority\" \"signature-agent\");created=...",
    "signature": "sig1=:...:"
  }
}
```

| field | rules |
|---|---|
| `path` | Path **only**. `/pricing`, never `/pricing?ref=x` and never a full URL. Leaving the query string on makes every distinct URL its own page in the analytics — the most common mistake by far. |
| `method` | Uppercase. `"GET"`. |
| `statusCode` | A number, not a string. `200`. |
| `headers` | Lowercase names → verbatim values. This is what classification runs on, so send as many as you can get. |

## Headers worth carrying

Roughly in order of value:

| header | what it buys you |
|---|---|
| `user-agent` | everything starts here |
| `from` | GPTBot, Googlebot |
| `signature-agent`, `signature-input`, `signature` | Web Bot Auth — **proves** an agent is who it claims to be, rather than taking a User-Agent at face value |
| `accept`, `accept-language` | browsers vs `*/*` bots |
| `sec-ch-ua`, `sec-ch-ua-platform`, `sec-ch-ua-mobile` | client hints — browsers only |
| `sec-fetch-dest`, `sec-fetch-mode`, `sec-fetch-site` | presence alone is the signal |

> **Warning: Never forward these**
>
> `authorization`, `cookie`, `set-cookie`. Both adapters strip them before anything leaves your account. Keep that behaviour if you rewrite one.

## obsero.mjs — the whole contract in one file

Dependency-free, about 150 lines: `buildEvent`, `sendEvent`, `validateEvent`. Run it to send a test event and confirm your tracking ID works before you deploy anything:

```
node obsero.mjs --token <tracking-id> --domain acme.com
```

Then, in your own code, this is genuinely all it takes:

*the entire integration*

```
import { buildEvent, sendEvent } from "./obsero.mjs";

await sendEvent(
  buildEvent({
    path: req.url,
    method: req.method,
    statusCode: res.statusCode,
    headers: req.headers,
  }),
  { token: process.env.OBSERO_SITE_TOKEN, domain: "acme.com" },
);
```

If you can call that from your edge middleware, your origin server or your log processor, you are done. You do not need anything else in this repo — and if you are on Cloudflare, Fastly, Akamai or a bare origin, this *is* the integration. Do not port a cloud directory you do not need.

`buildEvent` does the two things that are easy to get wrong for you: it strips the query string from `path` (and tolerates being handed an absolute URL), and it lowercases, trims and filters the headers.

## Check it before you ship it

`validateEvent` is the machine-checkable form of everything on this page. It returns a list of problems, empty when the payload is good:

```
cd aws            # or gcp
make test
```

That runs the **real** adapter parser over sample log lines and checks each resulting payload against `validateEvent`. No cloud account needed. If it passes, Obsero will accept what you produce.

## Adapting an adapter

Both adapters have the same three-part shape, and only one part is the contract:

```
1. THE CONTRACT     build the payload and POST it     <- keep this
2. LOG PARSING      log line/entry -> event           <- yours to rewrite
3. TRANSPORT        unwrap the batch, ack it          <- yours to rewrite
```

- `aws/ingestion/lambda/adapter/index.mjs`
- `gcp/ingestion/adapter/index.mjs`

Change parts 2 and 3 to match your log source, leave part 1 alone, then run `make test`. Fixtures live in each cloud's `test/adapter.test.mjs` — add one for any new log shape rather than testing against live traffic.

## Getting a tracking ID

Ask Obsero for one, then keep it out of the repo. Both `site/terraform/` directories read it from a gitignored `terraform.tfvars`:

```
obsero_site_token = "..."
```

`setup.sh` writes that file for you, `chmod 600`. The deployed adapters read `x-obsero-domain` off each request's own Host header, so there is no domain to configure — only the token.

## Code on this page

- [`obsero.mjs`](https://github.com/Obsero-ai/obsero-public/blob/main/obsero.mjs) -- buildEvent, validateEvent, sendEvent
- [`aws/test/adapter.test.mjs`](https://github.com/Obsero-ai/obsero-public/blob/main/aws/test/adapter.test.mjs) -- the AWS adapter checked against the contract
- [`gcp/test/adapter.test.mjs`](https://github.com/Obsero-ai/obsero-public/blob/main/gcp/test/adapter.test.mjs) -- the GCP adapter checked against the contract


---

<!-- Generated by docs/build.py from aws.html. Edit the HTML, not this file. -->

# AWS — CloudFront

Streams every request hitting a CloudFront distribution to Obsero.

```
viewer -> CloudFront -> Firehose -> adapter Lambda -> POST /v1/events
                                      \-> S3 backup bucket (undeliverable batches)
```

Everything runs in your own AWS account. The adapter strips `authorization`, `cookie` and `set-cookie` before anything leaves it.

## Try it without an AWS account

```
cd aws
make test                                        # adapter payload vs the contract
make event TOKEN=<tracking-id> DOMAIN=acme.com    # POST one event, check the ID
```

Then, to watch it work end to end, [the quickstart](quickstart.md) — or the make targets directly:

```
make deploy            # demo site + pipeline, connected to each other
make traffic N=40      # mock browser / AI-agent / crawler traffic
make check             # score what actually reached the endpoint
make logs              # follow the adapter forwarding, live
make destroy           # tear it all down again
```

## Wiring it into your own distribution

One module block. The module **never touches your distribution** — in standard mode it only needs the distribution's ARN, and in realtime mode it hands back an ARN for you to attach — so it works whether or not Terraform manages your CloudFront.

*standard mode — the whole integration*

```
module "obsero_ingestion" {
  source = "./ingestion"

  name_prefix = "acme-prod"
  site_token  = var.obsero_site_token

  # CloudFront delivers straight into Firehose. No idle cost.
  log_source        = "standard"
  distribution_arns = [
    aws_cloudfront_distribution.site.arn,
    "arn:aws:cloudfront::123456789012:distribution/E2ABC123", # one Terraform does not manage
  ]
}
```

`distribution_arns` can be empty: the pipeline is then built idle, costing nothing, until you add one. If a distribution already has a standard logging v2 source — the CloudFront console creates one the first time anyone enables that logging — also pass it in `existing_delivery_sources`, because a distribution can only have one:

```
existing_delivery_sources = {
  "arn:aws:cloudfront::123456789012:distribution/E2ABC123" = "CreatedByCloudFront-E2ABC123"
}
```

For `realtime`, drop `distribution_arns` and attach the exported ARN to each cache behaviour you want logged instead:

```
default_cache_behavior {
  # ...
  realtime_log_config_arn = module.obsero_ingestion.realtime_log_config_arn
}
```

Behaviours without the ARN emit nothing, so you can roll it out one path pattern at a time. See `ingestion/examples/` for both shapes and `ingestion/README.md` for every input.

## Connecting distributions you already run

With `setup.sh`, this is step 3: it lists every distribution in the account, detects existing logging sources for you, and writes both variables above into `aws/site/terraform/terraform.tfvars`. See [using your own site](quickstart.md#your-own-site) for the walkthrough.

```
./setup.sh aws connect                        # pick from a list
./setup.sh aws connect --dist E2ABC123,E3DEF456 # exactly these
./setup.sh aws connect --dist +E2ABC123         # add one
./setup.sh aws connect --dist none              # disconnect all
```

To check what is connected from the AWS side, independent of Terraform state:

```
aws logs describe-deliveries --region us-east-1 \
  --query 'deliveries[].[deliverySourceName,deliveryDestinationArn]' --output table
aws logs describe-delivery-sources --region us-east-1 \
  --query 'deliverySources[].[name,resourceArns[0]]' --output table
```

Each connected distribution has one delivery whose destination ends in `<name_prefix>-firehose`. Disconnecting deletes that delivery, and the source too if the module created it; a source that was there first is left alone.

## What the module creates

For reviewing before you apply it in a real account. Everything is prefixed with `name_prefix` and tagged `ManagedBy = obsero-ingestion`. Source: `aws/ingestion/main.tf` and `aws/ingestion/lambda.tf`.

| Resource | How many | Why it exists |
|---|---|---|
| `aws_kinesis_firehose_delivery_stream.this` | 1 | Buffers log records and POSTs batches to the adapter. Tagged `LogDeliveryEnabled = true` in standard mode — [without it nothing is delivered](#gotchas). |
| `aws_lambda_function.adapter` + `aws_lambda_function_url.adapter` | 1 | Turns each log record into one Obsero event and returns Firehose's ack. [Why a Lambda at all](#why-lambda). |
| `aws_s3_bucket.backup` | 1 | Batches the endpoint never accepted, after `retry_duration`. Expires after `backup_retention_days`. |
| `aws_cloudwatch_log_delivery_destination.this` | 1 (standard) | Points CloudFront standard logging at the Firehose. Shared by every connected distribution; free while idle. |
| `aws_cloudwatch_log_delivery_source.this` | 1 per connected distribution without one | Declares the distribution's access logs as a source. Skipped for distributions in `existing_delivery_sources`. |
| `aws_cloudwatch_log_delivery.this` | 1 per connected distribution | The connection itself: source → destination, with the fields in `standard_log_fields`. |
| IAM roles, CloudWatch log groups | 2 + 2 | Firehose's role (backup bucket, logs, Kinesis in realtime) and the adapter's execution role; one log group each, kept `log_retention_days`. |
| `aws_kinesis_stream.this`, `aws_cloudfront_realtime_log_config.this` | realtime only | The ~$11/month shard and the config you attach to cache behaviours. |

Bad inputs fail at plan time, not as empty events: `terraform_data.validate_config` checks the log field lists and that every `existing_delivery_sources` key is also connected.

## Which log source? The one real decision

CloudFront makes you choose.

|  | `standard` default | `realtime` |
|---|---|---|
| Idle cost | **none** | ~$11/mo per Kinesis shard |
| At 1M req/mo | ~$0.15 | ~$11 |
| Latency | ~1–2 min | seconds |
| Headers | User-Agent, Referer, Host, X-Forwarded-For | **full viewer header set** |
| Web Bot Auth signatures | no | **yes** |
| Client hints (`sec-ch-ua`) | no | **yes** |

Start on `standard`. Move to `realtime` when you want to *verify* an agent rather than take its User-Agent at face value: standard logging v2 has no `cs-headers` field, so `signature-agent` / `signature-input` / `signature` never arrive at all. That is not a tuning problem — it is a different log source.

[GCP has no such trade](gcp.md#headers): it carries named headers on the cheap path.

## Before you trust the numbers

Four behaviours that are easy to mistake for bugs. All measured on a live deployment, all written up in `aws/NOTES.md`.

> **Caution: Real-time logs are best-effort**
>
> CloudFront delivers real-time logs on a best-effort basis; some requests are never written to Kinesis at all. Measured across five runs (188 requests) at `sampling_rate = 100`: four delivered 100%, one delivered 20 of 30. The lost records correlated with nothing — not status code, path, client or latency — and Kinesis `IncomingRecords` confirmed they were never sent rather than lost downstream.
>
> Treat the counts as a high-fidelity sample, not an exact ledger. Do not reconcile them against origin logs or bill directly on them.

> **Note: Delivery is at-least-once**
>
> If any event in a Firehose batch fails to forward, the adapter fails the whole batch so Firehose retries — which duplicates the events that already landed. After `retry_duration` the batch goes to the backup bucket rather than being dropped.

> **Note: Latency floor: 60–90s**
>
> CloudFront flushes real-time logs continuously, but Firehose buffers for at least 60s. That is a floor, not a tuning knob.

> **Warning: The LogDeliveryEnabled tag**
>
> Standard logging v2 writes through the `AWSServiceRoleForLogDelivery` service-linked role, whose policy allows `firehose:PutRecordBatch` **only on streams tagged `LogDeliveryEnabled = true`**. Without the tag CloudFront delivers **nothing**, and there is no error anywhere — the delivery reports healthy, the Firehose reports healthy, and `IncomingRecords` simply stays at zero.
>
> The module applies the tag automatically. Worth knowing if you ever hand-build this.

### Two CloudFront parsing behaviours (realtime only)

Both are handled by the adapter. They are documented because they bite anyone reading the raw Kinesis stream themselves. Standard mode is self-describing JSON, so neither applies there.

1. **Field order is not your field order.** CloudFront emits real-time log fields in its own canonical order, ignoring the order in the log config. Records are tab-separated with no header row, so a positional parser that trusts the config order reads every column shifted. The module sorts `log_fields` into canonical order and hands that same list to the adapter; `log_fields_in_order` exports it.
2. **`cs-uri-stem` includes the query string** in real-time logs, unlike standard access logs where it is path-only. The adapter strips it, or every distinct query string would register as its own path in the analytics.

## Why the adapter Lambda exists at all

Firehose's HTTP endpoint destination cannot post to `/v1/events` directly:

- it wraps records in `{requestId, timestamp, records:[{data}]}`
- it sends the key as `X-Amz-Firehose-Access-Key`, and cannot send arbitrary headers such as `x-obsero-domain` / `x-obsero-key`
- it requires a specific JSON ack back, or treats the delivery as failed

The adapter unwraps the envelope, converts each log line into one Obsero event, strips excluded headers, and returns the ack. Its Function URL is unauthenticated at the AWS layer — Firehose cannot SigV4-sign to a Function URL — so the shared Firehose access key is what the handler checks. The URL is not secret, but is useless without that key.

### Concurrency footprint

The adapter uses **one concurrent Lambda execution per Kinesis shard**. Firehose reads each shard serially and invokes once per buffer window, so at the default `kinesis_shard_count = 1` the steady state is exactly 1 concurrent execution, ~1s per invocation, one invocation per minute. Measured, not estimated.

That matters on accounts with a low concurrency ceiling. New AWS accounts are capped at **10 concurrent executions account-wide**, and at that ceiling `reserved_concurrent_executions` cannot be set at all — AWS requires unreserved concurrency to stay at or above 10, so any reservation is rejected. The adapter therefore shares an unprotected pool with every other function in the account.

If another function saturates the pool, the Function URL returns 429, Firehose treats it as a failed delivery, retries for `retry_duration`, and then writes the batch to the backup bucket. Nothing is lost, but those events do not reach Obsero until the backup is replayed. Watch `Throttles` if you share a small pool.

## Cost

Lambda, CloudFront and S3 usage at demo volume sit inside the AWS free tier. Kinesis Data Streams does not have a free tier — one provisioned shard is about **$11/month** ($0.015/shard-hour) and is `realtime` mode's floor cost, since CloudFront real-time logs can only be delivered to Kinesis. Firehose adds ~$0.029/GB ingested, which is cents at typical volumes.

In `standard` mode there is no Kinesis stream and therefore no idle cost at all; Firehose only reaches $11/month at roughly 380M requests.

## Tearing it down

```
make destroy      # terraform destroy, then check nothing is left standing
make leftovers    # just tell me what is still deployed
```

Both call `destroy.sh`, which empties the S3 buckets first — Terraform refuses to delete a bucket holding objects it did not create, and that is precisely what the backup bucket collects.

The module itself keeps `backup_force_destroy = false`, so a customer's `terraform destroy` will *stop* rather than silently discard events that never reached Obsero. The demo site sets it to `true`, because nobody is going to replay those.

## Code on this page

- [`aws/ingestion/main.tf`](https://github.com/Obsero-ai/obsero-public/blob/main/aws/ingestion/main.tf) -- the pipeline: Firehose, log deliveries, backup bucket
- [`aws/ingestion/variables.tf`](https://github.com/Obsero-ai/obsero-public/blob/main/aws/ingestion/variables.tf) -- every module input, with its validation
- [`aws/ingestion/lambda/adapter/index.mjs`](https://github.com/Obsero-ai/obsero-public/blob/main/aws/ingestion/lambda/adapter/index.mjs) -- the adapter Lambda
- [`aws/site/terraform/main.tf`](https://github.com/Obsero-ai/obsero-public/blob/main/aws/site/terraform/main.tf) -- the demo stack: mock site plus one module block
- [`aws/NOTES.md`](https://github.com/Obsero-ai/obsero-public/blob/main/aws/NOTES.md) -- behaviour measured on live deployments
- [`aws/AGENTS.md`](https://github.com/Obsero-ai/obsero-public/blob/main/aws/AGENTS.md) -- the AWS playbook for coding agents


---

<!-- Generated by docs/build.py from gcp.html. Edit the HTML, not this file. -->

# Google Cloud

Streams every request hitting a Cloud Load Balancing backend service to Obsero.

```
viewer -> Cloud CDN + external Application LB -> Cloud Logging
       -> Log Router sink -> Pub/Sub -> Cloud Run adapter -> POST /v1/events
                          \-> dead-letter topic (replayable)
```

Everything runs in your own project. The adapter strips `authorization`, `cookie` and `set-cookie` before anything leaves it.

## Read this first

> **Warning: Backend buckets produce no request logs**
>
> `BackendBucket` has **no `logConfig` field at all** in the Compute v1 API, so a static site served straight from Cloud Storage through Cloud CDN emits nothing to ship. Your origin has to be a backend **service**.
>
> Apps on Cloud Run, GKE or a MIG already are one and are unaffected. A pure GCS origin has to change before it can be onboarded — which is why the demo site here serves its static pages from Cloud Run rather than a bucket.

## Try it without a GCP project

```
cd gcp
make test                                        # adapter payload vs the contract
make event TOKEN=<tracking-id> DOMAIN=acme.com    # POST one event, check the ID
```

Then, to watch it work end to end, [the quickstart](quickstart.md) — or the make targets directly:

```
make site              # site + LB + header logging only
make sample            # confirm the headers reach Cloud Logging
make deploy            # the rest: sink, Pub/Sub, adapter
make traffic N=40      # mock browser / AI-agent / crawler traffic
make check             # score what actually reached the endpoint
make destroy           # the load balancer bills at idle -- do this
```

The two-stage start is deliberate: `make sample` confirms the header allow-list behaves before you build anything downstream of it.

## Wiring it into your own load balancer

*the whole integration*

```
module "obsero_ingestion" {
  source = "./ingestion"

  project_id  = var.project_id
  name_prefix = "acme-prod"
  site_token  = var.obsero_site_token

  backend_service_name = google_compute_backend_service.site.name
}
```

Unlike the AWS module there is no ARN to attach — this one configures logging on the named backend service directly. Better DX, at the cost of mutating a resource you own. It is reversible; `terraform destroy` takes the allow-list back off.

## The headline win: named request headers on the cheap path

CloudFront forces a choice — pay ~$11/month for real-time logs to get the full header set, or take standard logging and lose everything but User-Agent. **GCP has no such trade.** `BackendServiceLogConfig.loggingHttpRequestHeaders` names the headers you want and they arrive through the ordinary, cheap path:

```
from,                                            # GPTBot, Googlebot
signature-agent, signature-input, signature,     # Web Bot Auth
accept, accept-language,                         # browsers vs */* bots
sec-ch-ua, sec-ch-ua-platform, sec-ch-ua-mobile, # client hints
sec-fetch-dest                                   # presence alone is the signal
```

> **Caution: The API caps the list at 10**
>
> An eleventh header is rejected outright. Ten is enough only because `user-agent`, `referer` and the client IP arrive as first-class `httpRequest` fields whether or not you name them — spending slots on those would waste them. The adapter merges both sources.
>
> So the question is never “can I add one more?” It is which one you are dropping. `sec-fetch-mode` and `sec-fetch-site` were the two cut from the default list.

It is an allow-list, not “all headers”, which is arguably the better default: the payload carries only what classification needs, so nothing incidental leaves your project.

### What the logged headers actually look like

The allow-list lands under `jsonPayload.loggingHttpRequestHeaders`, as an **array of pairs**, with every value **base64-encoded**:

```
"loggingHttpRequestHeaders": [
  { "headerKey": "from",            "headerValue": "Ym90QG9wZW5haS5jb20=" },
  { "headerKey": "signature-agent", "headerValue": "Imh0dHBzOi8vY2hhdGdwdC5jb20i" }
]
```

- **The base64 is a feature.** It is why Web Bot Auth signatures survive byte-for-byte — quotes, commas and embedded colons included. CloudFront's percent-encoded `cs-headers` needed careful splitting on `%0A` *before* decoding, or a header value could forge an extra header. Nothing here can.
- **An absent header is simply missing from the array.** There is no empty-string placeholder, so “not sent” and “sent empty” are indistinguishable.
- **`httpRequest.userAgent` and `httpRequest.remoteIp` are populated regardless**, confirming those slots are free.

## Permissions — roles/editor is not enough

`roles/editor` deliberately excludes log-routing configuration and resource-level `setIamPolicy`, which are the core of this pipeline. Check these before anyone runs apply.

| Role | For |
|---|---|
| `roles/editor` | load balancer, CDN, backend services, NEGs, Pub/Sub topics and subscriptions, Artifact Registry |
| `roles/logging.configWriter` | the Log Router sink, and the `_Default` exclusion that stops you paying to store logs you already routed. **Not implied by Editor** — apply fails at the sink. |
| `roles/pubsub.admin` | `pubsub.topics.setIamPolicy`, to grant the sink's writer identity publisher on the topic. **`roles/pubsub.editor` does not cover it** — without it the sink is created but silently publishes nothing. |
| `roles/run.admin` | deploying the adapter and setting its invoker policy |
| `roles/iam.serviceAccountAdmin` | the two service accounts the module creates |

```
PROJECT=your-project
MEMBER=user:you@example.com

for ROLE in logging.configWriter pubsub.admin run.admin iam.serviceAccountAdmin; do
  gcloud projects add-iam-policy-binding "$PROJECT" \
    --member="$MEMBER" --role="roles/$ROLE"
done
```

Check it directly rather than inferring from role names. There is no `gcloud projects test-iam-permissions` subcommand, so the REST call is the way — an empty `permissions` array back means none are granted:

```
curl -s -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  -d '{"permissions":[
        "logging.sinks.create","logging.sinks.update",
        "logging.exclusions.create","logging.buckets.update",
        "pubsub.topics.setIamPolicy"
      ]}' \
  "https://cloudresourcemanager.googleapis.com/v1/projects/$PROJECT:testIamPermissions"
```

### APIs

`setup.sh` checks these and offers to enable them:

```
gcloud services enable \
  compute.googleapis.com run.googleapis.com pubsub.googleapis.com \
  logging.googleapis.com artifactregistry.googleapis.com iam.googleapis.com \
  --project="$PROJECT"
```

Only project-level IAM is visible locally; org-level constraints surface at apply time. The two most likely to bite are a constraint blocking external HTTP(S) load balancers, and `constraints/iam.disableServiceAccountKeyCreation`.

## The Terraform trap that fails silently

Neither `hashicorp/google` nor `google-beta` (v7) models `loggingHttpRequestHeaders` — `log_config` exposes only `enable`, `optional_fields`, `optional_mode` and `sample_rate`. The module therefore shells out to `gcloud` from a `terraform_data` + `local-exec`.

> **Warning: The allow-list survives Terraform only if it is re-asserted**
>
> The first version keyed the shell-out on a hash of the inputs and paired it with a destroy-time provisioner that removed the list again. On the next full `terraform apply` that destroy provisioner fired during a replacement and wrote back a `logConfig` with no `loggingHttpRequestHeaders`.
>
> Logs kept flowing. They simply arrived with nothing to classify on, and `terraform plan` said “No changes” because Terraform cannot see a field it does not model.
>
> The fix is two resources rather than one. `terraform_data.request_logging` triggers on `timestamp()`, so every apply re-installs the list and any drift heals itself. Removal moved to `terraform_data.request_logging_cleanup`, whose triggers never change, so it fires only on a genuine `terraform destroy`.
>
> **The cost is one unavoidable change in every plan. Do not “clean that up”** — the trade is an invisible failure of the pipeline's whole purpose against a noisy plan.

## Before you trust the numbers

> **Caution: Config changes take minutes to reach the edge**
>
> After re-installing the allow-list, requests sent ~60 seconds later were still logged with no headers. The same probe at ~10 minutes carried all of them. Budget several minutes before trusting a run taken straight after an apply, and treat a sudden run of headerless events after a deploy as propagation rather than breakage.

> **Note: Excluding from _Default really does hide the logs**
>
> Once `exclude_from_default_bucket` takes effect, `gcloud logging read` returns nothing for the load balancer — the entries are routed but never stored. That is the intended saving, and it means **Pub/Sub is the only place to read raw entries**. `make sample` reads storage, so it goes quiet once the exclusion lands; pull the verify subscription instead.

> **Note: Delivery is at-least-once**
>
> A push that fails after the adapter already forwarded some of its events is retried whole. After `max_delivery_attempts` the message parks on the dead-letter topic, where it stays queued and replayable.

> **Note: The adapter image is built locally, not in Cloud Build**
>
> `terraform apply` builds and pushes a container unless you set `adapter_image`. That needs `docker` and an authenticated `gcloud` on the machine running apply. In a default-configured project Cloud Build runs as the default compute service account, which cannot read its own staging bucket — a local build needs only a configured credential helper, and is faster besides.

## Cost

Per 1M requests/month, ~1 KB per log entry:

| Component | Cost |
|---|---|
| Pub/Sub | ~$0.04 |
| Cloud Run adapter | $0 — free tier, scales to zero |
| Cloud Logging routing | $0 — routing to a sink is not charged |
| Cloud Logging storage | $0 with `exclude_from_default_bucket` |
| **Pipeline total** | **under $1/month** |

The load balancer itself is **not** free — roughly $18/month for the forwarding rule plus per-GB processing, whatever the traffic. That does not matter to a customer who already runs one; it matters a lot for a demo rig that exists only to make traffic.

> **Caution: Confirm before quoting**
>
> Check current forwarding-rule and data-processing pricing before quoting any of these numbers to anyone.

## Tearing it down

```
make destroy      # terraform destroy, then check nothing is left standing
make leftovers    # just tell me what is still deployed
```

Do this between runs. The forwarding rule bills at idle whether or not traffic is flowing, and it is the single most expensive thing in the repo.

`terraform destroy` needs `project_id`, which has no default. Keep it in `site/terraform/terraform.tfvars`; the script checks for it up front rather than letting Terraform stall on a prompt. `destroy.sh --sweep` deletes leftovers rather than only reporting them — useful if a destroy half-finishes, since a Cloud Run service or backend service left behind will not show up in any later plan.

## Still unverified

Do not assert these:

- per-header value truncation in `loggingHttpRequestHeaders` — the *count* limit is settled at 10, and nothing in a 40-request run came back truncated
- whether request logging is sampled independently of `sample_rate` under load, the way CloudFront real-time logs turned out to be best-effort. Three runs of 40 delivered 39, 40 and 40 — no evidence of loss, far too small a sample to call it exact
- end-to-end latency beyond the observed ~30–60s

## Code on this page

- [`gcp/ingestion/main.tf`](https://github.com/Obsero-ai/obsero-public/blob/main/gcp/ingestion/main.tf) -- the pipeline: sink, Pub/Sub, Cloud Run adapter
- [`gcp/ingestion/variables.tf`](https://github.com/Obsero-ai/obsero-public/blob/main/gcp/ingestion/variables.tf) -- every module input, with its validation
- [`gcp/ingestion/adapter/index.mjs`](https://github.com/Obsero-ai/obsero-public/blob/main/gcp/ingestion/adapter/index.mjs) -- the Cloud Run adapter
- [`gcp/site/terraform/main.tf`](https://github.com/Obsero-ai/obsero-public/blob/main/gcp/site/terraform/main.tf) -- the demo stack: mock site plus one module block
- [`gcp/PERMISSIONS.md`](https://github.com/Obsero-ai/obsero-public/blob/main/gcp/PERMISSIONS.md) -- the IAM roles, and why Editor is not enough
- [`gcp/NOTES.md`](https://github.com/Obsero-ai/obsero-public/blob/main/gcp/NOTES.md) -- behaviour measured on live deployments


---

<!-- Generated by docs/build.py from troubleshooting.html. Edit the HTML, not this file. -->

# Troubleshooting

Symptom, cause, fix. Most of what gets reported as a bug here is documented behaviour of the underlying service.

## Work down this list first

In order. Do not skip to Terraform.

1. `cd <cloud> && make test` — does the adapter still produce a valid payload? Needs no cloud account. If this fails, nothing downstream matters.
2. `./setup.sh <cloud> status` — what is actually deployed, including resources that fell out of Terraform state.
3. `node obsero.mjs --token <id> --domain <host>` — is the tracking ID accepted? A rejected token means the pipeline forwards into nothing, *silently*.
4. `make -C <cloud> logs` — is the adapter being invoked, and what does it say?
5. Then the tables below.

## Either cloud

| Symptom | Cause | Fix |
|---|---|---|
| Analytics shows one page per query string | `path` is being sent with the query string on it | Send the path only. `buildEvent` strips it for you; a custom parser must too. This is the most common mistake by far. |
| Duplicate events | Delivery is at-least-once on both clouds | Expected. One failed event fails its whole batch, which is retried, duplicating what already landed. |
| Nothing arrives and nothing errors | Often a rejected tracking ID | `node obsero.mjs --token <id> --domain <host>`. One second to rule out. |
| `setup.sh` step 2 refuses to apply | The tracking ID in `terraform.tfvars` is still the placeholder `set-in-step-2` | Run step 2 and enter a real ID, or `./setup.sh <cloud> pipeline --token <id>`. |
| `sec-fetch-mode` is `cors`, and `connection: keep-alive` appeared | Node's `fetch` overrides both | Harness artefact only. Every User-Agent and every other header passes through untouched. |
| A resource survives `terraform destroy` | Partial destroy, or lost state — neither shows up in a plan | `./destroy.sh <cloud> --check` reports them with the exact command to remove each; `--sweep` runs those commands. |

## AWS

| Symptom | Cause | Fix |
|---|---|---|
| Nothing arrives, everything reports healthy, Kinesis `IncomingRecords` is 0 | The `LogDeliveryEnabled = true` tag is missing from the Firehose stream (standard mode) | The module applies it automatically. If someone hand-built the stream, add it. There is **no error message for this anywhere**. |
| Nothing yet, and the deploy finished under a minute ago | The 60–90s latency floor | Wait. CloudFront flushes continuously but Firehose buffers for at least 60s, and that is not a knob. |
| 10 of 30 requests missing, no pattern by status, path, client or latency (realtime) | CloudFront delivers real-time logs **best-effort** | Not a bug and not fixable from here. Treat counts as a high-fidelity sample, never a ledger. Do not reconcile against origin logs or bill on them. |
| Every column in a parsed record is shifted (realtime) | CloudFront emits fields in *its own* canonical order, ignoring the log config, with no header row | The module sorts `log_fields` canonically and hands the same list to the adapter; `log_fields_in_order` exports it. Never trust the config order positionally. |
| No `signature-agent` / `signature-input` / `signature` ever arrives | `log_source = "standard"`. Standard logging v2 has no `cs-headers` field. | Switch to `realtime`. Not a config tweak — a different log source, at a ~$11/mo floor. |
| Adapter returns 429; events land in the backup bucket | Lambda concurrency exhausted — new accounts cap at 10 account-wide, and at that ceiling `reserved_concurrent_executions` cannot be set at all | Nothing is lost; replay the backup bucket. Watch `Throttles` on the adapter. |
| `terraform destroy` refuses to delete a bucket | It holds objects Terraform did not create — exactly what the backup bucket collects | Use `./destroy.sh aws`, which empties it first. The module keeps `backup_force_destroy = false` on purpose, so a real destroy *stops* rather than discarding events that never reached Obsero. |
| Connect says `could not list CloudFront distributions` | Your AWS identity lacks `cloudfront:ListDistributions`, or the CLI is not logged in | Run `aws cloudfront list-distributions` yourself — its error names the missing permission. |
| Connect lists nothing, or not the distribution you expected | The CLI is pointed at a different account | `aws sts get-caller-identity` shows which. Set `AWS_PROFILE` and re-run; distributions are global, so region does not matter here. |
| Apply fails with a `ConflictException` on a delivery source | The distribution already had a standard logging v2 source that was not passed in `existing_delivery_sources` | Re-run `./setup.sh aws connect`: it re-reads the sources and fills the variable in. In your own Terraform, add the source's name there. Only one source per distribution is allowed. |
| Apply fails because a connected distribution does not exist | It was deleted — typically the mock site, destroyed and redeployed with a new ID | Steps 2 and 3 drop missing distributions automatically. By hand: `./setup.sh aws connect --dist none`, then connect again. |
| Connected, but nothing arrives from your own site | No traffic yet, or it has not been 90 seconds | Request a page, wait out the [latency floor](aws.md#gotchas), then `make -C aws logs`. `aws logs describe-deliveries --region us-east-1` confirms the delivery exists. |
| The connect picker says it needs bash 4 | macOS ships bash 3.2 | `brew install bash`, then re-run. Or skip the picker: `./setup.sh aws connect --dist <id>` still needs bash 4, so use the Terraform variables directly. |
| A CloudFront distribution survives the destroy | CloudFront needs two operations, minutes apart | Disable it, wait for `Deployed`, then delete. `destroy.sh` reports it rather than pretending it can be done in one pass. |

## GCP

| Symptom | Cause | Fix |
|---|---|---|
| There are no LB request logs at all | The origin is a backend **bucket**. `BackendBucket` has no `logConfig` field in the Compute v1 API. | The origin has to become a backend **service** — Cloud Run, GKE, a MIG. No config change fixes this. |
| Events arrive with no headers to classify on, and `terraform plan` says “No changes” | The allow-list was wiped by a destroy-time provisioner during a replacement. Terraform cannot see a field it does not model. | Already fixed as two resources: `terraform_data.request_logging` re-installs on every apply, `..._cleanup` fires only on destroy. **Do not remove the every-apply change to quiet the plan.** |
| Headerless events for a few minutes after an apply | Config propagation to the edge | Wait. Measured: headerless at ~60s, all headers present at ~10 minutes. |
| `gcloud logging read` returns nothing for the load balancer | `exclude_from_default_bucket` is working — entries are routed but never stored | That is the intended saving. **Pub/Sub is the only place to read raw entries.** Pull the verify subscription instead of `make sample`. |
| The sink exists but publishes nothing | The writer identity lacks publisher on the topic | Needs `roles/pubsub.admin` at apply time — `pubsub.editor` does not cover `pubsub.topics.setIamPolicy`. |
| `terraform apply` fails at the sink | `roles/logging.configWriter` missing | Editor does not imply it. [See the roles table →](gcp.md#permissions) |
| Adding an eleventh logged header is rejected | The API caps `loggingHttpRequestHeaders` at 10 | Decide which one to drop. `user-agent`, `referer` and the client IP arrive as first-class fields anyway, so never spend a slot on those. |
| The container image build fails | Missing `docker`, or an unconfigured credential helper | The build is deliberately local — Cloud Build's default compute service account cannot read its own staging bucket in a default project. Or set `adapter_image` to skip the build. |
| `terraform destroy` prompts for `project_id` | It has no default | Keep it in `gcp/site/terraform/terraform.tfvars`. `destroy.sh` checks for it up front. |
| A service account survives `--sweep` | Service account IDs cap at 30 characters, so the module truncates the prefix to 21 | A name match on the full prefix misses them. `destroy.sh` already truncates the same way; pass `--prefix` if you renamed things. |
| An external LB cannot be created at all | Org policy | Only project-level IAM is visible locally; org constraints surface at apply time. The other likely one is `constraints/iam.disableServiceAccountKeyCreation`. |

## Where the measured behaviour lives

`aws/NOTES.md` and `gcp/NOTES.md` record behaviour observed on live deployments rather than inferred from docs. They are separate from the READMEs because none of it is needed to *use* the modules — only to trust their numbers, or to hand-build a pipeline yourself.

If you measure something new, add it there as well as here.

## Code on this page

- [`aws/NOTES.md`](https://github.com/Obsero-ai/obsero-public/blob/main/aws/NOTES.md) -- AWS behaviour measured on live deployments
- [`gcp/NOTES.md`](https://github.com/Obsero-ai/obsero-public/blob/main/gcp/NOTES.md) -- GCP behaviour measured on live deployments
- [`destroy.sh`](https://github.com/Obsero-ai/obsero-public/blob/main/destroy.sh) -- what --check and --sweep look for