<!-- Generated by docs/build.py from troubleshooting.html. Edit the HTML, not this file. -->

# Troubleshooting

Symptom, cause, fix. Most of what gets reported as a bug here is documented behaviour of the underlying service.

## Work down this list first

In order. Do not skip to Terraform.

1. `cd <cloud> && make test` — does the adapter still produce a valid payload? Needs no cloud account. If this fails, nothing downstream matters.
2. `./setup.sh <cloud> status` — what is actually deployed, including resources that fell out of Terraform state.
3. `node obsero.mjs --token <id> --domain <host>` — is the tracking ID accepted? A rejected token means the pipeline forwards into nothing, *silently*.
4. `make -C <cloud> logs` — is the adapter being invoked, and what does it say?
5. Then the tables below.

## Either cloud

| Symptom | Cause | Fix |
|---|---|---|
| Analytics shows one page per query string | `path` is being sent with the query string on it | Send the path only. `buildEvent` strips it for you; a custom parser must too. This is the most common mistake by far. |
| Duplicate events | Delivery is at-least-once on both clouds | Expected. One failed event fails its whole batch, which is retried, duplicating what already landed. |
| Nothing arrives and nothing errors | Often a rejected tracking ID | `node obsero.mjs --token <id> --domain <host>`. One second to rule out. |
| `setup.sh` step 2 refuses to apply | The tracking ID in `terraform.tfvars` is still the placeholder `set-in-step-2` | Run step 2 and enter a real ID, or `./setup.sh <cloud> pipeline --token <id>`. |
| `sec-fetch-mode` is `cors`, and `connection: keep-alive` appeared | Node's `fetch` overrides both | Harness artefact only. Every User-Agent and every other header passes through untouched. |
| A resource survives `terraform destroy` | Partial destroy, or lost state — neither shows up in a plan | `./destroy.sh <cloud> --check` reports them with the exact command to remove each; `--sweep` runs those commands. |

## AWS

| Symptom | Cause | Fix |
|---|---|---|
| Nothing arrives, everything reports healthy, Kinesis `IncomingRecords` is 0 | The `LogDeliveryEnabled = true` tag is missing from the Firehose stream (standard mode) | The module applies it automatically. If someone hand-built the stream, add it. There is **no error message for this anywhere**. |
| Nothing yet, and the deploy finished under a minute ago | The 60–90s latency floor | Wait. CloudFront flushes continuously but Firehose buffers for at least 60s, and that is not a knob. |
| 10 of 30 requests missing, no pattern by status, path, client or latency (realtime) | CloudFront delivers real-time logs **best-effort** | Not a bug and not fixable from here. Treat counts as a high-fidelity sample, never a ledger. Do not reconcile against origin logs or bill on them. |
| Every column in a parsed record is shifted (realtime) | CloudFront emits fields in *its own* canonical order, ignoring the log config, with no header row | The module sorts `log_fields` canonically and hands the same list to the adapter; `log_fields_in_order` exports it. Never trust the config order positionally. |
| No `signature-agent` / `signature-input` / `signature` ever arrives | `log_source = "standard"`. Standard logging v2 has no `cs-headers` field. | Switch to `realtime`. Not a config tweak — a different log source, at a ~$11/mo floor. |
| Adapter returns 429; events land in the backup bucket | Lambda concurrency exhausted — new accounts cap at 10 account-wide, and at that ceiling `reserved_concurrent_executions` cannot be set at all | Nothing is lost; replay the backup bucket. Watch `Throttles` on the adapter. |
| `terraform destroy` refuses to delete a bucket | It holds objects Terraform did not create — exactly what the backup bucket collects | Use `./destroy.sh aws`, which empties it first. The module keeps `backup_force_destroy = false` on purpose, so a real destroy *stops* rather than discarding events that never reached Obsero. |
| Connect says `could not list CloudFront distributions` | Your AWS identity lacks `cloudfront:ListDistributions`, or the CLI is not logged in | Run `aws cloudfront list-distributions` yourself — its error names the missing permission. |
| Connect lists nothing, or not the distribution you expected | The CLI is pointed at a different account | `aws sts get-caller-identity` shows which. Set `AWS_PROFILE` and re-run; distributions are global, so region does not matter here. |
| Apply fails with a `ConflictException` on a delivery source | The distribution already had a standard logging v2 source that was not passed in `existing_delivery_sources` | Re-run `./setup.sh aws connect`: it re-reads the sources and fills the variable in. In your own Terraform, add the source's name there. Only one source per distribution is allowed. |
| Apply fails because a connected distribution does not exist | It was deleted — typically the mock site, destroyed and redeployed with a new ID | Steps 2 and 3 drop missing distributions automatically. By hand: `./setup.sh aws connect --dist none`, then connect again. |
| Connected, but nothing arrives from your own site | No traffic yet, or it has not been 90 seconds | Request a page, wait out the [latency floor](aws.md#gotchas), then `make -C aws logs`. `aws logs describe-deliveries --region us-east-1` confirms the delivery exists. |
| The connect picker says it needs bash 4 | macOS ships bash 3.2 | `brew install bash`, then re-run. Or skip the picker: `./setup.sh aws connect --dist <id>` still needs bash 4, so use the Terraform variables directly. |
| A CloudFront distribution survives the destroy | CloudFront needs two operations, minutes apart | Disable it, wait for `Deployed`, then delete. `destroy.sh` reports it rather than pretending it can be done in one pass. |

## GCP

| Symptom | Cause | Fix |
|---|---|---|
| There are no LB request logs at all | The origin is a backend **bucket**. `BackendBucket` has no `logConfig` field in the Compute v1 API. | The origin has to become a backend **service** — Cloud Run, GKE, a MIG. No config change fixes this. |
| Events arrive with no headers to classify on, and `terraform plan` says “No changes” | The allow-list was wiped by a destroy-time provisioner during a replacement. Terraform cannot see a field it does not model. | Already fixed as two resources: `terraform_data.request_logging` re-installs on every apply, `..._cleanup` fires only on destroy. **Do not remove the every-apply change to quiet the plan.** |
| Headerless events for a few minutes after an apply | Config propagation to the edge | Wait. Measured: headerless at ~60s, all headers present at ~10 minutes. |
| `gcloud logging read` returns nothing for the load balancer | `exclude_from_default_bucket` is working — entries are routed but never stored | That is the intended saving. **Pub/Sub is the only place to read raw entries.** Pull the verify subscription instead of `make sample`. |
| The sink exists but publishes nothing | The writer identity lacks publisher on the topic | Needs `roles/pubsub.admin` at apply time — `pubsub.editor` does not cover `pubsub.topics.setIamPolicy`. |
| `terraform apply` fails at the sink | `roles/logging.configWriter` missing | Editor does not imply it. [See the roles table →](gcp.md#permissions) |
| Adding an eleventh logged header is rejected | The API caps `loggingHttpRequestHeaders` at 10 | Decide which one to drop. `user-agent`, `referer` and the client IP arrive as first-class fields anyway, so never spend a slot on those. |
| The container image build fails | Missing `docker`, or an unconfigured credential helper | The build is deliberately local — Cloud Build's default compute service account cannot read its own staging bucket in a default project. Or set `adapter_image` to skip the build. |
| `terraform destroy` prompts for `project_id` | It has no default | Keep it in `gcp/site/terraform/terraform.tfvars`. `destroy.sh` checks for it up front. |
| A service account survives `--sweep` | Service account IDs cap at 30 characters, so the module truncates the prefix to 21 | A name match on the full prefix misses them. `destroy.sh` already truncates the same way; pass `--prefix` if you renamed things. |
| An external LB cannot be created at all | Org policy | Only project-level IAM is visible locally; org constraints surface at apply time. The other likely one is `constraints/iam.disableServiceAccountKeyCreation`. |

## Where the measured behaviour lives

`aws/NOTES.md` and `gcp/NOTES.md` record behaviour observed on live deployments rather than inferred from docs. They are separate from the READMEs because none of it is needed to *use* the modules — only to trust their numbers, or to hand-build a pipeline yourself.

If you measure something new, add it there as well as here.

## Code on this page

- [`aws/NOTES.md`](https://github.com/Obsero-ai/obsero-public/blob/main/aws/NOTES.md) -- AWS behaviour measured on live deployments
- [`gcp/NOTES.md`](https://github.com/Obsero-ai/obsero-public/blob/main/gcp/NOTES.md) -- GCP behaviour measured on live deployments
- [`destroy.sh`](https://github.com/Obsero-ai/obsero-public/blob/main/destroy.sh) -- what --check and --sweep look for