Obsero ingestion Documentation
The contract GitHub

Troubleshooting

Symptom, cause, fix. Most of what gets reported as a bug here is documented behaviour of the underlying service.

#Work down this list first

In order. Do not skip to Terraform.

  1. cd <cloud> && make test — does the adapter still produce a valid payload? Needs no cloud account. If this fails, nothing downstream matters.
  2. ./setup.sh <cloud> status — what is actually deployed, including resources that fell out of Terraform state.
  3. node obsero.mjs --token <id> --domain <host> — is the tracking ID accepted? A rejected token means the pipeline forwards into nothing, silently.
  4. make -C <cloud> logs — is the adapter being invoked, and what does it say?
  5. Then the tables below.

#Either cloud

SymptomCauseFix
Analytics shows one page per query string path is being sent with the query string on it Send the path only. buildEvent strips it for you; a custom parser must too. This is the most common mistake by far.
Duplicate events Delivery is at-least-once on both clouds Expected. One failed event fails its whole batch, which is retried, duplicating what already landed.
Nothing arrives and nothing errors Often a rejected tracking ID node obsero.mjs --token <id> --domain <host>. One second to rule out.
setup.sh step 2 refuses to apply The tracking ID in terraform.tfvars is still the placeholder set-in-step-2 Run step 2 and enter a real ID, or ./setup.sh <cloud> pipeline --token <id>.
sec-fetch-mode is cors, and connection: keep-alive appeared Node's fetch overrides both Harness artefact only. Every User-Agent and every other header passes through untouched.
A resource survives terraform destroy Partial destroy, or lost state — neither shows up in a plan ./destroy.sh <cloud> --check reports them with the exact command to remove each; --sweep runs those commands.

#AWS

SymptomCauseFix
Nothing arrives, everything reports healthy, Kinesis IncomingRecords is 0 The LogDeliveryEnabled = true tag is missing from the Firehose stream (standard mode) The module applies it automatically. If someone hand-built the stream, add it. There is no error message for this anywhere.
Nothing yet, and the deploy finished under a minute ago The 60–90s latency floor Wait. CloudFront flushes continuously but Firehose buffers for at least 60s, and that is not a knob.
10 of 30 requests missing, no pattern by status, path, client or latency (realtime) CloudFront delivers real-time logs best-effort Not a bug and not fixable from here. Treat counts as a high-fidelity sample, never a ledger. Do not reconcile against origin logs or bill on them.
Every column in a parsed record is shifted (realtime) CloudFront emits fields in its own canonical order, ignoring the log config, with no header row The module sorts log_fields canonically and hands the same list to the adapter; log_fields_in_order exports it. Never trust the config order positionally.
No signature-agent / signature-input / signature ever arrives log_source = "standard". Standard logging v2 has no cs-headers field. Switch to realtime. Not a config tweak — a different log source, at a ~$11/mo floor.
Adapter returns 429; events land in the backup bucket Lambda concurrency exhausted — new accounts cap at 10 account-wide, and at that ceiling reserved_concurrent_executions cannot be set at all Nothing is lost; replay the backup bucket. Watch Throttles on the adapter.
terraform destroy refuses to delete a bucket It holds objects Terraform did not create — exactly what the backup bucket collects Use ./destroy.sh aws, which empties it first. The module keeps backup_force_destroy = false on purpose, so a real destroy stops rather than discarding events that never reached Obsero.
Connect says could not list CloudFront distributions Your AWS identity lacks cloudfront:ListDistributions, or the CLI is not logged in Run aws cloudfront list-distributions yourself — its error names the missing permission.
Connect lists nothing, or not the distribution you expected The CLI is pointed at a different account aws sts get-caller-identity shows which. Set AWS_PROFILE and re-run; distributions are global, so region does not matter here.
Apply fails with a ConflictException on a delivery source The distribution already had a standard logging v2 source that was not passed in existing_delivery_sources Re-run ./setup.sh aws connect: it re-reads the sources and fills the variable in. In your own Terraform, add the source's name there. Only one source per distribution is allowed.
Apply fails because a connected distribution does not exist It was deleted — typically the mock site, destroyed and redeployed with a new ID Steps 2 and 3 drop missing distributions automatically. By hand: ./setup.sh aws connect --dist none, then connect again.
Connected, but nothing arrives from your own site No traffic yet, or it has not been 90 seconds Request a page, wait out the latency floor, then make -C aws logs. aws logs describe-deliveries --region us-east-1 confirms the delivery exists.
The connect picker says it needs bash 4 macOS ships bash 3.2 brew install bash, then re-run. Or skip the picker: ./setup.sh aws connect --dist <id> still needs bash 4, so use the Terraform variables directly.
A CloudFront distribution survives the destroy CloudFront needs two operations, minutes apart Disable it, wait for Deployed, then delete. destroy.sh reports it rather than pretending it can be done in one pass.

#GCP

SymptomCauseFix
There are no LB request logs at all The origin is a backend bucket. BackendBucket has no logConfig field in the Compute v1 API. The origin has to become a backend service — Cloud Run, GKE, a MIG. No config change fixes this.
Events arrive with no headers to classify on, and terraform plan says “No changes” The allow-list was wiped by a destroy-time provisioner during a replacement. Terraform cannot see a field it does not model. Already fixed as two resources: terraform_data.request_logging re-installs on every apply, ..._cleanup fires only on destroy. Do not remove the every-apply change to quiet the plan.
Headerless events for a few minutes after an apply Config propagation to the edge Wait. Measured: headerless at ~60s, all headers present at ~10 minutes.
gcloud logging read returns nothing for the load balancer exclude_from_default_bucket is working — entries are routed but never stored That is the intended saving. Pub/Sub is the only place to read raw entries. Pull the verify subscription instead of make sample.
The sink exists but publishes nothing The writer identity lacks publisher on the topic Needs roles/pubsub.admin at apply time — pubsub.editor does not cover pubsub.topics.setIamPolicy.
terraform apply fails at the sink roles/logging.configWriter missing Editor does not imply it. See the roles table →
Adding an eleventh logged header is rejected The API caps loggingHttpRequestHeaders at 10 Decide which one to drop. user-agent, referer and the client IP arrive as first-class fields anyway, so never spend a slot on those.
The container image build fails Missing docker, or an unconfigured credential helper The build is deliberately local — Cloud Build's default compute service account cannot read its own staging bucket in a default project. Or set adapter_image to skip the build.
terraform destroy prompts for project_id It has no default Keep it in gcp/site/terraform/terraform.tfvars. destroy.sh checks for it up front.
A service account survives --sweep Service account IDs cap at 30 characters, so the module truncates the prefix to 21 A name match on the full prefix misses them. destroy.sh already truncates the same way; pass --prefix if you renamed things.
An external LB cannot be created at all Org policy Only project-level IAM is visible locally; org constraints surface at apply time. The other likely one is constraints/iam.disableServiceAccountKeyCreation.

#Where the measured behaviour lives

aws/NOTES.md and gcp/NOTES.md record behaviour observed on live deployments rather than inferred from docs. They are separate from the READMEs because none of it is needed to use the modules — only to trust their numbers, or to hand-build a pipeline yourself.

If you measure something new, add it there as well as here.

Code on this page

Edit this page on GitHubView as Markdownllms.txt for agents

Was this page helpful?

Reference implementations for streaming CDN request logs to Obsero — a starting point, not a product. Fork it and keep the parts that match your stack. Source on GitHub.

Measured numbers on these pages come from live deployments and are recorded in each cloud's NOTES.md. Confirm current cloud pricing before quoting any cost figure.