AWS — CloudFront

Streams every request hitting a CloudFront distribution to Obsero.

Everything runs in your own AWS account. The adapter strips authorization, cookie and set-cookie before anything leaves it.

#Try it without an AWS account

cd aws
make test                                        # adapter payload vs the contract
make event TOKEN=<tracking-id> DOMAIN=acme.com    # POST one event, check the ID

Then, to watch it work end to end, the quickstart — or the make targets directly:

make deploy            # demo site + pipeline, connected to each other
make traffic N=40      # mock browser / AI-agent / crawler traffic
make check             # score what actually reached the endpoint
make logs              # follow the adapter forwarding, live
make destroy           # tear it all down again

#Wiring it into your own distribution

One module block. The module never touches your distribution — in standard mode it only needs the distribution's ARN, and in realtime mode it hands back an ARN for you to attach — so it works whether or not Terraform manages your CloudFront.

module "obsero_ingestion" {
  source = "./ingestion"

  name_prefix = "acme-prod"
  site_token  = var.obsero_site_token

  # CloudFront delivers straight into Firehose. No idle cost.
  log_source        = "standard"
  distribution_arns = [
    aws_cloudfront_distribution.site.arn,
    "arn:aws:cloudfront::123456789012:distribution/E2ABC123", # one Terraform does not manage
  ]
}

distribution_arns can be empty: the pipeline is then built idle, costing nothing, until you add one. If a distribution already has a standard logging v2 source — the CloudFront console creates one the first time anyone enables that logging — also pass it in existing_delivery_sources, because a distribution can only have one:

existing_delivery_sources = {
  "arn:aws:cloudfront::123456789012:distribution/E2ABC123" = "CreatedByCloudFront-E2ABC123"
}

For realtime, drop distribution_arns and attach the exported ARN to each cache behaviour you want logged instead:

default_cache_behavior {
  # ...
  realtime_log_config_arn = module.obsero_ingestion.realtime_log_config_arn
}

Behaviours without the ARN emit nothing, so you can roll it out one path pattern at a time. See ingestion/examples/ for both shapes and ingestion/README.md for every input.

#Connecting distributions you already run

With setup.sh, this is step 3: it lists every distribution in the account, detects existing logging sources for you, and writes both variables above into aws/site/terraform/terraform.tfvars. See using your own site for the walkthrough.

./setup.sh aws connect                        # pick from a list
./setup.sh aws connect --dist E2ABC123,E3DEF456 # exactly these
./setup.sh aws connect --dist +E2ABC123         # add one
./setup.sh aws connect --dist none              # disconnect all

To check what is connected from the AWS side, independent of Terraform state:

aws logs describe-deliveries --region us-east-1 \
  --query 'deliveries[].[deliverySourceName,deliveryDestinationArn]' --output table
aws logs describe-delivery-sources --region us-east-1 \
  --query 'deliverySources[].[name,resourceArns[0]]' --output table

Each connected distribution has one delivery whose destination ends in <name_prefix>-firehose. Disconnecting deletes that delivery, and the source too if the module created it; a source that was there first is left alone.

#What the module creates

For reviewing before you apply it in a real account. Everything is prefixed with name_prefix and tagged ManagedBy = obsero-ingestion. Source: aws/ingestion/main.tf and aws/ingestion/lambda.tf.

Bad inputs fail at plan time, not as empty events: terraform_data.validate_config checks the log field lists and that every existing_delivery_sources key is also connected.

#Which log source? The one real decision

CloudFront makes you choose.

Start on standard. Move to realtime when you want to verify an agent rather than take its User-Agent at face value: standard logging v2 has no cs-headers field, so signature-agent / signature-input / signature never arrive at all. That is not a tuning problem — it is a different log source.

GCP has no such trade: it carries named headers on the cheap path.

#Before you trust the numbers

Four behaviours that are easy to mistake for bugs. All measured on a live deployment, all written up in aws/NOTES.md.

#Two CloudFront parsing behaviours (realtime only)

Both are handled by the adapter. They are documented because they bite anyone reading the raw Kinesis stream themselves. Standard mode is self-describing JSON, so neither applies there.

1. Field order is not your field order. CloudFront emits real-time log fields in its own canonical order, ignoring the order in the log config. Records are tab-separated with no header row, so a positional parser that trusts the config order reads every column shifted. The module sorts log_fields into canonical order and hands that same list to the adapter; log_fields_in_order exports it.
2. cs-uri-stem includes the query string in real-time logs, unlike standard access logs where it is path-only. The adapter strips it, or every distinct query string would register as its own path in the analytics.

#Why the adapter Lambda exists at all

Firehose's HTTP endpoint destination cannot post to /v1/events directly:

• it wraps records in {requestId, timestamp, records:[{data}]}
• it sends the key as X-Amz-Firehose-Access-Key, and cannot send arbitrary headers such as x-obsero-domain / x-obsero-key
• it requires a specific JSON ack back, or treats the delivery as failed

The adapter unwraps the envelope, converts each log line into one Obsero event, strips excluded headers, and returns the ack. Its Function URL is unauthenticated at the AWS layer — Firehose cannot SigV4-sign to a Function URL — so the shared Firehose access key is what the handler checks. The URL is not secret, but is useless without that key.

#Concurrency footprint

The adapter uses one concurrent Lambda execution per Kinesis shard. Firehose reads each shard serially and invokes once per buffer window, so at the default kinesis_shard_count = 1 the steady state is exactly 1 concurrent execution, ~1s per invocation, one invocation per minute. Measured, not estimated.

That matters on accounts with a low concurrency ceiling. New AWS accounts are capped at 10 concurrent executions account-wide, and at that ceiling reserved_concurrent_executions cannot be set at all — AWS requires unreserved concurrency to stay at or above 10, so any reservation is rejected. The adapter therefore shares an unprotected pool with every other function in the account.

If another function saturates the pool, the Function URL returns 429, Firehose treats it as a failed delivery, retries for retry_duration, and then writes the batch to the backup bucket. Nothing is lost, but those events do not reach Obsero until the backup is replayed. Watch Throttles if you share a small pool.

#Cost

Lambda, CloudFront and S3 usage at demo volume sit inside the AWS free tier. Kinesis Data Streams does not have a free tier — one provisioned shard is about $11/month ($0.015/shard-hour) and is realtime mode's floor cost, since CloudFront real-time logs can only be delivered to Kinesis. Firehose adds ~$0.029/GB ingested, which is cents at typical volumes.

In standard mode there is no Kinesis stream and therefore no idle cost at all; Firehose only reaches $11/month at roughly 380M requests.

#Tearing it down

make destroy      # terraform destroy, then check nothing is left standing
make leftovers    # just tell me what is still deployed

Both call destroy.sh, which empties the S3 buckets first — Terraform refuses to delete a bucket holding objects it did not create, and that is precisely what the backup bucket collects.

The module itself keeps backup_force_destroy = false, so a customer's terraform destroy will stop rather than silently discard events that never reached Obsero. The demo site sets it to true, because nobody is going to replay those.