<!-- Generated by docs/build.py from aws.html. Edit the HTML, not this file. -->

# AWS — CloudFront

Streams every request hitting a CloudFront distribution to Obsero.

```
viewer -> CloudFront -> Firehose -> adapter Lambda -> POST /v1/events
                                      \-> S3 backup bucket (undeliverable batches)
```

Everything runs in your own AWS account. The adapter strips `authorization`, `cookie` and `set-cookie` before anything leaves it.

## Try it without an AWS account

```
cd aws
make test                                        # adapter payload vs the contract
make event TOKEN=<tracking-id> DOMAIN=acme.com    # POST one event, check the ID
```

Then, to watch it work end to end, [the quickstart](quickstart.md) — or the make targets directly:

```
make deploy            # demo site + pipeline, connected to each other
make traffic N=40      # mock browser / AI-agent / crawler traffic
make check             # score what actually reached the endpoint
make logs              # follow the adapter forwarding, live
make destroy           # tear it all down again
```

## Wiring it into your own distribution

One module block. The module **never touches your distribution** — in standard mode it only needs the distribution's ARN, and in realtime mode it hands back an ARN for you to attach — so it works whether or not Terraform manages your CloudFront.

*standard mode — the whole integration*

```
module "obsero_ingestion" {
  source = "./ingestion"

  name_prefix = "acme-prod"
  site_token  = var.obsero_site_token

  # CloudFront delivers straight into Firehose. No idle cost.
  log_source        = "standard"
  distribution_arns = [
    aws_cloudfront_distribution.site.arn,
    "arn:aws:cloudfront::123456789012:distribution/E2ABC123", # one Terraform does not manage
  ]
}
```

`distribution_arns` can be empty: the pipeline is then built idle, costing nothing, until you add one. If a distribution already has a standard logging v2 source — the CloudFront console creates one the first time anyone enables that logging — also pass it in `existing_delivery_sources`, because a distribution can only have one:

```
existing_delivery_sources = {
  "arn:aws:cloudfront::123456789012:distribution/E2ABC123" = "CreatedByCloudFront-E2ABC123"
}
```

For `realtime`, drop `distribution_arns` and attach the exported ARN to each cache behaviour you want logged instead:

```
default_cache_behavior {
  # ...
  realtime_log_config_arn = module.obsero_ingestion.realtime_log_config_arn
}
```

Behaviours without the ARN emit nothing, so you can roll it out one path pattern at a time. See `ingestion/examples/` for both shapes and `ingestion/README.md` for every input.

## Connecting distributions you already run

With `setup.sh`, this is step 3: it lists every distribution in the account, detects existing logging sources for you, and writes both variables above into `aws/site/terraform/terraform.tfvars`. See [using your own site](quickstart.md#your-own-site) for the walkthrough.

```
./setup.sh aws connect                        # pick from a list
./setup.sh aws connect --dist E2ABC123,E3DEF456 # exactly these
./setup.sh aws connect --dist +E2ABC123         # add one
./setup.sh aws connect --dist none              # disconnect all
```

To check what is connected from the AWS side, independent of Terraform state:

```
aws logs describe-deliveries --region us-east-1 \
  --query 'deliveries[].[deliverySourceName,deliveryDestinationArn]' --output table
aws logs describe-delivery-sources --region us-east-1 \
  --query 'deliverySources[].[name,resourceArns[0]]' --output table
```

Each connected distribution has one delivery whose destination ends in `<name_prefix>-firehose`. Disconnecting deletes that delivery, and the source too if the module created it; a source that was there first is left alone.

## What the module creates

For reviewing before you apply it in a real account. Everything is prefixed with `name_prefix` and tagged `ManagedBy = obsero-ingestion`. Source: `aws/ingestion/main.tf` and `aws/ingestion/lambda.tf`.

| Resource | How many | Why it exists |
|---|---|---|
| `aws_kinesis_firehose_delivery_stream.this` | 1 | Buffers log records and POSTs batches to the adapter. Tagged `LogDeliveryEnabled = true` in standard mode — [without it nothing is delivered](#gotchas). |
| `aws_lambda_function.adapter` + `aws_lambda_function_url.adapter` | 1 | Turns each log record into one Obsero event and returns Firehose's ack. [Why a Lambda at all](#why-lambda). |
| `aws_s3_bucket.backup` | 1 | Batches the endpoint never accepted, after `retry_duration`. Expires after `backup_retention_days`. |
| `aws_cloudwatch_log_delivery_destination.this` | 1 (standard) | Points CloudFront standard logging at the Firehose. Shared by every connected distribution; free while idle. |
| `aws_cloudwatch_log_delivery_source.this` | 1 per connected distribution without one | Declares the distribution's access logs as a source. Skipped for distributions in `existing_delivery_sources`. |
| `aws_cloudwatch_log_delivery.this` | 1 per connected distribution | The connection itself: source → destination, with the fields in `standard_log_fields`. |
| IAM roles, CloudWatch log groups | 2 + 2 | Firehose's role (backup bucket, logs, Kinesis in realtime) and the adapter's execution role; one log group each, kept `log_retention_days`. |
| `aws_kinesis_stream.this`, `aws_cloudfront_realtime_log_config.this` | realtime only | The ~$11/month shard and the config you attach to cache behaviours. |

Bad inputs fail at plan time, not as empty events: `terraform_data.validate_config` checks the log field lists and that every `existing_delivery_sources` key is also connected.

## Which log source? The one real decision

CloudFront makes you choose.

|  | `standard` default | `realtime` |
|---|---|---|
| Idle cost | **none** | ~$11/mo per Kinesis shard |
| At 1M req/mo | ~$0.15 | ~$11 |
| Latency | ~1–2 min | seconds |
| Headers | User-Agent, Referer, Host, X-Forwarded-For | **full viewer header set** |
| Web Bot Auth signatures | no | **yes** |
| Client hints (`sec-ch-ua`) | no | **yes** |

Start on `standard`. Move to `realtime` when you want to *verify* an agent rather than take its User-Agent at face value: standard logging v2 has no `cs-headers` field, so `signature-agent` / `signature-input` / `signature` never arrive at all. That is not a tuning problem — it is a different log source.

[GCP has no such trade](gcp.md#headers): it carries named headers on the cheap path.

## Before you trust the numbers

Four behaviours that are easy to mistake for bugs. All measured on a live deployment, all written up in `aws/NOTES.md`.

> **Caution: Real-time logs are best-effort**
>
> CloudFront delivers real-time logs on a best-effort basis; some requests are never written to Kinesis at all. Measured across five runs (188 requests) at `sampling_rate = 100`: four delivered 100%, one delivered 20 of 30. The lost records correlated with nothing — not status code, path, client or latency — and Kinesis `IncomingRecords` confirmed they were never sent rather than lost downstream.
>
> Treat the counts as a high-fidelity sample, not an exact ledger. Do not reconcile them against origin logs or bill directly on them.

> **Note: Delivery is at-least-once**
>
> If any event in a Firehose batch fails to forward, the adapter fails the whole batch so Firehose retries — which duplicates the events that already landed. After `retry_duration` the batch goes to the backup bucket rather than being dropped.

> **Note: Latency floor: 60–90s**
>
> CloudFront flushes real-time logs continuously, but Firehose buffers for at least 60s. That is a floor, not a tuning knob.

> **Warning: The LogDeliveryEnabled tag**
>
> Standard logging v2 writes through the `AWSServiceRoleForLogDelivery` service-linked role, whose policy allows `firehose:PutRecordBatch` **only on streams tagged `LogDeliveryEnabled = true`**. Without the tag CloudFront delivers **nothing**, and there is no error anywhere — the delivery reports healthy, the Firehose reports healthy, and `IncomingRecords` simply stays at zero.
>
> The module applies the tag automatically. Worth knowing if you ever hand-build this.

### Two CloudFront parsing behaviours (realtime only)

Both are handled by the adapter. They are documented because they bite anyone reading the raw Kinesis stream themselves. Standard mode is self-describing JSON, so neither applies there.

1. **Field order is not your field order.** CloudFront emits real-time log fields in its own canonical order, ignoring the order in the log config. Records are tab-separated with no header row, so a positional parser that trusts the config order reads every column shifted. The module sorts `log_fields` into canonical order and hands that same list to the adapter; `log_fields_in_order` exports it.
2. **`cs-uri-stem` includes the query string** in real-time logs, unlike standard access logs where it is path-only. The adapter strips it, or every distinct query string would register as its own path in the analytics.

## Why the adapter Lambda exists at all

Firehose's HTTP endpoint destination cannot post to `/v1/events` directly:

- it wraps records in `{requestId, timestamp, records:[{data}]}`
- it sends the key as `X-Amz-Firehose-Access-Key`, and cannot send arbitrary headers such as `x-obsero-domain` / `x-obsero-key`
- it requires a specific JSON ack back, or treats the delivery as failed

The adapter unwraps the envelope, converts each log line into one Obsero event, strips excluded headers, and returns the ack. Its Function URL is unauthenticated at the AWS layer — Firehose cannot SigV4-sign to a Function URL — so the shared Firehose access key is what the handler checks. The URL is not secret, but is useless without that key.

### Concurrency footprint

The adapter uses **one concurrent Lambda execution per Kinesis shard**. Firehose reads each shard serially and invokes once per buffer window, so at the default `kinesis_shard_count = 1` the steady state is exactly 1 concurrent execution, ~1s per invocation, one invocation per minute. Measured, not estimated.

That matters on accounts with a low concurrency ceiling. New AWS accounts are capped at **10 concurrent executions account-wide**, and at that ceiling `reserved_concurrent_executions` cannot be set at all — AWS requires unreserved concurrency to stay at or above 10, so any reservation is rejected. The adapter therefore shares an unprotected pool with every other function in the account.

If another function saturates the pool, the Function URL returns 429, Firehose treats it as a failed delivery, retries for `retry_duration`, and then writes the batch to the backup bucket. Nothing is lost, but those events do not reach Obsero until the backup is replayed. Watch `Throttles` if you share a small pool.

## Cost

Lambda, CloudFront and S3 usage at demo volume sit inside the AWS free tier. Kinesis Data Streams does not have a free tier — one provisioned shard is about **$11/month** ($0.015/shard-hour) and is `realtime` mode's floor cost, since CloudFront real-time logs can only be delivered to Kinesis. Firehose adds ~$0.029/GB ingested, which is cents at typical volumes.

In `standard` mode there is no Kinesis stream and therefore no idle cost at all; Firehose only reaches $11/month at roughly 380M requests.

## Tearing it down

```
make destroy      # terraform destroy, then check nothing is left standing
make leftovers    # just tell me what is still deployed
```

Both call `destroy.sh`, which empties the S3 buckets first — Terraform refuses to delete a bucket holding objects it did not create, and that is precisely what the backup bucket collects.

The module itself keeps `backup_force_destroy = false`, so a customer's `terraform destroy` will *stop* rather than silently discard events that never reached Obsero. The demo site sets it to `true`, because nobody is going to replay those.

## Code on this page

- [`aws/ingestion/main.tf`](https://github.com/Obsero-ai/obsero-public/blob/main/aws/ingestion/main.tf) -- the pipeline: Firehose, log deliveries, backup bucket
- [`aws/ingestion/variables.tf`](https://github.com/Obsero-ai/obsero-public/blob/main/aws/ingestion/variables.tf) -- every module input, with its validation
- [`aws/ingestion/lambda/adapter/index.mjs`](https://github.com/Obsero-ai/obsero-public/blob/main/aws/ingestion/lambda/adapter/index.mjs) -- the adapter Lambda
- [`aws/site/terraform/main.tf`](https://github.com/Obsero-ai/obsero-public/blob/main/aws/site/terraform/main.tf) -- the demo stack: mock site plus one module block
- [`aws/NOTES.md`](https://github.com/Obsero-ai/obsero-public/blob/main/aws/NOTES.md) -- behaviour measured on live deployments
- [`aws/AGENTS.md`](https://github.com/Obsero-ai/obsero-public/blob/main/aws/AGENTS.md) -- the AWS playbook for coding agents