# Quickstart One command. Deploy a mock site, build a real ingestion pipeline, connect the two, watch classified traffic arrive, then tear it all down. Or skip the mock site and connect a site you already run. > **Warning: This costs real money** > > The demo rigs bill while they are up. The GCP forwarding rule is about **$18/month whether or not any traffic hits it**. Run step 4 when you are done — not tomorrow. ## Before you start | You need | Both | AWS | GCP | |---|---|---|---| | Terraform ≥ 1.5 | ✓ | | | | Node ≥ 18 | ✓ | | | | Authenticated CLI | | `aws configure` | `gcloud auth login` | | Docker | | — | ✓ images are built locally | | A tracking ID from Obsero | ✓ | | | `setup.sh` checks all of this for you and tells you exactly what is missing before it creates anything. On GCP it also checks the required APIs and offers to enable them — but **not** the IAM roles, which are the thing most likely to bite. [Check those first →](gcp.md#permissions) ## Your checklist Everything you do, in order, for AWS. Each item says what to run and what you should see; if you see something else, stop there — the item after it will not work either. [The steps in detail](#the-steps) explains what each one builds. 1. ### Install the tools ``` terraform -version # v1.5 or newer node --version # v18 or newer aws --version # any v2 ``` Missing one? `setup.sh` names it and links the installer, so you can also just run it and see. 2. ### Log the AWS CLI in ``` aws sts get-caller-identity ``` **You should see** your account number and an ARN. An error means the CLI has no credentials: run `aws configure`, or `export AWS_PROFILE=` if you use named profiles. Everything below runs as this identity, in this account. 3. ### Get the code ``` git clone https://github.com/Obsero-ai/obsero-public.git cd obsero-public ``` 4. ### Check your tracking ID The site token Obsero issued for your domain. It is sent as `x-obsero-key` on every event and is what proves the domain is yours. Check it before building anything — a rejected ID makes the pipeline forward into nothing, silently: ``` node obsero.mjs --token --domain ``` **You should see** the event accepted. Then open the menu — everything from here is a number typed into it: ``` ./setup.sh aws ``` 5. ### Press `1` — deploy the mock site Takes 3–5 minutes. **You should see** `Site is up` and a `https://….cloudfront.net` URL. Open it: a small site with a few pages. Skip this item if you are [connecting your own site](#your-own-site). 6. ### Press `2` — deploy the pipeline Paste your tracking ID and press Enter; each character shows as `*`, then it confirms the first and last four. Answer `y` to the test event. **You should see** `Obsero accepted it`, then, 2–3 minutes later, `Pipeline is live` with `connected: nothing yet`. That is expected — the next item connects it. 7. ### Press `3` — connect A numbered list of every CloudFront distribution in the account. Type the number next to `the mock site`, press Enter to see it ticked, press Enter on an empty line, then `y`. **You should see** `Connected` and the distribution ID. 8. ### Press `5` — send traffic and score it Sends requests as 19 client personas, waits about 90 seconds for delivery, then scores how many reached Obsero with their headers intact. **You should see** every request correlated and no header loss. Some missing right after connecting is the 60–90s latency floor: run it again. 9. ### Press `4` when you are finished Type `destroy` to confirm. It removes everything `setup.sh` created and then checks nothing is left billing. Distributions it did not create are only disconnected, never deleted. Press `6` at any point to see what is deployed and what is connected. ## You're done when - `./setup.sh aws status` shows the pipeline **deployed** and at least one distribution **connected**. - `make -C aws logs` shows the adapter forwarding events within 90 seconds of a request to a connected site. - The traffic score correlates every request — the measured baseline is 78 of 78 on AWS, with zero header loss. - Events show up in Obsero under your domain, with AI agents and crawlers told apart from browsers. Stuck on one of these? [Troubleshooting](troubleshooting.md#order) has the order to check things in. ## Use your own site instead of the mock The mock site is only there so you have traffic to look at. If you already run a site on CloudFront, connect that instead — the pipeline is the same either way. ``` ./setup.sh aws # then: 2 # the pipeline, with your tracking ID 3 # tick your distribution, listed by domain ``` Or, without the menu: ``` ./setup.sh aws pipeline --token ./setup.sh aws connect --dist ``` > **Note: What happens to your distribution: nothing** > > Connecting never edits the distribution's configuration. It adds a CloudFront *standard logging v2* delivery that copies each request's log record into the pipeline's Firehose. Traffic, caching and latency are untouched. If the distribution already uses standard logging v2 — to S3, say — that keeps working: its existing logging source is reused, and left in place when you disconnect. - **Which headers arrive.** Standard logging carries User-Agent, Referer, Host and X-Forwarded-For — enough to tell most agents apart, not enough to *verify* Web Bot Auth signatures. [The one real decision](aws.md#log-source) explains the trade. - **What it costs.** No idle cost; Firehose bills per GB, which is cents at typical volumes. [Cost →](aws.md#cost) - **Permissions.** Your AWS identity needs to list CloudFront distributions and create CloudWatch Logs deliveries, in `us-east-1` — where CloudFront logging always lives. - **Stopping.** Run `3` again and untick it, or `./setup.sh aws connect --dist none`. Destroying the pipeline disconnects everything too. ## Two checks that cost nothing Both of these pass or fail in seconds and rule out most of what goes wrong later. Run them before you create any infrastructure. ``` cd aws # or gcp make test # adapter payload vs the contract — no cloud account needed make event TOKEN= DOMAIN=acme.com ``` `make test` feeds sample log lines through the real adapter parser and asserts the result against `validateEvent`. `make event` POSTs one hand-made event, so you know your tracking ID is accepted before anything depends on it. ## The steps in detail ``` ./setup.sh ``` It asks which cloud, then shows this: ``` Obsero ingestion -- aws 1) Deploy the mock site a site worth collecting logs from 2) Deploy the ingestion pipeline asks for your tracking ID 3) Connect pick distributions already in your account 4) Destroy everything the rig bills while it is up 5) Send mock traffic and score what arrived 6) Status -- what is deployed right now 7) Switch cloud q) Quit ``` 1. ### Deploy the mock site A static site with real pages, behind a real CDN, so there is traffic worth logging. On AWS that is a private S3 bucket behind CloudFront; on GCP it is the same pages served from Cloud Run behind a global external Application Load Balancer with Cloud CDN in front. Nothing is sent to Obsero at this point, and no tracking ID is needed. The step prints the site URL when it finishes — open it. ``` ==> Site is up https://d1234abcd.cloudfront.net Open it. Nothing is being sent to Obsero yet -- step 2 builds the pipeline, step 3 connects this site (or any other distribution) to it. ``` Budget 3–5 minutes on AWS (CloudFront is the slow part) and 5–8 on GCP, which also builds a container image on the way. 2. ### Deploy the ingestion pipeline This is the step that needs your tracking ID — the site token Obsero issued you, sent as the `x-obsero-key` header on every forwarded event. Each character shows as `*` so you can see a paste landed, and the first and last four are echoed back to check. It is saved to `/site/terraform/terraform.tfvars`, which is gitignored and `chmod 600`, because `terraform destroy` and every later apply need it too. ``` ==> Your Obsero tracking ID The site token Obsero issued you. It is sent as the x-obsero-key header on every forwarded event, and is what proves the domain is yours. Tracking ID (paste, then Enter): ******************************** got 32 characters: wuJ7...oF_E v using wuJ7...oF_E ==> Collecting from nothing yet -- the pipeline is built idle, and step 3 connects it to any CloudFront distribution in your account, the mock site included Send one test event first, to check the ID is accepted? [y/N] ``` Say yes. A rejected tracking ID means the pipeline forwards into nothing, silently — better to find out in one second than after a ten-minute apply. Then the pipeline goes up: on AWS, CloudFront standard logs into Firehose into an adapter Lambda; on GCP, load balancer request logs through a Log Router sink into Pub/Sub into a Cloud Run adapter. On AWS the pipeline does not need the mock site and is built collecting from nothing; step 3 decides where it collects from. On GCP it attaches to the mock site's backend service here, and builds the site too if step 1 has not run. > **Note: Give it a minute before you judge it** > > **AWS:** first events land in 60–90 seconds. CloudFront flushes continuously but Firehose buffers for at least 60s, and that is a floor, not a setting. > > **GCP:** the header allow-list takes several minutes to reach the edge. Events that arrive headerless right after an apply are propagation, not breakage. 3. ### Connect (AWS) Lists every CloudFront distribution in your account — the mock site and anything you already run — and lets you tick the ones the pipeline collects from. Untick one to disconnect it. ``` ==> CloudFront distributions in this account 1) [x] E1A2B3C4D5E6F7 d1234abcd.cloudfront.net the mock site 2) [ ] E9Z8Y7X6W5V4U3 acme.com,www.acme.com reuses its existing logging source "CreatedByCloudFront-E9Z8Y7X6W5V4U3" numbers toggle (e.g. "1 3"), a = all, n = none, Enter = done, q = cancel Connect: 2 ``` Your distributions are never modified. Connecting adds a CloudFront standard logging v2 delivery from the distribution into the pipeline's Firehose. A distribution can have only one standard logging v2 source, so if it already has one — the CloudFront console creates it the first time anyone turns that logging on — it is reused and left in place on disconnect. The picks are saved as `connected_distribution_arns` in `terraform.tfvars`, so every later apply keeps them. 4. ### Destroy everything Terraform does the work; the wrapper handles what it cannot — emptying S3 buckets whose contents Terraform does not own, passing each stack the credentials it needs, and then **checking the cloud afterwards** for anything still standing. A lost state file or a half-finished destroy leaves resources that no `terraform plan` will ever mention. ``` ./destroy.sh --check # just tell me what is still deployed ./destroy.sh --dry-run # show the plan, touch nothing ./destroy.sh aws # one cloud ./destroy.sh --sweep # delete leftovers too, not just report them ``` CloudFront is the one exception: a distribution has to be disabled, left to propagate, and deleted minutes later, so the script reports it rather than pretending it can be done in one pass. ## Watch it work Option 5 sends a reproducible mix of traffic as 19 client personas — each carrying the headers that client really sends — waits out the delivery latency, then scores what actually arrived. ``` ./setup.sh aws traffic -n 40 ``` Every request is tagged `?mt=-`. The scorer reads the records back, replays them through the **real** adapter parser, correlates each tag, and diffs headers sent against headers received. Measured baselines: 78/78 correlated on AWS, 40/40 on GCP, zero header loss on either. | kind | personas | |---|---| | `ai-agent` | ChatGPT-User, Claude-User, Perplexity-User, OAI-SearchBot, DuckAssistBot | | `ai-crawler` | GPTBot, ClaudeBot, PerplexityBot, Meta-ExternalAgent, Bytespider, CCBot, Amazonbot | | `human` | Chrome/Windows, Safari/iPhone, Firefox/macOS, Chrome/Android | | `search-bot` | Googlebot, Bingbot, Applebot | ``` cd aws make traffic N=40 # weighted mix make traffic N=20 KIND=ai-agent # one class make traffic N=10 ONLY=gptbot SEED=7 # specific, reproducible make personas # list them ``` ## Without the menu Every menu item has a flag form, for CI or for driving it from a script: ``` ./setup.sh aws site ./setup.sh aws pipeline --token ./setup.sh aws connect --dist E1A2B3C4D5E6F7 # exactly these; --dist none clears ./setup.sh aws connect --dist +E9Z8Y7X6W5V4U3 # add one to what is connected ./setup.sh aws traffic -n 40 ./setup.sh aws status ./setup.sh aws destroy --yes ``` Or skip `setup.sh` entirely and use the make targets directly — `make` with no argument in either cloud directory lists them all. `make deploy` is steps 1–3 for the mock site in one go; `make e2e` is deploy, traffic, wait, score. ## What setup.sh is actually doing Worth knowing, because it explains the one thing that looks odd: the site, the pipeline and its connections are **one Terraform stack**, not three. Step 1 is a targeted apply of the site resources only. On AWS, step 2 is a targeted apply of the `module.ingestion` block, which does not reference the site, and step 3 re-applies that same module with a new `connected_distribution_arns` list. On GCP, step 2 is the plain untargeted apply. Because both stacks declare `obsero_site_token` with no default, step 1 writes the placeholder `set-in-step-2` into `terraform.tfvars` so it can plan at all. Step 2 refuses to apply while that is still the value. ## Code on this page - [`setup.sh`](https://github.com/Obsero-ai/obsero-public/blob/main/setup.sh) -- the interactive installer this page walks through - [`destroy.sh`](https://github.com/Obsero-ai/obsero-public/blob/main/destroy.sh) -- teardown, and the sweep for leftovers - [`aws/Makefile`](https://github.com/Obsero-ai/obsero-public/blob/main/aws/Makefile) -- every AWS task, one make target each - [`gcp/Makefile`](https://github.com/Obsero-ai/obsero-public/blob/main/gcp/Makefile) -- every GCP task, one make target each