Quickstart

One command. Deploy a mock site, build a real ingestion pipeline, connect the two, watch classified traffic arrive, then tear it all down. Or skip the mock site and connect a site you already run.

#Before you start

setup.sh checks all of this for you and tells you exactly what is missing before it creates anything. On GCP it also checks the required APIs and offers to enable them — but not the IAM roles, which are the thing most likely to bite. Check those first →

#Your checklist

Everything you do, in order, for AWS. Each item says what to run and what you should see; if you see something else, stop there — the item after it will not work either. The steps in detail explains what each one builds.

1. Install the tools

   terraform -version     # v1.5 or newer
   node --version         # v18 or newer
   aws --version          # any v2

   Missing one? setup.sh names it and links the installer, so you can also just run it and see.

2. Log the AWS CLI in

   aws sts get-caller-identity

   You should see your account number and an ARN. An error means the CLI has no credentials: run aws configure, or export AWS_PROFILE=<name> if you use named profiles. Everything below runs as this identity, in this account.

3. Get the code

   git clone https://github.com/Obsero-ai/obsero-public.git
   cd obsero-public

4. Check your tracking ID

   The site token Obsero issued for your domain. It is sent as x-obsero-key on every event and is what proves the domain is yours. Check it before building anything — a rejected ID makes the pipeline forward into nothing, silently:

   node obsero.mjs --token <tracking-id> --domain <your-domain>

   You should see the event accepted. Then open the menu — everything from here is a number typed into it:

   ./setup.sh aws

5. Press 1 — deploy the mock site

   Takes 3–5 minutes. You should see Site is up and a https://….cloudfront.net URL. Open it: a small site with a few pages. Skip this item if you are connecting your own site.

6. Press 2 — deploy the pipeline

   Paste your tracking ID and press Enter; each character shows as *, then it confirms the first and last four. Answer y to the test event. You should see Obsero accepted it, then, 2–3 minutes later, Pipeline is live with connected: nothing yet. That is expected — the next item connects it.

7. Press 3 — connect

   A numbered list of every CloudFront distribution in the account. Type the number next to the mock site, press Enter to see it ticked, press Enter on an empty line, then y. You should see Connected and the distribution ID.

8. Press 5 — send traffic and score it

   Sends requests as 19 client personas, waits about 90 seconds for delivery, then scores how many reached Obsero with their headers intact. You should see every request correlated and no header loss. Some missing right after connecting is the 60–90s latency floor: run it again.

9. Press 4 when you are finished

   Type destroy to confirm. It removes everything setup.sh created and then checks nothing is left billing. Distributions it did not create are only disconnected, never deleted.

Press 6 at any point to see what is deployed and what is connected.

#You're done when

• ./setup.sh aws status shows the pipeline deployed and at least one distribution connected.
• make -C aws logs shows the adapter forwarding events within 90 seconds of a request to a connected site.
• The traffic score correlates every request — the measured baseline is 78 of 78 on AWS, with zero header loss.
• Events show up in Obsero under your domain, with AI agents and crawlers told apart from browsers.

Stuck on one of these? Troubleshooting has the order to check things in.

#Use your own site instead of the mock

The mock site is only there so you have traffic to look at. If you already run a site on CloudFront, connect that instead — the pipeline is the same either way.

./setup.sh aws   # then:
2                # the pipeline, with your tracking ID
3                # tick your distribution, listed by domain

Or, without the menu:

./setup.sh aws pipeline --token <tracking-id>
./setup.sh aws connect --dist <distribution-id>

• Which headers arrive. Standard logging carries User-Agent, Referer, Host and X-Forwarded-For — enough to tell most agents apart, not enough to verify Web Bot Auth signatures. The one real decision explains the trade.
• What it costs. No idle cost; Firehose bills per GB, which is cents at typical volumes. Cost →
• Permissions. Your AWS identity needs to list CloudFront distributions and create CloudWatch Logs deliveries, in us-east-1 — where CloudFront logging always lives.
• Stopping. Run 3 again and untick it, or ./setup.sh aws connect --dist none. Destroying the pipeline disconnects everything too.

#Two checks that cost nothing

Both of these pass or fail in seconds and rule out most of what goes wrong later. Run them before you create any infrastructure.

cd aws            # or gcp
make test         # adapter payload vs the contract — no cloud account needed
make event TOKEN=<tracking-id> DOMAIN=acme.com

make test feeds sample log lines through the real adapter parser and asserts the result against validateEvent. make event POSTs one hand-made event, so you know your tracking ID is accepted before anything depends on it.

#The steps in detail

./setup.sh

It asks which cloud, then shows this:

Obsero ingestion -- aws

  1) Deploy the mock site          a site worth collecting logs from
  2) Deploy the ingestion pipeline asks for your tracking ID
  3) Connect                       pick distributions already in your account
  4) Destroy everything            the rig bills while it is up

  5) Send mock traffic and score what arrived
  6) Status -- what is deployed right now
  7) Switch cloud
  q) Quit

1. Deploy the mock site

   A static site with real pages, behind a real CDN, so there is traffic worth logging. On AWS that is a private S3 bucket behind CloudFront; on GCP it is the same pages served from Cloud Run behind a global external Application Load Balancer with Cloud CDN in front.

   Nothing is sent to Obsero at this point, and no tracking ID is needed. The step prints the site URL when it finishes — open it.

   ==> Site is up
      https://d1234abcd.cloudfront.net
   
      Open it. Nothing is being sent to Obsero yet -- step 2 builds the pipeline,
      step 3 connects this site (or any other distribution) to it.

   Budget 3–5 minutes on AWS (CloudFront is the slow part) and 5–8 on GCP, which also builds a container image on the way.

2. Deploy the ingestion pipeline

   This is the step that needs your tracking ID — the site token Obsero issued you, sent as the x-obsero-key header on every forwarded event. Each character shows as * so you can see a paste landed, and the first and last four are echoed back to check. It is saved to <cloud>/site/terraform/terraform.tfvars, which is gitignored and chmod 600, because terraform destroy and every later apply need it too.

   ==> Your Obsero tracking ID
      The site token Obsero issued you. It is sent as the x-obsero-key header
      on every forwarded event, and is what proves the domain is yours.
   
      Tracking ID (paste, then Enter): ********************************
      got 32 characters: wuJ7...oF_E
    v using wuJ7...oF_E
   
   ==> Collecting from
      nothing yet -- the pipeline is built idle, and step 3 connects it
      to any CloudFront distribution in your account, the mock site included
   
      Send one test event first, to check the ID is accepted? [y/N]

   Say yes. A rejected tracking ID means the pipeline forwards into nothing, silently — better to find out in one second than after a ten-minute apply. Then the pipeline goes up: on AWS, CloudFront standard logs into Firehose into an adapter Lambda; on GCP, load balancer request logs through a Log Router sink into Pub/Sub into a Cloud Run adapter.

   On AWS the pipeline does not need the mock site and is built collecting from nothing; step 3 decides where it collects from. On GCP it attaches to the mock site's backend service here, and builds the site too if step 1 has not run.

   Give it a minute before you judge it

   AWS: first events land in 60–90 seconds. CloudFront flushes continuously but Firehose buffers for at least 60s, and that is a floor, not a setting.

   GCP: the header allow-list takes several minutes to reach the edge. Events that arrive headerless right after an apply are propagation, not breakage.

3. Connect (AWS)

   Lists every CloudFront distribution in your account — the mock site and anything you already run — and lets you tick the ones the pipeline collects from. Untick one to disconnect it.

   ==> CloudFront distributions in this account
   
      1) [x] E1A2B3C4D5E6F7  d1234abcd.cloudfront.net
                the mock site
      2) [ ] E9Z8Y7X6W5V4U3  acme.com,www.acme.com
                reuses its existing logging source "CreatedByCloudFront-E9Z8Y7X6W5V4U3"
   
      numbers toggle (e.g. "1 3"), a = all, n = none, Enter = done, q = cancel
   Connect: 2

   Your distributions are never modified. Connecting adds a CloudFront standard logging v2 delivery from the distribution into the pipeline's Firehose. A distribution can have only one standard logging v2 source, so if it already has one — the CloudFront console creates it the first time anyone turns that logging on — it is reused and left in place on disconnect.

   The picks are saved as connected_distribution_arns in terraform.tfvars, so every later apply keeps them.

4. Destroy everything

   Terraform does the work; the wrapper handles what it cannot — emptying S3 buckets whose contents Terraform does not own, passing each stack the credentials it needs, and then checking the cloud afterwards for anything still standing. A lost state file or a half-finished destroy leaves resources that no terraform plan will ever mention.

   ./destroy.sh --check      # just tell me what is still deployed
   ./destroy.sh --dry-run    # show the plan, touch nothing
   ./destroy.sh aws          # one cloud
   ./destroy.sh --sweep      # delete leftovers too, not just report them

   CloudFront is the one exception: a distribution has to be disabled, left to propagate, and deleted minutes later, so the script reports it rather than pretending it can be done in one pass.

#Watch it work

Option 5 sends a reproducible mix of traffic as 19 client personas — each carrying the headers that client really sends — waits out the delivery latency, then scores what actually arrived.

./setup.sh aws traffic -n 40

Every request is tagged ?mt=<runId>-<seq>. The scorer reads the records back, replays them through the real adapter parser, correlates each tag, and diffs headers sent against headers received. Measured baselines: 78/78 correlated on AWS, 40/40 on GCP, zero header loss on either.

cd aws
make traffic N=40                    # weighted mix
make traffic N=20 KIND=ai-agent      # one class
make traffic N=10 ONLY=gptbot SEED=7 # specific, reproducible
make personas                        # list them

#Without the menu

Every menu item has a flag form, for CI or for driving it from a script:

./setup.sh aws site
./setup.sh aws pipeline --token <tracking-id>
./setup.sh aws connect --dist E1A2B3C4D5E6F7   # exactly these; --dist none clears
./setup.sh aws connect --dist +E9Z8Y7X6W5V4U3  # add one to what is connected
./setup.sh aws traffic -n 40
./setup.sh aws status
./setup.sh aws destroy --yes

Or skip setup.sh entirely and use the make targets directly — make with no argument in either cloud directory lists them all. make deploy is steps 1–3 for the mock site in one go; make e2e is deploy, traffic, wait, score.

#What setup.sh is actually doing

Worth knowing, because it explains the one thing that looks odd: the site, the pipeline and its connections are one Terraform stack, not three. Step 1 is a targeted apply of the site resources only. On AWS, step 2 is a targeted apply of the module.ingestion block, which does not reference the site, and step 3 re-applies that same module with a new connected_distribution_arns list. On GCP, step 2 is the plain untargeted apply.

Because both stacks declare obsero_site_token with no default, step 1 writes the placeholder set-in-step-2 into terraform.tfvars so it can plan at all. Step 2 refuses to apply while that is still the value.