Skip to main content

Lesson: The Collector

What you'll learn

  • What a collector / agent does and why you need one between your systems and the backends.
  • How an agent scrapes metrics and tails/ships logs.
  • The building blocks of a Grafana Alloy pipeline: components and references.
  • How to read and write a small Alloy config that scrapes metrics and ships logs to Loki.
  • Why Alloy is replacing older single-purpose agents like Promtail.

By the end you'll be able to read an Alloy config and describe how data flows from a host to Grafana.

The lesson

1. Why you need a collector at all

Your apps and servers produce telemetry, but they rarely send it to the backend themselves. Something has to: gather it, label it, batch it, retry on failure, and forward it to the right place. That "something" is a collector (also called an agent) — a small program running on each host (or as a DaemonSet pod on each Kubernetes node) whose only job is to collect telemetry and ship it onward.

  HOST / K8s NODE
  +-------------------------------------------+
  |  app  ->  /metrics endpoint  --.          |
  |  app  ->  log files  ----------.\          |
  |                                 vv         |
  |              GRAFANA ALLOY (the collector) |
  |       scrape | relabel | batch | ship      |
  +-----------------|--------------------|------+
                    | metrics            | logs
                    v                    v
              Mimir / InfluxDB        Loki (10.100.100.5)
                    \                  /
                     v                v
                   GRAFANA (10.100.100.4)

Without an agent, every app would need to know the address of every backend, handle retries, and re-implement labeling. The agent centralizes all of that.

2. Scraping metrics vs shipping logs

Collectors handle the two main pillars differently:

Metrics — pull (scrape). Many systems expose a /metrics HTTP endpoint in Prometheus text format. The agent scrapes it: every 15 seconds (a common interval) it does GET /metrics, reads the current numbers, and forwards them. The agent pulls on a schedule.

http_requests_total{method="GET",status="200"} 48211
process_cpu_seconds_total 1394.2
go_goroutines 42

Logs — push (tail). Logs are a stream of events, so the agent tails log files (or reads the systemd journal, or container stdout) and pushes each new line to Loki as it appears. It also attaches labels (job, host, namespace) so you can filter them later in LogQL.

The lab currently uses Promtail to tail logs and ship them to Loki at 10.100.100.5. Promtail does logs only. Grafana Alloy is the modern replacement that does logs and metrics and traces in one agent — which is why we teach Alloy here.

3. What Grafana Alloy is

Grafana Alloy is a single, vendor-neutral collector that can scrape metrics, tail logs, and receive traces, then forward them to Loki, Mimir, Tempo, or any OpenTelemetry-compatible backend. You configure it with a declarative file made of components.

A component is a named block that does one thing (scrape, relabel, write). Components are wired together by references: one component's output becomes another's input. This forms a pipeline.

  discover  -->  scrape  -->  process/relabel  -->  write (remote)

The config syntax looks like this (Alloy's "River"-style configuration language):

// A component is: block_type "label" { arguments }
some_component "my_name" {
  argument = value
  forward_to = [other.component.receiver]   // a reference
}

4. A small Alloy pipeline — metrics

Here is a minimal pipeline that scrapes a local app's /metrics endpoint and writes the metrics to a Mimir/Prometheus-compatible backend:

// 1. Define WHAT to scrape (a static target on this host).
prometheus.scrape "myapp" {
  targets = [
    { "__address__" = "127.0.0.1:8080" },
  ]
  scrape_interval = "15s"
  forward_to      = [prometheus.remote_write.default.receiver]
}

// 2. Define WHERE to send the scraped metrics.
prometheus.remote_write "default" {
  endpoint {
    url = "http://mimir.example.com/api/v1/push"
  }
}

Read it top to bottom: prometheus.scrape "myapp" pulls /metrics from 127.0.0.1:8080 every 15s and forwards the result to prometheus.remote_write.default.receiver, which pushes it to the backend. The .receiver is the input handle that components expose for others to forward into.

5. A small Alloy pipeline — logs to Loki

Now the logs side, shipping to the lab Loki at 10.100.100.5:

// 1. Find log files to read.
local.file_match "applogs" {
  path_targets = [
    { "__path__" = "/var/log/myapp/*.log", "job" = "myapp" },
  ]
}

// 2. Tail those files.
loki.source.file "applogs" {
  targets    = local.file_match.applogs.targets
  forward_to = [loki.process.add_labels.receiver]
}

// 3. Optionally parse/relabel lines.
loki.process "add_labels" {
  stage.logfmt {}                 // split key=value fields
  stage.labels { values = { level = "" } }   // promote 'level' to a label
  forward_to = [loki.write.lab.receiver]
}

// 4. Ship to Loki.
loki.write "lab" {
  endpoint {
    url = "http://10.100.100.5:3100/loki/api/v1/push"
  }
}

The data flows file_match → source.file → process → write. Each arrow is a forward_to reference. Labels like job="myapp" and level are what you later filter on in LogQL ({job="myapp"} | level="error").

6. Relabeling — small but important

Relabeling rewrites or drops labels before data is stored. It keeps cardinality under control and adds useful context. A common example: keep only the labels you need, and drop noisy targets.

discovery.relabel "trim" {
  targets = local.file_match.applogs.targets
  rule {
    source_labels = ["__path__"]
    target_label  = "filename"
  }
}

Done well, relabeling is how you avoid the high-cardinality trap from the first lesson — you decide which labels are worth keeping.

7. Running Alloy and seeing it work

Alloy runs as a service (a systemd unit on a VM, or a DaemonSet in Kubernetes) and ships with a built-in web UI (default port 12345) where you can see every component, its health, and the live data flowing through. After editing the config you reload it (systemctl reload alloy or sending it a reload). Then open Grafana at 10.100.100.4 → Explore → Loki and query {job="myapp"} — if your lines appear, your pipeline works end to end.

In the capstone you'll rely on exactly this path: your app on the k8s cluster emits logs, Alloy (or Promtail) tails them, Loki at 10.100.100.5 stores them, and Grafana queries them.

Dig deeper

Search terms

  • grafana alloy tutorial getting started
  • grafana alloy loki.write config example
  • prometheus scrape metrics endpoint explained
  • promtail vs grafana alloy
  • alloy forward_to component reference
  • prometheus relabeling cardinality control

Check yourself

  1. Why does an architecture put a collector/agent between apps and the backends, instead of having apps push directly?
  2. Explain the difference between scraping metrics and tailing logs (pull vs push).
  3. In an Alloy config, what does forward_to do, and what is a .receiver?
  4. Which lab URL would a loki.write component point at, and on what port?
  5. What problem does relabeling help you avoid, and how?

No comments to display

Back to top