Promtail everywhere, and good labels

The agent (Promtail) runs on every host, and what it ships — and how it's labelled — is what makes the logs useful. A label is how you slice the haystack later.

Across the fleet, Promtail ships and labels:

host           shipped streams                         key labels
-------------  -------------------------------------   --------------------
all VMs        syslog, auth                            host, job
k8s nodes      pod logs (/var/log/pods/*)              host, namespace, pod, container
               kubelet, containerd (journald)          host, unit
databases      error + slow-query logs                 host, job
edge/proxy     HAProxy / access logs (+ geo enrich)    host, status, method, ...

The standout is the Kubernetes pod logs: by parsing the log file path, every pod line gets labelled with its namespace, pod, and container. That's what lets you ask "show me everything from the kong-gateway namespace in the last hour" and actually get an answer.

Lesson learned: logs are only as useful as their labels. Shipping raw text to one place is a start; labelling it by host, service, namespace, and severity is what turns a pile of lines into something you can query. Spend the effort on the labels — it's where the payoff is.