Lessons on data and observability

• Match storage to the workload: databases get local disk on dedicated VMs, never NFS.
• Tune for the box you have: give the engine the RAM, tell it the disk is SSD, log slow queries.
• MariaDB ≠ MySQL: running both proved it. Check settings against your actual engine (the innodb_log_file_size vs innodb_redo_log_capacity trap) — and check your config's load order too (z99- beats 99- because the include dir loads by filename).
• Per-app credentials, always: own database, own user, own password; admin accounts stay local-only.
• Multi-line logs need a multiline parser: a slow-query entry spans several lines, so each multi-line source gets its own Promtail job with a multiline stage — never globbed with single-line files. And mind file permissions: MySQL writes its slow log owned by the mysql group, so the log shipper shipped nothing until it was added to that group. A log you can't read is a log you don't have.
• Ship every log to one place: node logs, pod logs, and all three databases' logs land in Loki and surface in Grafana. Observability you set up before you need it is the kind that's actually there when you do.