feat: AI VM stack (WP-12) and Alloy telemetry on k8s, legacy and DB VMs (WP-15)

WP-12(실험 설계서 6장) vm/ai-env: apply.sh(시크릿 4종을 Secret Manager에서 파일로, 미등록이면
성능 저하 모드로 기동), compose(agent 8080·hitl 8081·rag-db pgvector 0.8.7-pg16), ai.env(Discord
서버 Selfheal Lab 1557424985793101824, approvals 1557425140302872698, escalation 1557425282363691048,
OPENAI_MODEL gpt-6.1-sol 후보), 프롬프트 3종, 러너북 11종(조치별·토폴로지·3층 절차).

WP-15(실험 설계서 4.1·4.3): apps/alloy.yaml(Alloy Helm 1.13.0 DaemonSet, 클러스터링으로 KSM·Argo CD·
cadvisor 스크랩 분산, demo 등 파드 로그 → Loki), apps/kube-state-metrics.yaml(Helm 8.6.0).
legacy-api compose에 Alloy 사이드카(up{job="legacy-api"}, 컨테이너 로그). DB VM은
vm/postgres-common/alloy.sh로 apt 설치 + prometheus.exporter.postgres(pg_replication_lag_seconds)
+ custom query pg_is_wal_replay_paused + PostgreSQL 로그.

vm/harness/apply.sh: 준비 대기 루프의 curl에 || true(러너 재기동 순간 curl 56으로 중단되던 문제,
2026-10-08 harness-sync 실측).

vm/postgres-common/alloy.sh

@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+# Alloy on the PostgreSQL VMs (WP-15, experiment design 4.1 / 4.3). Called by vm/postgres-{primary,replica}/apply.sh.
+# The harness role password is read from Secret Manager (selfheal-db-password) through the VM service account. Installs Alloy from the Grafana apt repository (systemd service `alloy`,
+# config /etc/alloy/config.alloy, environment /etc/default/alloy; https://grafana.com/docs/alloy/latest/configure/linux/),
+# exposes the built-in postgres exporter (pg_replication_lag_seconds) plus a custom query for pg_is_wal_replay_paused
+# (not in the upstream replication collector), and ships /var/log/postgresql to Loki. Idempotent.
+set -euo pipefail
+HERE="$(cd "$(dirname "$0")" && pwd)"
+if ! command -v alloy >/dev/null; then
+  mkdir -p /etc/apt/keyrings
+  wget -qO /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
+  chmod 644 /etc/apt/keyrings/grafana.asc
+  echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" > /etc/apt/sources.list.d/grafana.list
+  apt-get update -qq
+  apt-get install -y -qq alloy >/dev/null
+fi
+usermod -aG adm alloy   # /var/log/postgresql/*.log is postgres:adm 0640
+install -d -m 755 /etc/alloy
+install -m 644 "$HERE/config.alloy" /etc/alloy/config.alloy
+install -m 644 "$HERE/queries.yaml" /etc/alloy/queries.yaml
+PROJECT="$(curl -sf -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/project/project-id)"
+TOKEN="$(curl -sf -H 'Metadata-Flavor: Google' 'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token' | jq -r .access_token)"
+PGPASSWORD="$(curl -sf -H "Authorization: Bearer ${TOKEN}" "https://secretmanager.googleapis.com/v1/projects/${PROJECT}/secrets/selfheal-db-password/versions/latest:access" | jq -r .payload.data | base64 -d)"
+unset TOKEN
+[ -n "$PGPASSWORD" ] || { echo "empty DB password" >&2; exit 1; }
+IP="$(hostname -I | awk '{print $1}')"
+umask 077
+cat > /etc/default/alloy <<ENV
+CONFIG_FILE="/etc/alloy/config.alloy"
+CUSTOM_ARGS="--server.http.listen-addr=127.0.0.1:12345 --storage.path=/var/lib/alloy/data"
+INSTANCE="$(hostname)"
+PG_DSN="postgresql://harness:${PGPASSWORD}@${IP}:5432/app?sslmode=require"
+ENV
+unset PGPASSWORD
+systemctl enable alloy >/dev/null 2>&1 || true
+systemctl restart alloy
+for i in $(seq 1 20); do
+  c=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:12345/-/ready || true)
+  [ "$c" = "200" ] && break; sleep 3
+done
+echo "alloy ready=$c"

vm/postgres-common/config.alloy

@@ -0,0 +1,36 @@
+// Alloy on db-primary / db-replica (design 4.1 / 4.3): postgres exporter (built-in collectors: pg_replication_lag_seconds,
+// pg_up, pg_stat_*) + custom pg_is_wal_replay_paused -> Mimir; PostgreSQL logs -> Loki. instance = VM name.
+prometheus.exporter.postgres "db" {
+  data_source_names         = [sys.env("PG_DSN")]
+  custom_queries_config_path = "/etc/alloy/queries.yaml"
+  disable_settings_metrics  = true
+}
+
+prometheus.scrape "db" {
+  job_name        = "postgres"
+  scrape_interval = "15s"
+  targets         = prometheus.exporter.postgres.db.targets
+  forward_to      = [prometheus.relabel.instance.receiver]
+}
+
+prometheus.relabel "instance" {
+  forward_to = [prometheus.remote_write.mimir.receiver]
+  rule { target_label = "instance", replacement = sys.env("INSTANCE") }
+}
+
+prometheus.remote_write "mimir" {
+  endpoint { url = "http://10.20.0.10:9009/api/v1/push" }
+}
+
+local.file_match "pglogs" {
+  path_targets = [{ "__path__" = "/var/log/postgresql/*.log", "job" = "postgres", "instance" = sys.env("INSTANCE") }]
+}
+
+loki.source.file "pglogs" {
+  targets    = local.file_match.pglogs.targets
+  forward_to = [loki.write.loki.receiver]
+}
+
+loki.write "loki" {
+  endpoint { url = "http://10.20.0.10:3100/loki/api/v1/push" }
+}

vm/postgres-common/queries.yaml

@@ -0,0 +1,9 @@
+# Custom postgres_exporter queries (Alloy prometheus.exporter.postgres custom_queries_config_path).
+# Metric name = <key>_<metric>: pg_is_wal_replay_paused (used by the DbReplicationLag rule, design 4.3).
+pg_is_wal_replay:
+  query: "SELECT CASE WHEN pg_is_in_recovery() AND pg_is_wal_replay_paused() THEN 1 ELSE 0 END AS paused"
+  master: true
+  metrics:
+    - paused:
+        usage: "GAUGE"
+        description: "1 when this server is a replica and WAL replay is paused"