feat: AI VM stack (WP-12) and Alloy telemetry on k8s, legacy and DB VMs (WP-15)
WP-12(실험 설계서 6장) vm/ai-env: apply.sh(시크릿 4종을 Secret Manager에서 파일로, 미등록이면
성능 저하 모드로 기동), compose(agent 8080·hitl 8081·rag-db pgvector 0.8.7-pg16), ai.env(Discord
서버 Selfheal Lab 1557424985793101824, approvals 1557425140302872698, escalation 1557425282363691048,
OPENAI_MODEL gpt-6.1-sol 후보), 프롬프트 3종, 러너북 11종(조치별·토폴로지·3층 절차).
WP-15(실험 설계서 4.1·4.3): apps/alloy.yaml(Alloy Helm 1.13.0 DaemonSet, 클러스터링으로 KSM·Argo CD·
cadvisor 스크랩 분산, demo 등 파드 로그 → Loki), apps/kube-state-metrics.yaml(Helm 8.6.0).
legacy-api compose에 Alloy 사이드카(up{job="legacy-api"}, 컨테이너 로그). DB VM은
vm/postgres-common/alloy.sh로 apt 설치 + prometheus.exporter.postgres(pg_replication_lag_seconds)
+ custom query pg_is_wal_replay_paused + PostgreSQL 로그.
vm/harness/apply.sh: 준비 대기 루프의 curl에 || true(러너 재기동 순간 curl 56으로 중단되던 문제,
2026-10-08 harness-sync 실측).
This commit is contained in:
seon committed
1 parent
59995f3601
commit
7ce2dc68fe
31 files changed
+528
-4
No files matched your search
Executable
+56
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env bash
|
||||
# AI VM (10.20.0.12), experiment design 6. Idempotent: first boot (startup script) and `make vm-sync` / `make ai-sync`.
|
||||
# Secrets come from Secret Manager through the VM service account (Terraform design 6.11):
|
||||
# selfheal-openai-api-key, selfheal-discord-bot-token (registered by the operator), selfheal-k8s-ai-token, selfheal-k8s-ca-cert (make token)
|
||||
# The two operator secrets may not exist yet: the services then start degraded (agent /diagnose 503, hitl without
|
||||
# gateway) and this script still succeeds, so the VM can be verified before the keys arrive.
|
||||
# Non-secret settings (Discord IDs, model) are in ai.env (committed); AWS role ARN comes from instance metadata.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
MD=http://metadata.google.internal/computeMetadata/v1
|
||||
md() { curl -sf -H 'Metadata-Flavor: Google' "$MD/$1"; }
|
||||
PROJECT="$(md project/project-id)"
|
||||
AWS_ROLE_ARN="$(md instance/attributes/aws-role-arn || true)"
|
||||
command -v jq >/dev/null || { apt-get update -qq && apt-get install -y -qq jq >/dev/null; }
|
||||
TOKEN="$(md instance/service-accounts/default/token | jq -r .access_token)"
|
||||
# prints the secret value, or nothing when the secret has no version yet (HTTP 404 / FAILED_PRECONDITION)
|
||||
secret() { curl -s -H "Authorization: Bearer ${TOKEN}" "https://secretmanager.googleapis.com/v1/projects/${PROJECT}/secrets/$1/versions/latest:access" | jq -r '.payload.data // empty' | base64 -d; }
|
||||
|
||||
SEC=/opt/selfheal/ai/secrets
|
||||
mkdir -p "$SEC"; chmod 700 /opt/selfheal/ai "$SEC"
|
||||
umask 077
|
||||
secret selfheal-openai-api-key > "$SEC/openai-api-key"
|
||||
secret selfheal-discord-bot-token > "$SEC/discord-bot-token"
|
||||
secret selfheal-k8s-ai-token > "$SEC/k8s-token"
|
||||
secret selfheal-k8s-ca-cert > "$SEC/k8s-ca.crt"
|
||||
unset TOKEN
|
||||
for f in openai-api-key discord-bot-token; do [ -s "$SEC/$f" ] || echo "note: secret $f is empty (not registered yet); service runs degraded" >&2; done
|
||||
for f in k8s-token k8s-ca.crt; do [ -s "$SEC/$f" ] || echo "note: $f is empty (run make token on the operator host)" >&2; done
|
||||
chown -R 10002:10002 "$SEC"; chmod 600 "$SEC"/* # agent/hitl run as uid 10002
|
||||
|
||||
ENV_FILE=/opt/selfheal/ai.env
|
||||
if [ ! -f "$ENV_FILE" ] || ! grep -q '^RAG_DB_PASSWORD=' "$ENV_FILE"; then
|
||||
RDB_PW="$(head -c 24 /dev/urandom | base64 | tr -d '/+=')"
|
||||
else
|
||||
RDB_PW="$(sed -n 's/^RAG_DB_PASSWORD=//p' "$ENV_FILE")"
|
||||
fi
|
||||
AI_TAG="$(cat image.tag)"
|
||||
{
|
||||
cat ai.env
|
||||
printf 'AWS_ROLE_ARN=%s\n' "$AWS_ROLE_ARN"
|
||||
printf 'AWS_REGION=ap-northeast-1\n'
|
||||
printf 'GCP_PROJECT=%s\n' "$PROJECT"
|
||||
printf 'AI_TAG=%s\n' "$AI_TAG"
|
||||
printf 'RAG_DB_PASSWORD=%s\n' "$RDB_PW"
|
||||
} > "$ENV_FILE"
|
||||
unset RDB_PW
|
||||
|
||||
docker compose --env-file "$ENV_FILE" -f compose.yaml up -d --pull always --remove-orphans
|
||||
# both services read their secret files at start (hitl needs the token to open the gateway): recreate so new secrets apply
|
||||
docker compose --env-file "$ENV_FILE" -f compose.yaml up -d --force-recreate --no-deps agent hitl
|
||||
for i in $(seq 1 30); do
|
||||
a=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/healthz || true); h=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8081/healthz || true)
|
||||
if [ "$a$h" = "200200" ]; then break; fi; sleep 5
|
||||
done
|
||||
echo "ready: agent=$a hitl=$h"
|
||||
docker compose --env-file "$ENV_FILE" -f compose.yaml ps --format 'table {{.Name}}\t{{.Status}}'
|
||||
Reference in new issue
Block a user