seon
3f686c1aea
第 3 回キャンペーンの実測を反映する。V-G1 では ILB が 1 分以内に api の接続を 遅いインスタンスから逃がすため 5 分窓の p95 が 0.5 秒を越えず、ApiLatencyHigh が 発火しなかった（1 分窓の p95 は 3 分半 2.3 秒）。式を 1 分窓にし 2 分継続に変える。 C-G2 は 200MB で working set が limit の 0.797〜0.803 を行き来して閾値 0.8 を 保てなかったため 225MB にし、ApiMemoryPressure の継続を 1 分にする。
vm/harness
Harness VM (10.20.0.11) of the self-healing lab: experiment design 5 (runner, measure DB, blackbox, k6) plus Alloy.
| file | purpose |
|---|---|
apply.sh |
boot / make vm-sync entry: secrets from Secret Manager to /opt/selfheal/harness/secrets, env file, compose up |
compose.yaml |
runner (8080), measure-db, blackbox (9115), alloy (12345), k6 (profile campaign) |
image.tag |
runner image tag (selfheal-public/harness:<sha> built from selfheal-lab-apps/harness) |
scenarios/*.yaml |
the 12 scenarios of design 7, loaded by the runner at start |
config.alloy |
blackbox + runner scrape -> Mimir remote_write |
blackbox.yaml |
http_2xx (ALB /api/orders, label target=alb) and tcp_connect (target=db-primary) |
k6/load.js |
background load 20 rps (design 5.5) |
Runner API (design 5.2):
| endpoint | use |
|---|---|
GET /selfcheck |
measure DB, k8s (chaos-mesh pods), AWS OIDC caller identity, GCP instances, Mimir probe, ALB |
POST /runs {"campaign","scenario_ids":[...],"baseline_s","cooldown_s","observe_timeout_s","min_observe_s","drift_before"} |
start a run (one at a time) |
GET /runs, GET /runs/{id}, POST /runs/{id}/abort, POST /runs/{id}/finish {"drift_after"} |
inspect / stop / close |
POST /hooks/alert |
Grafana contact point |
POST /proposals |
AI VM registers a layer-2 proposal; returns the approval token |
POST /actions/execute {"action_id","target","token"} |
catalogue action with a human token or an approved proposal token |
GET /actions/catalog, GET /scenarios, GET /metrics |
reference |
Campaign load: docker compose --env-file /opt/selfheal/harness.env --profile campaign up -d k6 (stop: ... stop k6; exit code and /logs/summary.json in the k6-logs volume).