seon
840b14c20c
2026-10-18 m2 두 번째 런에서 X-M2 복원 SSH 가 connection reset 으로 실패해 legacy-p787 에 netem loss 50% qdisc 가 남았고, 이후 14런의 V-G1·X-M2 주입이 "Exclusivity flag on" 으로 전부 실패했다. replace 는 root qdisc 가 남아 있어도 덮어쓰므로 주입이 막히지 않는다 (복원은 그대로 del). 남은 qdisc 는 수동으로 제거했다(16:06 UTC).
20 lines
984 B
YAML
20 lines
984 B
YAML
# design 7 V-G1: grey zone, VM. tc netem delay 400ms on one legacy instance; health check passes, /api/price p95 breaks. Correct: mig.recreate_instance after approval.
|
|
id: V-G1
|
|
path: gray
|
|
asset: vm
|
|
expected_layer: 2
|
|
ground_truth_label: network_latency_legacy
|
|
ground_truth_action: mig.recreate_instance
|
|
description: tc qdisc add dev ens4 root netem delay 400ms on the legacy instance that currently serves the api traffic (legacy:busy)
|
|
# 2026-10-08: the ILB hashes the api pods' few keep-alive connections onto one instance; in campaign 2 legacy:0 served 0 rps,
|
|
# so the delay was never on the request path. legacy:busy = the instance with the most legacy-api requests over the last
|
|
# 2 minutes (Mimir); the harness caches the choice per scenario so the revert hits the same instance.
|
|
inject:
|
|
method: ssh
|
|
host: legacy:busy
|
|
command: sudo tc qdisc replace dev ens4 root netem delay 400ms
|
|
revert:
|
|
method: ssh
|
|
host: legacy:busy
|
|
command: sudo tc qdisc del dev ens4 root || true
|