feat: harness VM stack (runner, measure-db, blackbox, Alloy, k6) and the 12 scenarios
Experiment design 5 and 7 for the harness VM (10.20.0.11): vm/harness/apply.sh pulls the secrets from Secret Manager, compose runs runner:de076a7, measure-db, blackbox, Alloy (probe_success -> Mimir) and k6 under the campaign profile. Scenario files C-K1..X-M2 are loaded by the runner. Harness ClusterRole gains replicasets get/list for rollout undo.
This commit is contained in:
seon committed
1 parent
de52d4a1ec
commit
af37d19e2a
21 files changed
+505
-1
No files matched your search
@@ -0,0 +1,32 @@
|
||||
# design 7 C-G1: grey zone, container. NetworkChaos delay api -> db-primary; pods stay Ready, p95 breaks. Correct answer: escalate.
|
||||
id: C-G1
|
||||
path: gray
|
||||
asset: container
|
||||
expected_layer: 2
|
||||
ground_truth_label: network_latency_api_to_db
|
||||
ground_truth_action: escalate
|
||||
description: NetworkChaos delay 300ms jitter 50ms, direction to, externalTargets 10.20.0.20, app=api
|
||||
notes:
|
||||
open_item: "design 11.2: whether the delay applies through the flannel SNAT path is measured in the campaign"
|
||||
inject:
|
||||
method: chaos
|
||||
manifest: |
|
||||
apiVersion: chaos-mesh.org/v1alpha1
|
||||
kind: NetworkChaos
|
||||
metadata:
|
||||
namespace: chaos-mesh
|
||||
spec:
|
||||
action: delay
|
||||
mode: all
|
||||
direction: to
|
||||
externalTargets: ["10.20.0.20"]
|
||||
selector:
|
||||
namespaces: [demo]
|
||||
labelSelectors:
|
||||
app: api
|
||||
delay:
|
||||
latency: "300ms"
|
||||
jitter: "50ms"
|
||||
duration: "20m"
|
||||
revert:
|
||||
method: chaos_delete
|
||||
@@ -0,0 +1,30 @@
|
||||
# design 7 C-G2: grey zone, container. StressChaos memory on api pods; latency rises, intermittent 5xx. Correct: k8s.rollout_restart after approval.
|
||||
id: C-G2
|
||||
path: gray
|
||||
asset: container
|
||||
expected_layer: 2
|
||||
ground_truth_label: memory_pressure_api
|
||||
ground_truth_action: k8s.rollout_restart
|
||||
description: StressChaos memory workers 2 size 80%, app=api, duration 10m
|
||||
notes:
|
||||
open_item: "design 11.3: whether the stress re-attaches to the restarted pods for the remaining duration"
|
||||
inject:
|
||||
method: chaos
|
||||
manifest: |
|
||||
apiVersion: chaos-mesh.org/v1alpha1
|
||||
kind: StressChaos
|
||||
metadata:
|
||||
namespace: chaos-mesh
|
||||
spec:
|
||||
mode: all
|
||||
selector:
|
||||
namespaces: [demo]
|
||||
labelSelectors:
|
||||
app: api
|
||||
stressors:
|
||||
memory:
|
||||
workers: 2
|
||||
size: "80%"
|
||||
duration: "10m"
|
||||
revert:
|
||||
method: chaos_delete
|
||||
@@ -0,0 +1,15 @@
|
||||
# design 7 C-I1: irreversible, container. Schema change on db-primary (DROP COLUMN status). Layer 2 diagnoses -> escalate -> a human restores (db.data_fix).
|
||||
# Runs as the postgres OS user over SSH (the harness DB role has SELECT only). Revert: the human re-adds the column; recorded via /actions/execute db.data_fix.
|
||||
id: C-I1
|
||||
path: irreversible
|
||||
asset: container
|
||||
expected_layer: 3
|
||||
ground_truth_label: schema_mismatch
|
||||
ground_truth_action: escalate
|
||||
description: ALTER TABLE app.orders DROP COLUMN status on db-primary
|
||||
inject:
|
||||
method: ssh
|
||||
host: db-primary
|
||||
command: sudo -u postgres psql -d app -v ON_ERROR_STOP=1 -c "ALTER TABLE app.orders DROP COLUMN status"
|
||||
revert:
|
||||
method: none
|
||||
@@ -0,0 +1,24 @@
|
||||
# design 7 C-K1: known signature, container. PodChaos pod-kill one api pod; ReplicaSet recreates it, PDB keeps the SLO.
|
||||
id: C-K1
|
||||
path: known
|
||||
asset: container
|
||||
expected_layer: 1
|
||||
ground_truth_label: api_pod_killed
|
||||
ground_truth_action: none
|
||||
description: PodChaos pod-kill, mode one, selector app=api
|
||||
inject:
|
||||
method: chaos
|
||||
manifest: |
|
||||
apiVersion: chaos-mesh.org/v1alpha1
|
||||
kind: PodChaos
|
||||
metadata:
|
||||
namespace: chaos-mesh
|
||||
spec:
|
||||
action: pod-kill
|
||||
mode: one
|
||||
selector:
|
||||
namespaces: [demo]
|
||||
labelSelectors:
|
||||
app: api
|
||||
revert:
|
||||
method: chaos_delete
|
||||
@@ -0,0 +1,26 @@
|
||||
# design 7 C-K2: known signature, container. PodChaos pod-failure 60s on one api pod -> liveness fails -> kubelet restarts.
|
||||
id: C-K2
|
||||
path: known
|
||||
asset: container
|
||||
expected_layer: 1
|
||||
ground_truth_label: api_pod_failure
|
||||
ground_truth_action: none
|
||||
description: PodChaos pod-failure 60s, app=api
|
||||
escalate_after_s: 300
|
||||
inject:
|
||||
method: chaos
|
||||
manifest: |
|
||||
apiVersion: chaos-mesh.org/v1alpha1
|
||||
kind: PodChaos
|
||||
metadata:
|
||||
namespace: chaos-mesh
|
||||
spec:
|
||||
action: pod-failure
|
||||
mode: one
|
||||
duration: "60s"
|
||||
selector:
|
||||
namespaces: [demo]
|
||||
labelSelectors:
|
||||
app: api
|
||||
revert:
|
||||
method: chaos_delete
|
||||
@@ -0,0 +1,15 @@
|
||||
# design 7 C-K3: known signature, container. Out-of-Git change (scale api to 1); Argo CD selfHeal restores replicas 3.
|
||||
id: C-K3
|
||||
path: known
|
||||
asset: container
|
||||
expected_layer: 1
|
||||
ground_truth_label: gitops_drift
|
||||
ground_truth_action: none
|
||||
description: harness scales deployment/api to 1 outside Git
|
||||
inject:
|
||||
method: kubectl_scale
|
||||
namespace: demo
|
||||
deployment: api
|
||||
replicas: 1
|
||||
revert:
|
||||
method: none
|
||||
@@ -0,0 +1,16 @@
|
||||
# design 7 V-G1: grey zone, VM. tc netem delay 400ms on one legacy instance; health check passes, /api/price p95 breaks. Correct: mig.recreate_instance after approval.
|
||||
id: V-G1
|
||||
path: gray
|
||||
asset: vm
|
||||
expected_layer: 2
|
||||
ground_truth_label: network_latency_legacy
|
||||
ground_truth_action: mig.recreate_instance
|
||||
description: tc qdisc add dev ens4 root netem delay 400ms on legacy instance 0
|
||||
inject:
|
||||
method: ssh
|
||||
host: legacy:0
|
||||
command: sudo tc qdisc add dev ens4 root netem delay 400ms
|
||||
revert:
|
||||
method: ssh
|
||||
host: legacy:0
|
||||
command: sudo tc qdisc del dev ens4 root || true
|
||||
@@ -0,0 +1,18 @@
|
||||
# design 7 V-G2: grey zone, VM. Pause WAL replay on db-replica; no SLO impact, only DbReplicationLag fires. Correct: db.replay_resume after approval.
|
||||
id: V-G2
|
||||
path: gray
|
||||
asset: vm
|
||||
expected_layer: 2
|
||||
ground_truth_label: replication_paused
|
||||
ground_truth_action: db.replay_resume
|
||||
description: SELECT pg_wal_replay_pause() on db-replica (harness role)
|
||||
inject:
|
||||
method: psql
|
||||
host: db-replica
|
||||
database: postgres
|
||||
sql: SELECT pg_wal_replay_pause()
|
||||
revert:
|
||||
method: psql
|
||||
host: db-replica
|
||||
database: postgres
|
||||
sql: SELECT pg_wal_replay_resume()
|
||||
@@ -0,0 +1,15 @@
|
||||
# design 7 V-I1: irreversible, VM. Stop db-primary. Layer 2 diagnoses -> escalate -> a human promotes the replica and switches the DSN (db.promote).
|
||||
# The old primary is not started again (STONITH); revert is none by design. Recovery of the lab afterwards: Terraform / restore.
|
||||
id: V-I1
|
||||
path: irreversible
|
||||
asset: vm
|
||||
expected_layer: 3
|
||||
ground_truth_label: db_primary_down
|
||||
ground_truth_action: escalate
|
||||
description: gcloud compute instances stop selfheal-db-primary
|
||||
inject:
|
||||
method: gcloud
|
||||
op: instance_stop
|
||||
instance: db-primary
|
||||
revert:
|
||||
method: none
|
||||
@@ -0,0 +1,15 @@
|
||||
# design 7 V-K1: known signature, VM. docker kill legacy-api on one MIG instance; compose restart:always brings it back before the health check fails.
|
||||
id: V-K1
|
||||
path: known
|
||||
asset: vm
|
||||
expected_layer: 1
|
||||
ground_truth_label: legacy_process_killed
|
||||
ground_truth_action: none
|
||||
description: SSH to legacy instance 0, docker kill legacy-api
|
||||
escalate_after_s: 600
|
||||
inject:
|
||||
method: ssh
|
||||
host: legacy:0
|
||||
command: sudo docker kill legacy-api
|
||||
revert:
|
||||
method: none
|
||||
@@ -0,0 +1,15 @@
|
||||
# design 7 V-K2: known signature, VM. Delete one legacy instance; MIG autohealing recreates it (initial_delay 300s), ILB serves from the other.
|
||||
id: V-K2
|
||||
path: known
|
||||
asset: vm
|
||||
expected_layer: 1
|
||||
ground_truth_label: legacy_instance_lost
|
||||
ground_truth_action: none
|
||||
description: gcloud compute instances delete legacy instance 0
|
||||
escalate_after_s: 600
|
||||
inject:
|
||||
method: gcloud
|
||||
op: instance_delete
|
||||
instance: legacy:0
|
||||
revert:
|
||||
method: none
|
||||
@@ -0,0 +1,28 @@
|
||||
# design 7 X-M1: misjudge probe, container. 25s partition api <-> db, just under the liveness failure window (3 x 10s);
|
||||
# observe whether kubelet restarts containers needlessly.
|
||||
id: X-M1
|
||||
path: misjudge
|
||||
asset: container
|
||||
expected_layer: 1
|
||||
ground_truth_label: transient_partition
|
||||
ground_truth_action: none
|
||||
description: NetworkChaos partition 25s, api -> 10.20.0.20
|
||||
inject:
|
||||
method: chaos
|
||||
manifest: |
|
||||
apiVersion: chaos-mesh.org/v1alpha1
|
||||
kind: NetworkChaos
|
||||
metadata:
|
||||
namespace: chaos-mesh
|
||||
spec:
|
||||
action: partition
|
||||
mode: all
|
||||
direction: to
|
||||
externalTargets: ["10.20.0.20"]
|
||||
selector:
|
||||
namespaces: [demo]
|
||||
labelSelectors:
|
||||
app: api
|
||||
duration: "25s"
|
||||
revert:
|
||||
method: chaos_delete
|
||||
@@ -0,0 +1,17 @@
|
||||
# design 7 X-M2: misjudge probe, VM. 50% packet loss for 90s on one legacy instance; observe whether MIG autohealing recreates it for a transient fault.
|
||||
id: X-M2
|
||||
path: misjudge
|
||||
asset: vm
|
||||
expected_layer: 1
|
||||
ground_truth_label: transient_loss_legacy
|
||||
ground_truth_action: none
|
||||
description: tc netem loss 50% for 90s on legacy instance 0
|
||||
inject:
|
||||
method: ssh
|
||||
host: legacy:0
|
||||
command: sudo tc qdisc add dev ens4 root netem loss 50%
|
||||
wait_s: 90
|
||||
revert:
|
||||
method: ssh
|
||||
host: legacy:0
|
||||
command: sudo tc qdisc del dev ens4 root || true
|
||||
Reference in new issue
Block a user