monad bft latency testing

kubernetes test environment for monad bft latency testing with chaos mesh and observability.

quick start

./setup.sh    # install infrastructure
./build.sh    # build latency image
./deploy.sh   # deploy latency cluster

cluster configuration

• 10 node cluster with static IPs (10.1.0.1 - 10.1.0.10)
• latency-0: producer (sends 2MB messages every 10s)
• latency-1 to latency-9: consumers
• metrics exported to prometheus via otel collector
• logs aggregated via loki

access services

grafana

minikube service grafana -n monitoring

credentials: admin/admin

chaos mesh dashboard

kubectl port-forward -n chaos-mesh svc/chaos-dashboard 2333:2333

access at http://localhost:2333

chaos experiments

constant network latency

enable constant latency between pod buckets (5 buckets with 20-80ms incremental latency):

./chaos-experiments/manage.sh enable network-latency-constant

this creates a network topology with:

• bucket-1 (latency-0,1) to bucket-2 (latency-2,3): 20ms
• bucket-1 to bucket-3 (latency-4,5): 40ms
• bucket-1 to bucket-4 (latency-6,7): 60ms
• bucket-1 to bucket-5 (latency-8,9): 80ms
• other bucket pairs: 20-40ms incremental latency

periodic chaos experiments

run scheduled chaos experiments to test resilience:

cd chaos-experiments

./manage.sh enable network-packet-loss
./manage.sh enable network-partition
./manage.sh enable pod-kill
./manage.sh enable all

./manage.sh status
./manage.sh disable network-packet-loss
./manage.sh disable network-latency-constant

see chaos-experiments/README.md for details.

useful commands

kubectl get pods -l app=latency -o wide
kubectl logs -f latency-0
kubectl logs -f latency-1

rebuild and redeploy

cd ~/monad-bft && cargo build --release --example latency
cd ~/monad-testing
./build.sh
kubectl delete pod -l app=latency
./deploy.sh