Chaos Testing

Apply some chaos engineering by throwing in some HTTP errors or network delays. Understanding failure scenarios is a critical aspect of microservices architecture (aka distributed computing)

Before Start

You should have NO virtualservice nor destinationrule (in tutorial namespace) kubectl get virtualservice-$WORKSHOP_USER kubectl get destinationrule-$WORKSHOP_USER if so run:

./scripts/clean.sh tutorial-$WORKSHOP_USER

HTTP Error 503

By default, recommendation v1 and v2 are being randomly load-balanced as that is the default behavior in Kubernetes/OpenShift

$ oc get pods -l app=recommendation -n tutorial-$WORKSHOP_USER
or
$ kubectl get pods -l app=recommendation -n tutorial-$WORKSHOP_USER

NAME                                  READY     STATUS    RESTARTS   AGE
recommendation-v1-3719512284-7mlzw   2/2       Running   6          18h
recommendation-v2-2815683430-vn77w   2/2       Running   0          3h

You can inject 503’s, for approximately 50% of the requests

kubectl create -f istiofiles/destination-rule-recommendation.yml -n tutorial-$WORKSHOP_USER
kubectl create -f istiofiles/virtual-service-recommendation-503.yml -n tutorial-$WORKSHOP_USER

./scripts/run.sh http://istio-ingressgateway-istio-system.apps.jconf-d4f8.openshiftworkshop.com/customer$WORKSHOP_USER

customer => preference => recommendation v2 from '7778d6fb89-j6lts': 138
customer => Error: 503 - preference => Error: 503 - fault filter abort
customer => preference => recommendation v1 from '6976858b48-dswhz': 139
customer => Error: 503 - preference => Error: 503 - fault filter abort
customer => preference => recommendation v2 from '7778d6fb89-j6lts': 139
customer => preference => recommendation v1 from '6976858b48-dswhz': 140
customer => preference => recommendation v2 from '7778d6fb89-j6lts': 140
customer => preference => recommendation v1 from '6976858b48-dswhz': 141
customer => Error: 503 - preference => Error: 503 - fault filter abort
customer => Error: 503 - preference => Error: 503 - fault filter abort
customer => preference => recommendation v2 from '7778d6fb89-j6lts': 141

Clean up

kubectl delete -f istiofiles/virtual-service-recommendation-503.yml -n tutorial-$WORKSHOP_USER

Delay

The most insidious of possible distributed computing faults is not a "down" service but a service that is responding slowly, potentially causing a cascading failure in your network of services.

kubectl create -f istiofiles/virtual-service-recommendation-delay.yml -n tutorial-$WORKSHOP_USER
kubectl replace -f istiofiles/destination-rule-recommendation.yml -n tutorial-$WORKSHOP_USER

And hit the customer endpoint

You will notice many requests to the customer endpoint now have a delay. If you are monitoring the logs for recommendation v1 and v2, you will also see the delay happens BEFORE the recommendation service is actually called

stern recommendation -n tutorial-$WORKSHOP_USER

or `bash ./kubetail.sh recommendation -n tutorial-$WORKSHOP_USER `

Clean up

kubectl delete -f istiofiles/destination-rule-recommendation.yml -n tutorial-$WORKSHOP_USER
kubectl delete -f istiofiles/virtual-service-recommendation-delay.yml -n tutorial-$WORKSHOP_USER