Fault Tolerance

When trying to achieve the goal of resilience in our distributed systems, sometimes we need features like retries, fallbacks, and circuit breakers. Luckily for us, Quarkus provides these features for us through the fault tolerance extension.

Add the Fault Tolerance extension

Just open a new terminal window, and make sure you’re at the root of your tutorial-app project, then run:

  • Maven

  • Quarkus CLI

mvn quarkus:add-extension -Dextensions=quarkus-smallrye-fault-tolerance
quarkus extension add quarkus-smallrye-fault-tolerance

Add Retry to FruityViceService

Let’s add the retry policy in FruityViceService.

Change the FruityViceService Java interface in src/main/java in the com.redhat.developers package with the following contents:

package com.redhat.developers;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

import org.eclipse.microprofile.faulttolerance.Retry;
import org.eclipse.microprofile.rest.client.inject.RegisterRestClient;

@Path("/api/fruit")
@RegisterRestClient
public interface FruityViceService {

    @GET
    @Path("/{name}")
    @Produces(MediaType.APPLICATION_JSON)
    @Retry(maxRetries = 3, delay = 2000)
    FruityVice getFruitByName(@PathParam("name") String name);

}

Now in case of any error, 3 retries are done automatically, waiting for 2 seconds between retries.

Invoke the endpoint with Retry

Run the following command:

curl localhost:8080/fruit?season=Summer
[
  {
    "calories": 0,
    "carbohydrates": 29,
    "name": "Blueberry",
    "season": "Summer"
  },
  {
    "calories": 0,
    "carbohydrates": 96,
    "name": "Banana",
    "season": "Summer"
  },
  {
    "calories": 0,
    "carbohydrates": 30,
    "name": "Watermelon",
    "season": "Summer"
  }
]

No change from calls done previously, but now switch off your network so you do not have access to http://fruityvice.com.

Run the following command again:

curl localhost:8080/fruit?season=Summer

Now after waiting 6 seconds (3 retries x 2 seconds), the next exception is thrown java.net.UnknownHostException: fruityvice.com.

Add Fallback to Fruity Vice Service

Let’s add a fallback policy in case of an error in FruityViceService.

Change the FruityViceService Java interface in src/main/java in the com.redhat.developers package with the following contents:

package com.redhat.developers;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

import com.redhat.developers.FruityVice.Nutritions;

import org.eclipse.microprofile.faulttolerance.ExecutionContext;
import org.eclipse.microprofile.faulttolerance.Fallback;
import org.eclipse.microprofile.faulttolerance.FallbackHandler;
import org.eclipse.microprofile.faulttolerance.Retry;
import org.eclipse.microprofile.rest.client.inject.RegisterRestClient;

@Path("/api/fruit")
@RegisterRestClient
public interface FruityViceService {

    @GET
    @Path("/{name}")
    @Produces(MediaType.APPLICATION_JSON)
    @Retry(maxRetries = 3, delay = 2000)
    @Fallback(FruityViceFallback.class)
    FruityVice getFruitByName(@PathParam("name") String name);

    public static class FruityViceFallback implements FallbackHandler<FruityVice> {

        private static final FruityVice EMPTY_FRUITY_VICE = FruityVice.of("empty", Nutritions.of(0.0, 0.0));
        @Override
        public FruityVice handle(ExecutionContext context) {
            return EMPTY_FRUITY_VICE;
        }

    }

}

Now in case of any error, 3 retries are done automatically, waiting for 2 seconds between retries.

If the error persists, then the fallback method is executed.

Now after waiting for 6 seconds (3 retries x 2 seconds), an empty object is sent instead of an exception.

Invoke the endpoint with Retry and Fallback

Run the following command:

curl localhost:8080/fruit?season=Summer
[
  {
    "calories": 0,
    "carbohydrates": 0,
    "name": "Blueberry",
    "season": "Summer"
  },
  {
    "calories": 0,
    "carbohydrates": 0,
    "name": "Banana",
    "season": "Summer"
  },
  {
    "calories": 0,
    "carbohydrates": 0,
    "name": "Watermelon",
    "season": "Summer"
  }
]

Add Circuit Breaker to Fruity Vice Service

Let’s add the circuit breaker policy in FruityViceService.

Change the FruityViceService Java interface in src/main/java in the com.redhat.developers package with the following contents:

package com.redhat.developers;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

import org.eclipse.microprofile.faulttolerance.CircuitBreaker;
import org.eclipse.microprofile.faulttolerance.Retry;
import org.eclipse.microprofile.rest.client.inject.RegisterRestClient;

@Path("/api/fruit")
@RegisterRestClient
public interface FruityViceService {

    @GET
    @Path("/{name}")
    @Produces(MediaType.APPLICATION_JSON)
    @Retry(maxRetries = 3, delay = 2000)
    @CircuitBreaker(requestVolumeThreshold = 4, failureRatio = 0.75, delay = 5000)
    FruityVice getFruitByName(@PathParam("name") String name);

}

Now, if 3 (4 x 0.75) failures occur among the rolling window of 4 consecutive invocations, then the circuit is opened for 5000 ms and then will be back to half open. If the invocation succeeds, then the circuit is back to closed again.

Run the following command at least 5 times:

curl localhost:8080/fruit?season=Summer

The output changes from java.net.UnknownHostException: fruityvice.com (or any other network exception) in the first calls to org.eclipse.microprofile.faulttolerance.exceptions.CircuitBreakerOpenException: getFruitByName when the circuit is opened.

The big difference between the first exception and the second one is that the first one occurs because the circuit is closed while the system is trying to reach the host, while in the second one, the circuit is closed and the exception is thrown automatically without trying to reach the host.

You can use @Retry and @Fallback annotations together with @CircuitBreaker annotation.
If you turned your network off for this chapter, remember to turn it back on again after you finished the exercises for this chapter.