The second a part of this sequence was all about idea – we had a better take a look at some useful resilience design patterns. Now it is time to take some motion 💪. Let’s lastly deliver all threads collectively and see how we are able to understand this with Quarkus and Kotlin.
Do you continue to bear in mind the surprising state of affairs for all of this? We’re doomed trigger we dwell in an space with toxic snakes and should depend on our personal service to be warned concerning the harmful animals in time.
Let me introduce you to the Snake Alarm Detection Fulfilment State of affairs the place we’re going to deal with this in a resilient approach.
Think about small snake detector bins, which may magically detect snakes passing by. Generally the detectors have false positives with cute and innocent lizards. To deal with these false positives the snake detector bins can even ship a photograph of the detected animal by way of REST endpoint so it may be evaluated, for instance, by the use of machine studying. After we are certain that it is a snake, we wish to ship a push notifications to all cell units in that space. Sadly, now we have to make use of an exterior service for the push notifications and that is fairly costly. We now have two choices right here: one service, which could be very costly and 100% dependable however a bit of bit sluggish and one other providers, which is less expensive and sooner, however not so dependable in any respect.
The plan is to construct a strong roundtrip from a) detecting a “snake or lizard” with the field to b) sending a warning push notification to everybody within the space. Ideally the push notification goes out quick and we don’t spend that a lot cash on sending these push notifications.
Let’s strive: the snake detector field might make a POST request to a SnakeAlarmEndpoint. This might set off a SnakeAlarmRepository to emit an occasion into an AMQP messaging channel named snake-alarm-out. Then we subscribe to these occasions in one other snake-alarm-in channel to have the ability to scale independently. Given the illustration under, you may see our SnakeAlarmProcessor, which may course of new SnakeAlarms coming in from the AMQP channel snake-alarms-in.
The SnakeAlarmProcessor itself calls the LizardCheck to confirm that now we have a snake and never a lizard coming in.
The LizardCheck has a runtime between 0,2s and 1,06s. Having the runtime measured, we are able to configure our SnakeAlarmProcessor name to the LizardCheck with a timeout of 1s and a most of three retries to make most calls achieve a suitable time.
Lastly we name the push notification service. The first service of alternative shall be tried first, however fails with a likelihood of fifty% and has an common response time of 1s. We give {that a} most of two retries. After that we name the secondary service of alternative as a fallback, which has a likelihood of ~100% and a response time of 3s.
Pattern output of the operating end result
Right here you discover the output of the ultimate end result. Attempt to see the place we had a false constructive and the place not. The place did we use the costly push notification service? It is all written within the logs.
2022-07-24 16:03:53,936 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Obtained an alarm for snake alarm success: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False constructive - Efficiently verified a lizard
2022-07-24 16:03:53,994 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to handle `snake-alarms`
2022-07-24 16:03:54,000 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{function=RECEIVER, first=1, final=1, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:54,001 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Switch{deal with=0, deliveryId=1, deliveryTag=x00, messageFormat=0, settled=false, extra=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[x00Ssxd0x00x00x00)x00x00x00x07@@xa1x0csnake-alarms@@@xa3x10application/jsonx00Suxa0x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake is also a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:54,460 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to handle `snake-alarms`
2022-07-24 16:03:54,463 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Switch{deal with=0, deliveryId=2, deliveryTag=x01, messageFormat=0, settled=false, extra=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[x00Ssxd0x00x00x00)x00x00x00x07@@xa1x0csnake-alarms@@@xa3x10application/jsonx00Suxa0x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake is also a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:54,464 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{function=RECEIVER, first=2, final=2, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:54,579 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Obtained an alarm for snake alarm success: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False constructive - Efficiently verified a lizard
2022-07-24 16:03:54,937 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Obtained an alarm for snake alarm success: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False constructive - Efficiently verified a lizard
2022-07-24 16:03:56,180 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to handle `snake-alarms`
2022-07-24 16:03:56,185 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{function=RECEIVER, first=3, final=3, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:56,186 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Switch{deal with=0, deliveryId=3, deliveryTag=x00, messageFormat=0, settled=false, extra=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[x00Ssxd0x00x00x00)x00x00x00x07@@xa1x0csnake-alarms@@@xa3x10application/jsonx00Suxa0x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake is also a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:56,618 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to handle `snake-alarms`
2022-07-24 16:03:56,622 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Switch{deal with=0, deliveryId=4, deliveryTag=x01, messageFormat=0, settled=false, extra=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[x00Ssxd0x00x00x00)x00x00x00x07@@xa1x0csnake-alarms@@@xa3x10application/jsonx00Suxa0x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake is also a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:56,625 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{function=RECEIVER, first=4, final=4, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:57,105 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to handle `snake-alarms`
2022-07-24 16:03:57,110 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{function=RECEIVER, first=5, final=5, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:57,111 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Switch{deal with=0, deliveryId=5, deliveryTag=x02, messageFormat=0, settled=false, extra=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[x00Ssxd0x00x00x00)x00x00x00x07@@xa1x0csnake-alarms@@@xa3x10application/jsonx00Suxa0x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake is also a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:57,219 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Obtained an alarm for snake alarm success: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False constructive - Efficiently verified a lizard
2022-07-24 16:03:57,534 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to handle `snake-alarms`
2022-07-24 16:03:57,538 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{function=RECEIVER, first=6, final=6, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:57,539 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Switch{deal with=0, deliveryId=6, deliveryTag=x00, messageFormat=0, settled=false, extra=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[x00Ssxd0x00x00x00)x00x00x00x07@@xa1x0csnake-alarms@@@xa3x10application/jsonx00Suxa0x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake is also a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:58,039 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Obtained an alarm for snake alarm success: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False constructive - Efficiently verified a lizard
2022-07-24 16:03:58,041 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Obtained an alarm for snake alarm success: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], Hazard - Efficiently verified a snake
2022-07-24 16:03:58,052 INFO [de.fay.sna.PushNotificationService] (vert.x-eventloop-thread-0) Push notification estimated in 3 sec
2022-07-24 16:03:58,420 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Obtained an alarm for snake alarm success: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False constructive - Efficiently verified a lizard
Postman
That is an exemplary POST request we’re going to use to fireplace up our providers. If we make the request proper, we must always obtain a 202 Accepted response. You will discover this as a Postman assortment within the Github sources.
Snake alarm endpoint
Our endpoint calls the SnakeAlarmRepository to deal with every little thing.
@Path("snake-alarm")
class SnakeAlarmEndpoint(
personal val snakeAlarmRepository: SnakeAlarmRepository
) {
@POST
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.TEXT_PLAIN)
droop enjoyable snakeAlarm(request: SnakeAlarmRequest): Response {
return when (snakeAlarmRepository.alarm(request.snakes)) {
true -> Response.accepted().construct()
false -> Response.serverError().construct()
}
}
}
The SnakeAlarmRepository emits the alarm with out verification utilizing an AMQP messaging channel named snake-alarms-out.
@ApplicationScoped
class SnakeAlarmRepository(
personal val pushNotificationService: PushNotificationService,
@Channel("snake-alarms-out") personal val snakeAlarmEmitter: Emitter<SnakeAlarm>,
personal val logger: Logger
) {
droop enjoyable alarm(objects: Record<SnakeItems>): Boolean {
val snakeAlarm = SnakeAlarm(
snakes = objects,
snakeVerification = "Snake is also a lizard :-)",
estimatedPushNotificationTimestamp = null
)
return strive {
snakeAlarmEmitter.ship(snakeAlarm).await()
true
} catch (e: Exception) {
logger.log(Stage.SEVERE, "Error whereas sending snake alarm", e)
false
}
}
}
Software Properties
That is the place we have to configure our loosely coupled AMPQ messaging channels, which is able to run utilizing a Docker picture (a pattern Docker file is supplied within the Github sources).
quarkus.log.degree=DEBUG
quarkus.log.class."org.jboss.resteasy".min-level=DEBUG
# The AMQP dealer location and credentials
amqp-host=localhost
amqp-port=5672
amqp-username=quarkus
amqp-password=quarkus
# Configure snake alarm messaging
mp.messaging.incoming.snake-alarms-in.connector=smallrye-amqp
mp.messaging.incoming.snake-alarms-in.handle=snake-alarms
mp.messaging.outgoing.snake-alarms-out.connector=smallrye-amqp
mp.messaging.outgoing.snake-alarms-out.handle=snake-alarms
After we obtain a message from this channel, the SnakeAlarmProcessor checks if it truly is a snake through the use of the LizardCheck.
@ApplicationScoped
class SnakeAlarmProcessor(
personal val pushNotificationService: PushNotificationService,
personal val lizardCheck: LizardCheck,
personal val logger: Logger
) {
@Incoming("snake-alarms-in")
droop enjoyable processSnakeAlarm(snakeAlarm: SnakeAlarm) {
val isVerifiedSnake = lizardCheck.checkSnakeAlarmValid(snakeAlarm)
val verificationResultText = if (isVerifiedSnake) "Hazard - Efficiently verified a snake" else "False constructive - Efficiently verified a lizard"
logger.data { "Obtained an alarm for snake alarm success: ${snakeAlarm.snakes}, $verificationResultText" }
if(isVerifiedSnake) pushNotificationService.announcePrimaryPushNotificationService()
}
}
The LizardCheck has usually timeouts. Therefor now we have configured timeout and retry choices through the use of annotations.
@ApplicationScoped
class LizardCheck {
@Timeout(1, unit = ChronoUnit.SECONDS)
@Retry(maxRetries = 3)
droop enjoyable checkSnakeAlarmValid(): Boolean {
if (Random.nextBoolean()) {
delay(Random.nextLong(200, 1060))
return false
}
return true
}
}
The PushNotificationServices has a way to announce every notification providers. The primary technique has the second technique has a fallback. If the fallback is used, now we have a max retry of two makes an attempt for that.
@ApplicationScoped
class PushNotificationService(
personal val logger: Logger
) {
@Fallback(fallbackMethod = "announceSecondaryPushNotificationService")
enjoyable announcePrimaryPushNotificationService(): Date {
val fail = Random.nextBoolean()
if (fail) {
error("Cannot attain main push notification service")
}
logger.data { "Push notification estimated in 1 sec" }
return Calendar.getInstance().apply {
add(Calendar.SECOND, 1)
}.time
}
@Retry(maxRetries = 2)
enjoyable announceSecondaryPushNotificationService(): Date {
logger.data { "Push notification estimated in 3 sec" }
return Calendar.getInstance().apply {
add(Calendar.SECOND, 3)
}.time
}
}
Let’s re-check the logs:
2022-07-24 16:03:53,936 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Obtained an alarm for snake alarm success: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False constructive - Efficiently verified a lizard
2022-07-24 16:03:53,994 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to handle `snake-alarms`
2022-07-24 16:03:54,000 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{function=RECEIVER, first=1, final=1, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:54,001 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Switch{deal with=0, deliveryId=1, deliveryTag=x00, messageFormat=0, settled=false, extra=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[x00Ssxd0x00x00x00)x00x00x00x07@@xa1x0csnake-alarms@@@xa3x10application/jsonx00Suxa0x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake is also a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:54,460 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to handle `snake-alarms`
2022-07-24 16:03:54,463 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Switch{deal with=0, deliveryId=2, deliveryTag=x01, messageFormat=0, settled=false, extra=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[x00Ssxd0x00x00x00)x00x00x00x07@@xa1x0csnake-alarms@@@xa3x10application/jsonx00Suxa0x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake is also a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:54,464 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{function=RECEIVER, first=2, final=2, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:54,579 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Obtained an alarm for snake alarm success: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False constructive - Efficiently verified a lizard
2022-07-24 16:03:54,937 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Obtained an alarm for snake alarm success: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False constructive - Efficiently verified a lizard
2022-07-24 16:03:56,180 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to handle `snake-alarms`
2022-07-24 16:03:56,185 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{function=RECEIVER, first=3, final=3, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:56,186 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Switch{deal with=0, deliveryId=3, deliveryTag=x00, messageFormat=0, settled=false, extra=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[x00Ssxd0x00x00x00)x00x00x00x07@@xa1x0csnake-alarms@@@xa3x10application/jsonx00Suxa0x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake is also a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:56,618 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to handle `snake-alarms`
2022-07-24 16:03:56,622 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Switch{deal with=0, deliveryId=4, deliveryTag=x01, messageFormat=0, settled=false, extra=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[x00Ssxd0x00x00x00)x00x00x00x07@@xa1x0csnake-alarms@@@xa3x10application/jsonx00Suxa0x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake is also a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:56,625 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{function=RECEIVER, first=4, final=4, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:57,105 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to handle `snake-alarms`
2022-07-24 16:03:57,110 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{function=RECEIVER, first=5, final=5, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:57,111 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Switch{deal with=0, deliveryId=5, deliveryTag=x02, messageFormat=0, settled=false, extra=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[x00Ssxd0x00x00x00)x00x00x00x07@@xa1x0csnake-alarms@@@xa3x10application/jsonx00Suxa0x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake is also a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:57,219 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Obtained an alarm for snake alarm success: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False constructive - Efficiently verified a lizard
2022-07-24 16:03:57,534 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to handle `snake-alarms`
2022-07-24 16:03:57,538 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{function=RECEIVER, first=6, final=6, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:57,539 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Switch{deal with=0, deliveryId=6, deliveryTag=x00, messageFormat=0, settled=false, extra=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[x00Ssxd0x00x00x00)x00x00x00x07@@xa1x0csnake-alarms@@@xa3x10application/jsonx00Suxa0x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake is also a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:58,039 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Obtained an alarm for snake alarm success: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False constructive - Efficiently verified a lizard
2022-07-24 16:03:58,041 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Obtained an alarm for snake alarm success: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], Hazard - Efficiently verified a snake
2022-07-24 16:03:58,052 INFO [de.fay.sna.PushNotificationService] (vert.x-eventloop-thread-0) Push notification estimated in 3 sec
2022-07-24 16:03:58,420 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Obtained an alarm for snake alarm success: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False constructive - Efficiently verified a lizard
Whether it is “Hazard – Efficiently verified a snake” or “False constructive – Efficiently verified a lizard” contained in the log: we at all times know if we verified a snake. The push notifications are estimated with 1 second for the sooner and fewer dependable main model and with 3 seconds in case the secondary fallback model succeeds inside the two retries.
Including resilience to Quarkus utilizing Kotlin is kind of straightforward, trigger most of it may be carried out through the use of annotations and unfastened coupling.
If you wish to strive the resilience instance from above, yow will discover it on Github. Have phun and keep curious.
Epilog
Snakes are cool animals. The snake detector is only a enjoyable fiction and doesn’t have the intent to help fears relating to wild animals. Subsequent time I will select a state of affairs serving to animals 🙂 – trigger we must always shield all of them. Purchase the best way: Tardigrades and ants are good examples for very resilient animals!
