Background#

Spring Boot 3 support graceful shutdown natively.

It enables the graceful shutdown feature by default.

# application.yml
server:
  shutdown: graceful
spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s

can refer to the official documentation for more details.

The timeout-per-shutdown-phase property defines how long Spring Boot will wait for ongoing requests to complete before shutting down, which is 30 seconds by default.

@ConfigurationProperties("spring.lifecycle")
public class LifecycleProperties {
   private Duration timeoutPerShutdownPhase = Duration.ofSeconds(30L);

   public Duration getTimeoutPerShutdownPhase() {
      return this.timeoutPerShutdownPhase;
   }

   public void setTimeoutPerShutdownPhase(Duration timeoutPerShutdownPhase) {
      this.timeoutPerShutdownPhase = timeoutPerShutdownPhase;
   }
}

But it’s not enough when running in Kubernetes, we need to configure more to avoid the errors caused by pod termination.

Kubernetes configuration#

In Kubernetes, we need to set the terminationGracePeriodSeconds in the pod spec to a value greater than the Spring Boot shutdown timeout. But it’s default to 30 seconds, which is equal to the Spring Boot default timeout. So we need to set it explicitly to a higher value, for example, 35 seconds.

can refer to the official documentation: pod-termination-flow

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 35
        containers:
            - name: my-app
            image: my-app:latest
            ports:
                - containerPort: 8080
            readinessProbe:
                httpGet:
                    path: /actuator/health/readiness
                    port: 8080
                initialDelaySeconds: 5
                periodSeconds: 10
            livenessProbe:
                httpGet:
                    path: /actuator/health/liveness
                    port: 8080
                initialDelaySeconds: 5
                periodSeconds: 10

But it’s still not enough.

k8s-graceful-shutdown

When Kubernetes sends SIGTERM to the pod, it will also asynchronously notify the service load balancer (like kube-proxy or ingress controller) to remove the pod from the backend list. However, there might be a network delay in this process.

During this delay, if Service A sends requests to Service B, the load balancer might still route requests to the old pod that has already received SIGTERM and stopped accepting new requests. This can lead to connection refused or 503 errors.

To avoid this. We can give some buffer time for the serviceB pod, before it starts the graceful shutdown process.

Kubernetes provides a way to do this using the preStop hook.

kubernetes 1.32 or above

spec:
  containers:
  - name: "example-container"
    image: "example-image"
    lifecycle:
      preStop:
        sleep:
          seconds: 10

Before kubernetes 1.32

spec:
  containers:
  - name: "example-container"
    image: "example-image"
    lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "sleep 10"]

Summary#

  1. The terminationGracePeriodSeconds should be greater than spring.lifecycle.timeout-per-shutdown-phase.
  2. Add a preStop hook to give some buffer time for the load balancer to update the routing table.
  3. preStop hook time + spring.lifecycle.timeout-per-shutdown-phase should be less than terminationGracePeriodSeconds.

example values:

  • terminationGracePeriodSeconds: 35
  • preStop hook time: 5
  • spring.lifecycle.timeout-per-shutdown-phase: 25