Graceful Shutdown in Go: Building a Small Lifecycle Lab
While trying to understand graceful shutdown, I noticed something:
It is easy to explain in writing, but harder to actually see.
So I built a small repo and tested different shutdown scenarios one by one:
The goal of this repository is not to provide a framework.
It is more like a small lifecycle lab.
The point is to make these behaviors visible:
- what happens when a process receives a signal?
- how does the server stop accepting new traffic?
- how are in-flight requests handled?
- how do background workers stop?
- in what order should resources close?
- what happens when the shutdown timeout expires?
- why are readiness and health not the same thing?
Graceful shutdown is often described as a single server.Shutdown(ctx) call.
But in real systems, the problem is not only shutting down the HTTP server.
The real problem is designing the lifecycle of the process.
Experiments
The main part of the repo is the set of small scenarios under experiments/.
Each scenario exists to observe one specific behavior.
Before running the experiments, I start the server:
make run
On startup, the server logs the port and the process PID.
From another terminal, I can check the current process state:
curl localhost:8080/healthz
curl localhost:8080/readyz
curl localhost:8080/state
Some experiments require changing config or timeout values:
CONFIG_PATH=config/server.json make run
ADDR=:8081 make run
SHUTDOWN_TIMEOUT=2s make run
01 - SIGINT
The first scenario is the simplest one.
While the process is running, I send Ctrl+C from the terminal.
make run
Then:
Ctrl+C
What I want to observe here is not just that the process exits.
I want to observe how it exits.
Expected log message bodies to look for, ignoring timestamps:
server starting: addr=:8080 pid=<pid>
runtime unit starting: name=heartbeat-worker
runtime unit starting: name=http-server
worker started: name=heartbeat
shutdown signal received
shutdown started: timeout=15s
runtime unit shutting down: name=http-server
runtime unit shutting down: name=heartbeat-worker
worker stopped: name=heartbeat
runtime resource closing: name=sqlite
shutdown completed
The startup lines are produced by goroutines, so their exact ordering can vary slightly. The shutdown lines are the important part of this experiment.
So the process is not just cut off immediately.
A controlled shutdown flow starts.
02 - SIGTERM
The second scenario triggers the same shutdown flow with SIGTERM.
First I start the server:
make run
From another terminal, I send SIGTERM to the PID printed in the logs:
kill -TERM <pid>
This scenario matters because production shutdowns usually begin with this signal.
In environments like Docker, Kubernetes, and systemd, the process is usually sent SIGTERM first.
The meaning is roughly:
prepare to stop
The expected shutdown log message bodies are the same, ignoring timestamps:
shutdown signal received
shutdown started: timeout=15s
runtime unit shutting down: name=http-server
runtime unit shutting down: name=heartbeat-worker
runtime resource closing: name=sqlite
shutdown completed
03 - Slow Request
In this scenario, I use the intentionally slow /slow endpoint.
First, start the server:
make run
From another terminal, start a slow request:
curl localhost:8080/slow
While the request is still running, trigger shutdown from another terminal:
kill -TERM <pid>
The actual log message bodies to look for are:
slow request started: duration=10s
shutdown signal received
shutdown started: timeout=15s
runtime unit shutting down: name=http-server
slow request completed
runtime unit shutting down: name=heartbeat-worker
worker stopped: name=heartbeat
runtime resource closing: name=sqlite
shutdown completed
So shutdown here is not just about exiting the process.
It is also about waiting for existing work.
The important detail is the position of slow request completed.
It appears after HTTP server shutdown starts, but before shutdown completes. That is the visible sign that the in-flight request was allowed to finish.
04 - Shutdown Timeout
In the previous scenario, the request completed within the shutdown timeout.
This time, I use a shorter timeout:
SHUTDOWN_TIMEOUT=2s make run
Then I start the slow request again:
curl localhost:8080/slow
And send SIGTERM to the process:
kill -TERM <pid>
This time the actual log message bodies are different:
slow request started: duration=10s
shutdown signal received
shutdown started: timeout=2s
runtime unit shutting down: name=http-server
slow request canceled: context canceled
server exited with error: http-server shutdown failed: context deadline exceeded
The server exited with error: ... line is printed by the bootstrap logger to the terminal, not by the application logger in logs/server.log.
Graceful shutdown does not mean waiting forever.
It means trying to shut down cleanly within a bounded amount of time.
One important implementation detail in this repo: when HTTP server shutdown fails because the timeout expires, runtime.ShutdownAll returns an error and resource closing is not reached.
So in this specific timeout scenario I should not expect this log line:
runtime resource closing: name=sqlite
Short version:
graceful shutdown = best effort
05 - Readiness Draining
In this scenario, I look at /healthz and /readyz during shutdown.
With the server running:
make run
First, check readiness in the normal state:
curl -i localhost:8080/readyz
To keep the process in draining long enough to inspect probes, start /slow from another terminal:
curl localhost:8080/slow
Then trigger shutdown from a third terminal:
kill -TERM <pid>
While shutdown is still in progress, check again:
curl -i localhost:8080/healthz
curl -i localhost:8080/readyz
Expected probe behavior:
/healthz -> 200
/readyz -> 503
This distinction matters.
The process may still be alive.
But it should not receive new traffic.
State /healthz /readyz
---------------------------------
starting 200 503
ready 200 200
draining 200 503
stopped down down
Being alive does not mean being ready to receive traffic.
06 - Worker Drain
This scenario looks at background worker behavior instead of HTTP request behavior.
Start the server:
make run
Watch the logs from another terminal:
tail -f logs/server.log
Then send shutdown:
kill -TERM <pid>
What I want to see is that the worker is not interrupted blindly.
Expected log message bodies to look for, ignoring timestamps:
worker started: name=heartbeat
job started: worker=heartbeat job_id=<id> kind=refresh-runtime-state
job completed: worker=heartbeat job_id=<id> kind=refresh-runtime-state age=<duration>
shutdown signal received
shutdown started: timeout=15s
runtime unit shutting down: name=http-server
runtime unit shutting down: name=heartbeat-worker
worker stopped: name=heartbeat
runtime resource closing: name=sqlite
shutdown completed
The worker is part of the lifecycle too.
During shutdown, it is not only the HTTP server that needs coordinated stopping.
Background work also needs to stop in a controlled way.
07 - Second Signal
In this scenario, I send a second signal while shutdown is already in progress.
First, start the server:
make run
Then send the first signal:
kill -TERM <pid>
While shutdown is still running, send another signal:
kill -TERM <pid>
This experiment is slightly different from the others.
The current repo documents the desired second-signal behavior, but it does not implement the hard-exit path yet.
Current behavior:
first signal starts graceful shutdown
second signal does not yet force immediate exit
Target behavior:
first signal graceful shutdown
second signal immediate force exit
This pattern is useful when shutdown blocks unexpectedly, but in this repo it is still a behavior to implement.
08 - Worker Source Failure
In the final scenario, I do not send a signal.
Instead, I intentionally fail the worker’s job source.
This behavior is controlled through config.
In the current repo, the experiment edits config/server.json:
{
"worker_source_fail_after": "7s"
}
Then run:
make run
Actual log message bodies to look for:
worker started: name=heartbeat
job started: worker=heartbeat job_id=<id> kind=refresh-runtime-state
job completed: worker=heartbeat job_id=<id> kind=refresh-runtime-state age=<duration>
worker stopped: name=heartbeat
runtime unit failed: heartbeat-worker: receive job: job source closed
shutdown started: timeout=15s
runtime unit shutting down: name=http-server
runtime unit shutting down: name=heartbeat-worker
runtime resource closing: name=sqlite
server exited with error: runtime unit failed: heartbeat-worker: receive job: job source closed
The server exited with error: ... line is terminal output from the bootstrap logger.
Notice that this path does not log shutdown signal received.
Shutdown is not started by SIGINT or SIGTERM here. It is started because the runtime manager receives a failed unit result from heartbeat-worker.
It also does not log shutdown completed, because App.Run returns the original runtime error after shutdown finishes.
The important distinction is:
job failure != worker failure
If a single job fails, the worker may be able to continue.
But if the source can no longer provide work, the worker itself is no longer healthy.
In that case, starting process shutdown may be the right behavior.
Lifecycle Model
Shutdown is really a state transition.
In this repo, the process moves through these states:
┌──────────┐
│ starting │
└────┬─────┘
│
▼
┌────────┐
│ ready │
└────┬───┘
│
▼
┌──────────┐
│ draining │
└────┬─────┘
│
▼
┌─────────┐
│ stopped │
└─────────┘
The important point is:
shutdown = ready -> draining -> stopped
The process does not go directly from ready to dead.
There is a draining phase in between.
Most of graceful shutdown happens in that phase.
Shutdown Order
Order matters during shutdown.
The flow in the repo looks like this:
SIGTERM
│
▼
state -> draining
│
▼
stop accepting traffic
│
▼
wait in-flight requests
│
▼
stop workers
│
▼
close resources
│
▼
exit
Running things stop first.
Dependencies close after that.
This is a small but important distinction.
If an HTTP request or worker still needs the database while the DB connection is already closed, shutdown may look controlled from the outside while breaking work inside the process.
That is why resources usually close last.
Code Side: How I Produced These Behaviors
To make these scenarios easy to observe, I had to split the process into a few explicit pieces.
Instead of managing everything inside one large main.go, I made the lifecycle visible.
main.go
The entry point is intentionally small.
The most important line is:
ctx, stop := signal.NotifyContext(...)
defer stop()
The idea is:
when SIGINT / SIGTERM arrives, the context is canceled
So the whole system listens to one shutdown signal.
After that, the lifecycle follows this shape:
start -> wait -> shutdown
main.go stays thin.
The actual process behavior is assembled in app and coordinated by the runtime manager.
app: The Process Itself
At first glance, app may look like a wiring layer.
But it has a more important role.
This is where the process defines:
- which units exist
- which resources exist
- in what order they start
- how shutdown proceeds
In other words, this is where the process topology is built.
So I think of app like this:
not a larger main
the process itself
Unit vs Resource
I used two separate concepts here.
A unit is something actively running:
- HTTP server
- worker
A resource is an opened dependency:
- DB
- connection pool
But this distinction is not always clean.
Some components live in a gray area.
For example:
- Kafka consumer
- NATS subscriber
- Redis stream consumer
They look like dependencies.
But they also have an active loop.
So the more useful question is:
is it actively doing work?
If the answer is yes, it is usually better to model it as a runtime unit.
Runtime Manager
The runtime manager is the coordination center of the process.
Its responsibilities are:
- start units
- track status
- manage shutdown order
- observe unit failures
The model looks like this:
┌───────────────┐
│ RuntimeManager│
└──────┬────────┘
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
HTTP Server Worker Worker
│ │ │
▼ ▼ ▼
Done() Done() Done()
│ │ │
└──────► results channel ◄────┘
The key point:
the manager does not poll
it listens
Each unit owns its own lifecycle.
The manager collects lifecycle events.
Done Channel Pattern
Each unit exposes this contract:
Done() <-chan struct{}
This lets the manager wait until a unit finishes:
<-unit.Done()
The flow is:
Worker goroutine
│
▼
[ work loop ]
│
▼
close(done)
│
▼
Manager:
<-Done()
│
▼
status update + result
The important part:
no polling
event-driven lifecycle
This looks small, but it becomes useful as the system grows.
The manager does not need to keep asking whether a unit has finished.
The unit reports completion by closing its done channel.
Lifecycle + Probe
The repo exposes three endpoints:
/healthz
/readyz
/state
/state is especially useful because it shows these things in one response:
- lifecycle state
- readiness
- check results
- unit statuses
- resource statuses
That makes it much easier to understand what the process is doing during shutdown.
It also gives a practical debug surface for observing the draining phase.
The Data Race Problem
In lifecycle-style systems, shared state is one of the easiest things to break.
Multiple parts of the process may touch state at the same time:
- a worker changes its status
- the manager reads or writes status
- a probe endpoint returns a response
- the shutdown flow changes the global lifecycle state
That is why the state layer uses sync.RWMutex.
This is not just a single boolean.
The state contains things like:
- maps
- errors
- lifecycle state
- timestamps
So it is composite state.
And when state is composite, race conditions become much easier to introduce.
Big Picture
The whole structure fits this model:
┌─────────────┐
│ SIGNAL │
└──────┬──────┘
▼
┌─────────────┐
│ LIFECYCLE │
└──────┬──────┘
▼
┌─────────────────────┐
│ RUNTIME MANAGER │
└──────┬──────┬──────┘
│ │
▼ ▼
Units Resources
│ │
▼ ▼
Drain Close
The signal is only the starting point.
The real work is lifecycle coordination.
Conclusion
This repo is not a framework.
It is a small shutdown lab.
But it helped clarify one point:
graceful shutdown = lifecycle design
And the fastest way to understand that is to run it and watch it happen.
Graceful shutdown becomes real in logs, signal flow, readiness behavior, worker shutdown, and timeout handling.
The best way to understand graceful shutdown is to see it while it is happening.