Go HTTP Server Graceful Shutdown: Fix the Common Bugs

Your deploy goes out, the pod restarts, and a handful of users get a 502 or a reset connection. The code has a shutdown block, so why did requests still die? Often the snippet you copied was a fragment, and the missing pieces are where the bugs live.

One example is the "handle graceful shutdown from day one" lesson in 7 Lessons I Learned Building Production APIs in Go. It shows a signal channel, a 10-second timeout, and a srv.Shutdown call. It does not show where ListenAndServe runs, what happens when it returns, or when to close the database pool.

This guide fills those gaps. The code here is unexecuted, and this guide makes no claim about whether the original snippet was tested.

What the ten-line snippet leaves out

The article's snippet starts at the point where the server is already running. Four things are missing:

1. The goroutine that runs ListenAndServe, so main is free to wait for a signal.
2. The http.ErrServerClosed check, so a normal shutdown does not look like a crash.
3. Cleanup order. The article says DB connections "close cleanly", but the snippet closes nothing.
4. Load balancer timing. A SIGTERM does not mean traffic has stopped arriving.

The snippet also calls log.Fatalf when shutdown fails. log.Fatalf exits immediately and skips deferred functions, including your cancel() and any cleanup you deferred. The skeleton below returns an exit code from a run() function instead.

Step 1: Set up the project

This skeleton assumes Go 1.22 or newer. Go 1.22 added the "GET /path" method patterns used in ServeMux, and log/slog needs 1.21. There are no third-party dependencies.

mkdir shutdown-demo && cd shutdown-demo
go mod init example.com/shutdown-demo

The pool type below stands in for *pgxpool.Pool or *sql.DB. Swap in your real pool and keep the Close() call where it is.

Step 2: Paste the complete main.go

Status: unexecuted. Compile and run it yourself before trusting it.

package main

import (
	"context"
	"errors"
	"fmt"
	"log/slog"
	"net/http"
	"os"
	"os/signal"
	"strconv"
	"sync/atomic"
	"syscall"
	"time"
)

const (
	drainDelay      = 2 * time.Second
	shutdownTimeout = 10 * time.Second
)

// pool stands in for *pgxpool.Pool or *sql.DB.
type pool struct{}

func (p *pool) Close() { slog.Info("db pool closed") }

func main() { os.Exit(run()) }

func run() int {
	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
	defer stop()

	db := &pool{}
	var ready atomic.Bool
	ready.Store(true)

	mux := http.NewServeMux()
	mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
	})
	mux.HandleFunc("GET /readyz", func(w http.ResponseWriter, r *http.Request) {
		if !ready.Load() {
			http.Error(w, "draining", http.StatusServiceUnavailable)
			return
		}
		w.WriteHeader(http.StatusOK)
	})
	mux.HandleFunc("GET /slow", func(w http.ResponseWriter, r *http.Request) {
		sec, err := strconv.Atoi(r.URL.Query().Get("sec"))
		if err != nil || sec < 1 {
			sec = 5
		}
		select {
		case <-time.After(time.Duration(sec) * time.Second):
			slog.Info("slow request finished", "sec", sec)
			fmt.Fprintf(w, "done after %ds\n", sec)
		case <-r.Context().Done():
			slog.Warn("client went away")
		}
	})

	srv := &http.Server{
		Addr:              ":8080",
		Handler:           mux,
		ReadHeaderTimeout: 5 * time.Second,
	} See [more on check storage layout before upgrading a proxy contract](/check-storage-layout-before-upgrading-a-proxy-contract) for additional background.

	serverErr := make(chan error, 1)
	go func() {
		slog.Info("listening", "addr", srv.Addr)
		if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
			serverErr <- err
		}
	}()

	select {
	case err := <-serverErr:
		slog.Error("server failed", "err", err)
		db.Close()
		return 1
	case <-ctx.Done():
		slog.Info("shutdown signal received")
	}
	stop() // a second Ctrl+C now kills the process the default way

	ready.Store(false)
	slog.Info("readiness off, draining", "delay", drainDelay)
	time.Sleep(drainDelay)

	shutdownCtx, cancel := context.WithTimeout(context.Background(), shutdownTimeout)
	defer cancel()

	exitCode := 0
	if err := srv.Shutdown(shutdownCtx); err != nil {
		slog.Error("graceful shutdown failed, forcing close", "err", err)
		_ = srv.Close()
		exitCode = 1
	}

	db.Close() // only after Shutdown has returned
	slog.Info("shutdown complete", "exit", exitCode)
	return exitCode
}

Step 3: Understand why each piece is there

Why ListenAndServe runs in a goroutine

ListenAndServe blocks until the server stops. If it runs on the main goroutine, nothing can wait for a signal and call Shutdown. Moving it into a goroutine lets main block on ctx.Done().

How to handle http.ErrServerClosed

The net/http documentation says that once Shutdown is called, ListenAndServe returns ErrServerClosed immediately. The program must wait for Shutdown to return rather than exit.

That is why the goroutine ignores ErrServerClosed and reports every other error, such as a port already in use. The waiting happens on the main goroutine, inside the Shutdown call.

Why signal.NotifyContext replaces the channel

signal.NotifyContext gives you a context that cancels on SIGINT or SIGTERM, so there is no buffered channel to size. Calling stop() right after the signal restores default behavior. A second Ctrl+C then terminates the process instead of being swallowed while you wait on a stuck handler.

Do not use this signal context as the server's BaseContext. Request contexts would be cancelled the moment the signal arrives, which defeats the point. The shutdown timeout uses a fresh context.Background() for the same reason: the signal context is already cancelled by then.

Why the pool closes last

In-flight handlers still query the database while Shutdown waits for them. If you close the pool first, those requests fail with errors even though the server looks healthy. db.Close() runs only after Shutdown returns, whether it succeeded or timed out. For more on this, see a closer look at migrate vpc peering to transit gateway: when it pays off.

Step 4: Verify with a slow request and SIGTERM

Build the binary directly. With go run, the PID you signal may belong to the go tool rather than your server. In one terminal, run:

go build -o app . && ./app

In a second terminal, start a 5-second request, then signal the server one second later:

curl -s "localhost:8080/slow?sec=5" &
sleep 1
pkill -TERM -x app
wait

Expected behavior, inferred from the code and not captured from a real run:

1. The server logs shutdown signal received, then readiness off, draining.
2. curl still prints done after 5s roughly four seconds later.
3. The log shows slow request finished before db pool closed. That order is the whole point.
4. The log ends with shutdown complete exit=0, and the process exits with status 0.
5. A new curl started after the drain delay gets "connection refused".

After the process exits, run echo $? in the first terminal to confirm the code. If db pool closed appears before slow request finished, the cleanup order is wrong.

Step 5: Break it on purpose

The shutdown timeout is 10 seconds. Start curl "localhost:8080/slow?sec=20" and send SIGTERM. After the drain delay and the 10-second timeout, Shutdown should return context deadline exceeded, the log should show the forced close, and the exit code should be 1.

Also read: related topic: best webdav mount tool for windows: pick and test one

The client likely sees an aborted connection, because srv.Close() drops it. This is an inference, so confirm it on your machine.

| Failure | Symptom | Fix | |---|---|---| | Handler outlasts the timeout | Exit code 1, context deadline exceeded in logs, client connection reset | Give handlers their own context timeouts shorter than the shutdown timeout. Raise the shutdown timeout only if the work is legitimate. | | Missing ErrServerClosed check, such as log.Fatal(srv.ListenAndServe()) | Process exits the instant Shutdown begins and in-flight requests die | Check errors.Is(err, http.ErrServerClosed) in the goroutine and wait for Shutdown on main. | | DB pool closed before Shutdown returns | In-flight requests fail with pool-closed errors while the server is draining | Close the pool after Shutdown, never from the signal handler or a bare defer that runs early. |

Step 6: Pick a timeout

There is no universal number. Start from your slowest legitimate request, add margin, and keep the total under whatever window your platform allows before it sends SIGKILL. Kubernetes controls that window with terminationGracePeriodSeconds, so check your cluster's value. The drain delay plus the shutdown timeout must fit inside it, or the platform kills the process mid-drain.

Shutdown does not wait for hijacked connections such as WebSockets. If you serve those, register cleanup with srv.RegisterOnShutdown and close them yourself.

Step 7: Account for load balancers and Kubernetes

SIGTERM and endpoint removal are separate events. In Kubernetes, a pod can receive SIGTERM while the load balancer or service proxy still routes new requests to it. Those requests reach a server that has stopped listening, and the client can see a 502 or a reset.

The skeleton handles this in two ways. It flips /readyz to 503 so a readiness probe pulls the pod out of rotation, then sleeps for drainDelay before calling Shutdown. Keep the delay short, and base it on how quickly your proxy or ingress actually stops sending traffic.

Some setups use a preStop sleep instead, which has a similar effect outside the application. The original article does not cover load balancers or Kubernetes, so measure propagation time on your own platform.

What to check next

Once the SIGTERM test passes, look at the components outside this file. A proxy with its own idle timeout, a background worker that is not tied to the signal context, or a Redis client closed in the wrong place can fail in the same way. Apply the same rule to each: stop accepting work, let in-flight work finish, then close the resource underneath it.