My Cleanup Code Waited Forever and Never Said a Word
Kill asks a process to stop. It doesn't make it stop, so a plain wait inside a cleanup trap can quietly skip every cleanup step after it.

The cleanup code looked right. I read it twice and would have approved it in review without thinking. It's the kind of code you trust because it's written in the careful order.
Here's the setup. A shell script started cloud-sql-proxy in the background. That's a small program that opens a connection to a database in Google Cloud so the rest of the script can talk to it. "In the background" means the script starts it, keeps its process ID in a variable, and moves on while the proxy keeps running.
When a script like that finishes, you want to shut the proxy down and clean up after it. Shell gives you a way to do that called a trap. A trap is a bit of code you register up front that runs when the script exits, whether it finished normally or failed partway. Mine looked roughly like this:
trap 'kill "$pid"; wait "$pid"; rm -f "$logfile"' EXIT
Stop the proxy. Wait for it to actually be gone. Then delete its log file, because you don't want to delete a file the proxy might still be writing to. That's the careful order. That's why it looks right.
What kill actually does
The word kill sounds final. It isn't. It sends a signal to the process, and by default that's SIGTERM, which is a polite request to shut down. The process gets to decide what to do with it. Most programs clean up and exit quickly. Some take their time. Some catch the signal and keep going.
wait does exactly what it says. It sits there until the process exits. It has no timeout. If the process is slow to exit, or never exits, wait never returns.
Put those two together inside a trap and look at what happens on the bad day. The proxy gets the request, doesn't leave, and the trap sits on wait. The rm after it never runs. Nothing prints an error, because nothing failed. The script is just stuck, or whatever ran it eventually gives up on it from the outside, and the trap never gets the chance to finish what it was doing.
And on every normal run, the proxy exits right away when asked. The wait returns in a blink and the log gets deleted. The happy path passes every single time. That's why it gets through review and through testing. The order is right. The thing it assumes, that the process will leave when asked, just isn't promised anywhere.
The part that bugs me is where this lives. A trap is where you'd normally notice that something went wrong. It's the code that runs when things break. So when the trap itself hangs, the place you'd go looking for the problem is the place that's stuck.
The fix is giving up on purpose
The fix is to put a limit on the wait. Ask nicely, check for a few seconds whether the process is gone, and if it isn't, stop asking:
kill "$pid" 2>/dev/null || true
for i in 1 2 3 4 5 6 7 8 9 10; do kill -0 "$pid" 2>/dev/null || break; sleep 0.5; done
kill -9 "$pid" 2>/dev/null || true
wait "$pid" 2>/dev/null || true
rm -f "$logfile"
kill -0 sends no signal at all. It just asks whether the process still exists. After five seconds, kill -9 sends SIGKILL, which the process can't catch or ignore. Then the cleanup runs, every time.
Its cousin in the same script
This showed up next to another line with the same personality. The script ran with set -euo pipefail, which is the common "be strict" setting. Among other things, it stops the script if any command fails, and it counts a pipeline as failed if any piece of it fails, not just the last piece.
Then there was a line like grep "ready" "$logfile" | tail -1. If grep finds no match, it exits with a non-zero code. That's how it says "nothing found." tail is perfectly happy and exits zero. But under pipefail the whole line counts as a failure, and the script stops right there. The fix is to say out loud that no match is fine: { grep "ready" "$logfile" || true; } | tail -1.
Both lines read as correct. Both pass on a normal run. And both quietly change what the rest of the script does, one by never moving on, the other by stopping early. Neither tells you.
The habit I took from this is simple. Anything that can block gets a time limit, and anything that can exit non-zero on purpose gets told that's okay. If you write cleanup code, assume the thing you're cleaning up won't cooperate.
References: the GNU Bash manual, for trap, wait, kill and set -o pipefail.
Beckett wrote a whole play about two men who can't leave until someone shows up who never does. A wait with no timeout is the same play, just with nobody watching.