linux - Signal handling in multi-threaded Python

Question

This should be very simple and I'm very surprised that I haven't been able to find this questions answered already on stackoverflow.

I have a daemon like program that needs to respond to the SIGTERM and SIGINT signals in order to work well with upstart. I read that the best way to do this is to run the main loop of the program in a separate thread from the main thread and let the main thread handle the signals. Then when a signal is received the signal handler should tell the main loop to exit by setting a sentinel flag that is routinely being checked in the main loop.

I've tried doing this but it is not working the way I expected. See the code below:

from threading import Thread
import signal
import time
import sys

stop_requested = False    

def sig_handler(signum, frame):
    sys.stdout.write("handling signal: %s
" % signum)
    sys.stdout.flush()

    global stop_requested
    stop_requested = True    

def run():
    sys.stdout.write("run started
")
    sys.stdout.flush()
    while not stop_requested:
        time.sleep(2)

    sys.stdout.write("run exited
")
    sys.stdout.flush()

signal.signal(signal.SIGTERM, sig_handler)
signal.signal(signal.SIGINT, sig_handler)

t = Thread(target=run)
t.start()
t.join()
sys.stdout.write("join completed
")
sys.stdout.flush()

I tested this in the following two ways:

1)

$ python main.py > output.txt&
[2] 3204
$ kill -15 3204

2)

$ python main.py
ctrl+c

In both cases I expect this written to the output:

run started
handling signal: 15
run exited
join completed

In the first case the program exits but all I see is:

run started

In the second case the SIGTERM signal is seemingly ignored when ctrl+c is pressed and the program doesn't exit.

What am I missing here?

Answer 1

The problem is that, as explained in Execution of Python signal handlers:

A Python signal handler does not get executed inside the low-level (C) signal handler. Instead, the low-level signal handler sets a flag which tells the virtual machine to execute the corresponding Python signal handler at a later point(for example at the next bytecode instruction)

…

A long-running calculation implemented purely in C (such as regular expression matching on a large body of text) may run uninterrupted for an arbitrary amount of time, regardless of any signals received. The Python signal handlers will be called when the calculation finishes.

Your main thread is blocked on threading.Thread.join, which ultimately means it's blocked in C on a pthread_join call. Of course that's not a "long-running calculation", it's a block on a syscall…?but nevertheless, until that call finishes, your signal handler can't run.

And, while on some platforms pthread_join will fail with EINTR on a signal, on others it won't. On linux, I believe it depends on whether you select BSD-style or default siginterrupt behavior, but the default is no.

---

So, what can you do about it?

Well, I'm pretty sure the changes to signal handling in Python 3.3 actually changed the default behavior on Linux so you won't need to do anything if you upgrade; just run under 3.3+ and your code will work as you're expecting. At least it does for me with CPython 3.4 on OS X and 3.3 on Linux. (If I'm wrong about this, I'm not sure whether it's a bug in CPython or not, so you may want to raise it on python-list rather than opening an issue…)

On the other hand, pre-3.3, the signal module definitely doesn't expose the tools you'd need to fix this problem yourself. So, if you can't upgrade to 3.3, the solution is to wait on something interruptible, like a Condition or an Event. The child thread notifies the event right before it quits, and the main thread waits on the event before it joins the child thread. This is definitely hacky. And I can't find anything that guarantees it will make a difference; it just happens to work for me in various builds of CPython 2.7 and 3.2 on OS X and 2.6 and 2.7 on Linux…