How do I safely put python signals into a queue?

I am working on python3 script that handles signals (e.g. signal.SIGWINCH) by putting them in a queue. A separate thread puts user input into that same queue, which is all processed by my program's main thread.

Occasionally, the entire program suddenly hangs. I've narrowed down the cause to the interaction between signal handlers and the queue.Queue class. After some small random number of signals have been queued, the signal handler blocks the queue when trying to place into it. Any thread that tries to interact with the queue in any way (eg. by calling queue.put(block=False), queue.get(block=False) or queue.empty()) subsequently hangs as well.

Why do signal handlers block my queue, even when using non-blocking functions? Is there a safe way I can add signals to a queue in a multithreaded program?

---

This can be reproduced by running the simplified code snippet below, while repeatedly resizing the terminal to trigger the event (tested in python 3.13, linux):

from queue import Queue, Empty
import signal

event_queue = Queue()

def signal_handler(signum, frame):
    event_queue.put(signum)

signal.signal(signal.SIGWINCH, signal_handler)

while True:
    try:
        print("Attempting to get an event from the queue...")
        evt = event_queue.get(block=False)
        print("Successfully got event from the queue.")
    except Empty:
        print("The queue is empty, try again.")

Eventually, after resizing the window an indeterminate number of times, the code hangs after the line "Attempting to get an event from the queue..."; i.e., it is hanging on queue.get(). But it shouldn't be, since I have specified block=False. I have also tried using event_queue.put(signum, block=False) in my signal handler within a try/except, but it still hangs.

(...)
Attempting to get an event from the queue...
The queue is empty, try again.
Attempting to get an event from the queue...
(code hangs indefinitely)

If I switch to multiprocessing.Queue or use queue.get(timeout=0.1), I seemingly no longer run into this issue. But both of these approaches have substantial speed cost/delay, and I am concerned whether either are fully safe, as even queue.Queue is supposed to be Read more.

Answer

queue.Queue queues are thread-safe. But you need reentrancy, which is an even harder requirement.

When a signal handler executes, it takes over a thread, interrupting whatever work is happening in that thread. All sorts of data structures could be in inconsistent states at this point. Locks don't solve the problem, because a signal handler can interrupt a thread while it's holding a lock, and execute in that thread, with the lock held, while whatever invariants the lock was supposed to protect are broken.

In this case, it looks like your signal handler is interrupting your get call, while it's holding the queue's lock. Your signal handler tries to call put, which tries to acquire the queue's lock again. Since the queue's lock is non-reentrant, the signal handler deadlocks itself.

---

If you need a queue that can safely be interacted with from signal handlers, the queue module provides queue.SimpleQueue for that. queue.SimpleQueue is specifically designed so that put can be safely called from a thread that is already executing another put or get call.