Flushing to disk while holding the write lock put every read in a queue for over ten seconds.
The problem was writing to disk while holding the write lock: for as long as the file write took, every read waited in line, sometimes for over ten seconds.
A harder one to spot is a non-reentrant read-write lock: code that already holds the write lock tries to take it again and deadlocks itself, with no error at all — the process just stops there.
Now locks only guard in-memory data structures, and file I/O happens outside them.
Written byShihao Qi