New PHP MongoDb driver and parallel processes

After refactoring described in my previous article I decided to check if still able to use mongo in parallel processes using pcntl_fork().

So I ran my code responsible for handling several items from the queue in parallel processes (This code was using very useful lib WorkerPool) and got this:

MongoDB\Driver\Exception\RuntimeException: Invalid response_to for query. Expected 23, got 22.

It’s looks like Mongo tries to respond to different process not one was asking for data. Looks strange and I decided to google it by “Invalid response_to for query” and got 2 (yes only two) results and both of them were links to Mongo driver source code on github.

After this I was trying to reconnect to MongoDb in child processes, use semaphore and refactor some code to be sure that my code works as expected. But I was getting the same error again and again. Behavior of the script was strange:

• in 70% it ended with the error mentioned above
• in 20% it worked as expected
• in 10% it was hanging

I tried the same code with old MongoDb driver and it worked. The only difference that my old driver for php 5.5 was talking to MongoDb 3.2 and new driver for php 7.0 was talking to MongoDb 2.6 so I upgraded latter on my VM. And … Nothing

It became clear that the issue is in new driver. And I continued to google.

Finally I found opened bug for php driver.

When using pcntl_fork() with the new MongoDB\Driver it is impossible to create a new MongoDB manager in the child process if one has been opened in the parent process. The newly created one in the child process seem to share a socket to MongoDB and throw errors based on receiving each others responses.

The source of problem is that driver includes the URI string ( MongoDB\Driver\Manager’s first argument) in its hash for persistent sockets. So if you use the same URI in child processes you will get the same socket.

As a work around I decided to add pid of current process to URI in child processes so I’ll be sure I get new hash and new socket in each process:

$manager = new \MongoDB\Driver\Manager(“mongodb://localhost:27017/?x=” . posix_getpid());

This work around fixed mentioned issue but also I realized that I have some places in the code that have to be refactored…