python - multiprocessing.pool.MaybeEncodingError: 'TypeError("cannot serialize '_io.BufferedReader' object",)'

Question

Why does the code below work only with multiprocessing.dummy, but not with simple multiprocessing.

import urllib.request
#from multiprocessing.dummy import Pool #this works
from multiprocessing import Pool

urls = ['http://www.python.org', 'http://www.yahoo.com','http://www.scala.org', 'http://www.google.com']

if __name__ == '__main__':
    with Pool(5) as p:
        results = p.map(urllib.request.urlopen, urls)

Error :

Traceback (most recent call last):
  File "urlthreads.py", line 31, in <module>
    results = p.map(urllib.request.urlopen, urls)
  File "C:UserspatriAnaconda3libmultiprocessingpool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:UserspatriAnaconda3libmultiprocessingpool.py", line 657, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '[<http.client.HTTPResponse object at 0x0000016AEF204198>]'. Reason: 'TypeError("cannot serialize '_io.BufferedReader' object")'

What's missing so that it works without "dummy" ?

Answer 1

The http.client.HTTPResponse-object you get back from urlopen() has a _io.BufferedReader-object attached, and this object cannot be pickled.

pickle.dumps(urllib.request.urlopen('http://www.python.org').fp)
Traceback (most recent call last):
...
    pickle.dumps(urllib.request.urlopen('http://www.python.org').fp)
TypeError: cannot serialize '_io.BufferedReader' object

multiprocessing.Pool will need to pickle (serialize) the results to send it back to the parent process and this fails here. Since dummy uses threads instead of processes, there will be no pickling, because threads in the same process share their memory naturally.

A general solution to this TypeError is:

1. read out the buffer and save the content (if needed)
2. remove the reference to '_io.BufferedReader' from the object you are trying to pickle

In your case, calling .read() on the http.client.HTTPResponse will empty and remove the buffer, so a function for converting the response into something pickleable could simply do this:

def read_buffer(response):
    response.text = response.read()
    return response

Example:

r = urllib.request.urlopen('http://www.python.org')
r = read_buffer(r)
pickle.dumps(r)
# Out: b'x80x03chttp.client
HTTPResponse...

Before you consider this approach, make sure you really want to use multiprocessing instead of multithreading. For I/O-bound tasks like you have it here, multithreading would be sufficient, since most of the time is spend in waiting (no need for cpu-time) for the response anyway. Multiprocessing and the IPC involved also introduces substantial overhead.