python - Piping output of subprocess.Popen to files

Question

I need to launch a number of long-running processes with subprocess.Popen, and would like to have the stdout and stderr from each automatically piped to separate log files. Each process will run simultaneously for several minutes, and I want two log files (stdout and stderr) per process to be written to as the processes run.

Do I need to continually call p.communicate() on each process in a loop in order to update each log file, or is there some way to invoke the original Popen command so that stdout and stderr are automatically streamed to open file handles?

Answer 1

You can pass stdout and stderr as parameters to Popen()

subprocess.Popen(self, args, bufsize=0, executable=None, stdin=None, stdout=None,
                 stderr=None, preexec_fn=None, close_fds=False, shell=False,
                 cwd=None, env=None, universal_newlines=False, startupinfo=None, 
                 creationflags=0)

For example

>>> import subprocess
>>> with open("stdout.txt","wb") as out, open("stderr.txt","wb") as err:
...    subprocess.Popen("ls",stdout=out,stderr=err)
... 
<subprocess.Popen object at 0xa3519ec>
>>>