subprocess - Interactive input/output using Python

Question

I have a program that interacts with the user (acts like a shell), and I want to run it using the Python subprocess module interactively. That means, I want the possibility to write to standard input and immediately get the output from standard output. I tried many solutions offered here, but none of them seems to work for my needs.

The code I've written is based on Running an interactive command from within Python.

import Queue
import threading
import subprocess
def enqueue_output(out, queue):
    for line in iter(out.readline, b''):
        queue.put(line)
    out.close()

def getOutput(outQueue):
    outStr = ''
    try:
        while True: # Adds output from the queue until it is empty
            outStr += outQueue.get_nowait()

    except Queue.Empty:
        return outStr

p = subprocess.Popen("./a.out", stdin=subprocess.PIPE, stout=subprocess.PIPE, stderr=subprocess.PIPE, bufsize = 1)
#p = subprocess.Popen("./a.out", stdin=subprocess.PIPE, stout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False, universal_newlines=True)

outQueue = Queue()
errQueue = Queue()

outThread = Thread(target=enqueue_output, args=(p.stdout, outQueue))
errThread = Thread(target=enqueue_output, args=(p.stderr, errQueue))

outThread.daemon = True
errThread.daemon = True

outThread.start()
errThread.start()

p.stdin.write("1
")
p.stdin.flush()
errors = getOutput(errQueue)
output = getOutput(outQueue)

p.stdin.write("5
")
p.stdin.flush()
erros = getOutput(errQueue)
output = getOutput(outQueue)

The problem is that the queue remains empty, as if there is no output. Only if I write to standard input all the input that the program needs to execute and terminate, then I get the output (which is not what I want). For example, if I do something like:

p.stdin.write("1
5
")
errors = getOutput(errQueue)
output = getOutput(outQueue)

Is there a way to do what I want to do?

---

The script will run on a Linux machine. I changed my script and deleted the universal_newlines=True + set the bufsize to 1 and flushed standard input immediately after write. Still I don't get any output.

Second try:

I tried this solution and it works for me:

from subprocess import Popen, PIPE

fw = open("tmpout", "wb")
fr = open("tmpout", "r")
p = Popen("./a.out", stdin = PIPE, stdout = fw, stderr = fw, bufsize = 1)
p.stdin.write("1
")
out = fr.read()
p.stdin.write("5
")
out = fr.read()
fw.close()
fr.close()

Answer 1

Two solutions for this issue on Linux:

First one is to use a file to write the output to, and read from it simultaneously:

from subprocess import Popen, PIPE

fw = open("tmpout", "wb")
fr = open("tmpout", "r")
p = Popen("./a.out", stdin = PIPE, stdout = fw, stderr = fw, bufsize = 1)
p.stdin.write("1
")
out = fr.read()
p.stdin.write("5
")
out = fr.read()
fw.close()
fr.close()

Second, as J.F. Sebastian offered, is to make p.stdout and p.stderr pipes non-blocking using fnctl module:

import os
import fcntl
from subprocess import Popen, PIPE  
def setNonBlocking(fd):
    """
    Set the file description of the given file descriptor to non-blocking.
    """
    flags = fcntl.fcntl(fd, fcntl.F_GETFL)
    flags = flags | os.O_NONBLOCK
    fcntl.fcntl(fd, fcntl.F_SETFL, flags)

p = Popen("./a.out", stdin = PIPE, stdout = PIPE, stderr = PIPE, bufsize = 1)
setNonBlocking(p.stdout)
setNonBlocking(p.stderr)

p.stdin.write("1
")
while True:
    try:
        out1 = p.stdout.read()
    except IOError:
        continue
    else:
        break
out1 = p.stdout.read()
p.stdin.write("5
")
while True:
    try:
        out2 = p.stdout.read()
    except IOError:
        continue
    else:
        break