Intermediate

While working in Python you may come across cases where you want a quick way to run programs you have already written, without tampering with the source code – cases like this are where the subprocess module can come into play.

The subprocess module, present in a standard installation of Python is used to run new applications or programs through Python code by creating new processes.

It gives us the ability to:

1. spawn new processes
2. connect to their input, output and error pipes
3. obtain their return codes

NOTE: The subprocess module works better on a Linux system as quite a few of the operations are Linux commands.

How to use subprocess to list files in a directory

A common use for subprocess would be to list files and folders in a directory. This makes use of the run method of subprocess.

The code will differ depending on what system you are on.

For Linux

import subprocess

subprocess.run('ls')

For Windows

import subprocess

subprocess.run("dir", shell=True)

In the code above we

• import subprocess
• call the run module from it
• pass the list directory command based on your system (ls/dir)
  • if you are on Windows you will have to additionally pass shell=True because dir is a shell command and you need to tell the system that you want to use it.

We automatically got output in the terminal even though we did not print it ourselves. That is how the run command works – it does not capture our command output by default.

Let’s see what happens if we try to capture our command by placing it in a variable

import subprocess

result = subprocess.run("dir", shell=True)
print(result)

The output is not what you would expect. We get a message containing the arguments that were passed and the return code.

If we wanted to capture the output we would modify out code to this:

import subprocess

result = subprocess.run(["dir"], shell=True, capture_output=True, text=True)

print(result.stdout)

We have added:

• capture_output=True: to capture the output
• text=True: to decode the output to a readable format since it is captured as bytes

We then print result.stdout : the result as standard output

The same can be achieved with the follow code:

import subprocess

result = subprocess.run(["ls", "-la"], stdout=subprocess.PIPE, text=True)

print(result.stdout)

The difference is stdout=subprocess.PIPE This code performs the same function as capture_output=True

In addition, we use the ls command with the argument of -la both passed as a list

How to redirect subprocess output to a file

Having the output of our command in the console is not that useful. It would serve use better if we could store it in a file. This is possible by modifying our code to look like this:

import subprocess

with open("output.txt", "w") as file:
   result = subprocess.run(["ls", "-la"], stdout=file, text=True)

Instead of having the output display in the terminal, we set the stdout to our file.

When we run this code: python sub.py it does not look like anything happened. However, if we check our folder a text file, with the output we had prior has been created

How to read the contents of a file with subprocess

Now let’s see how we can use subprocess to read the contents of a file. Below is a text file with a line of text

This is how we would read the contents of it:

import subprocess

result = subprocess.run(["cat", "file.txt"], capture_output=True, text=True)

print(result.stdout)

In the above code, we pass in our list; the cat command and the file name. The rest is the same as before

NOTE: The cat command allows us to create files, view the contents of a file, concatenate them, and redirect their output.

Additionally, we can use this output as input to access specific parts of the file. We do this using the file we wrote to previosly

import subprocess

result = subprocess.run(["cat", "output.txt"], capture_output=True, text=True)

# print(result.stdout)
input = subprocess.run(
    ["grep", "-n", "sub.py"], capture_output=True, text=True, input=result.stdout
)

print(input.stdout)

After running the cat command on output.txt, we do not print that output. Instead, we use it as the input of the next command – input

In the second list of our input line of code:

• we use grep to search for a string of text in the file
• pass the -n flag to specify we are looking for the entire line of text
• add “sub.py” as what we are searching for

Lastly, we print the output of this second command

We have successfully searched for a line of text in our file and output it.

How to call an external command using subprocess

Now that we have seen several ways we can make use of subprocess, let’s have a look at something more interesting…

If you remember, we built a script that gives us current weather in this article: Click Package To Create Command Line Interfaces in Python

We ran that program with this command

[program name] [location] -a [api_key]

Now, we are going to run it through subprocess.

In that same directory, create a new file and type the command we used to run the weather program in the run command.

For me:

import subprocess

subprocess.run("weather Harare -a [your api key]", shell=True)

After importing subprocess, we have added the same command we used to run our weather program the parentheses and quotation marks.

As you see, we are able to get the script to execute without touching weather.py

How to use subprocess.Popen to start programs

Another feature of subprocess is the Popen class. Yes, class. It gives us the ability to open files and or programs from Python.

Consider the code below:

import subprocess

subprocess.Popen("start excel", shell=True)

Executing the code above will start Excel. Alternatively, you could substitute excel with any other program on your system to open it with Python.

You could also choose to go further and create a CLI to run the command more efficiently.

---

In this article, we have taken a look at the subprocess module – what it is and what it is used for. We have used it to list files in a directory, output that results to a file and read the contents of a file. We have also used subprocess to execute a program we created before and we have seen how we can run open programs using the Popen class of subprocess.