While working in Python you may come across cases where you want a quick way to run programs you have already written, without tampering with the source code – cases like this are where the subprocess module can come into play.
The subprocess module, present in a standard installation of Python is used to run new applications or programs through Python code by creating new processes.
It gives us the ability to:
- spawn new processes
- connect to their input, output and error pipes
- obtain their return codes
subprocess module works better on a Linux system as quite a few of the operations are Linux commands.
How to use subprocess to list files in a directory
A common use for
subprocess would be to list files and folders in a directory. This makes use of the
run method of
The code will differ depending on what system you are on.
import subprocess subprocess.run('ls')
import subprocess subprocess.run("dir", shell=True)
In the code above we
- import subprocess
- call the run module from it
- pass the list directory command based on your system (ls/dir)
- if you are on Windows you will have to additionally pass
diris a shell command and you need to tell the system that you want to use it.
- if you are on Windows you will have to additionally pass
We automatically got output in the terminal even though we did not
Let’s see what happens if we try to capture our command by placing it in a variable
import subprocess result = subprocess.run("dir", shell=True) print(result)
The output is not what you would expect. We get a message containing the arguments that were passed and the return code.
If we wanted to capture the output we would modify out code to this:
import subprocess result = subprocess.run(["dir"], shell=True, capture_output=True, text=True) print(result.stdout)
We have added:
- capture_output=True: to capture the output
- text=True: to decode the output to a readable format since it is captured as bytes
We then print
result.stdout : the result as standard output
The same can be achieved with the follow code:
import subprocess result = subprocess.run(["ls", "-la"], stdout=subprocess.PIPE, text=True) print(result.stdout)
The difference is
stdout=subprocess.PIPE This code performs the same function as
In addition, we use the
ls command with the argument of
-la both passed as a list
How to redirect subprocess output to a file
Having the output of our command in the console is not that useful. It would serve use better if we could store it in a file. This is possible by modifying our code to look like this:
import subprocess with open("output.txt", "w") as file: result = subprocess.run(["ls", "-la"], stdout=file, text=True)
Instead of having the output display in the terminal, we set the
stdout to our file.
When we run this code:
python sub.py it does not look like anything happened. However, if we check our folder a text file, with the output we had prior has been created
How to read the contents of a file with subprocess
Now let’s see how we can use
subprocess to read the contents of a file. Below is a text file with a line of text
This is how we would read the contents of it:
import subprocess result = subprocess.run(["cat", "file.txt"], capture_output=True, text=True) print(result.stdout)
In the above code, we pass in our list; the cat command and the file name. The rest is the same as before
NOTE: The cat command allows us to create files, view the contents of a file, concatenate them, and redirect their output.
Additionally, we can use this output as input to access specific parts of the file. We do this using the file we wrote to previosly
import subprocess result = subprocess.run(["cat", "output.txt"], capture_output=True, text=True) # print(result.stdout) input = subprocess.run( ["grep", "-n", "sub.py"], capture_output=True, text=True, input=result.stdout ) print(input.stdout)
After running the
cat command on output.txt, we do not print that output. Instead, we use it as the input of the next command – input
In the second list of our input line of code:
- we use grep to search for a string of text in the file
- pass the -n flag to specify we are looking for the entire line of text
- add “sub.py” as what we are searching for
Lastly, we print the output of this second command
We have successfully searched for a line of text in our file and output it.
How to call an external command using subprocess
Now that we have seen several ways we can make use of
subprocess, let’s have a look at something more interesting…
If you remember, we built a script that gives us current weather in this article: Click Package To Create Command Line Interfaces in Python
We ran that program with this command
[program name] [location] -a [api_key]
Now, we are going to run it through subprocess.
In that same directory, create a new file and type the command we used to run the weather program in the run command.
import subprocess subprocess.run("weather Harare -a [your api key]", shell=True)
subprocess, we have added the same command we used to run our weather program the parentheses and quotation marks.
As you see, we are able to get the script to execute without touching
How to use subprocess.Popen to start programs
Another feature of
subprocess is the Popen class. Yes, class. It gives us the ability to open files and or programs from Python.
Consider the code below:
import subprocess subprocess.Popen("start excel", shell=True)
Executing the code above will start Excel. Alternatively, you could substitute excel with any other program on your system to open it with Python.
You could also choose to go further and create a CLI to run the command more efficiently.
In this article, we have taken a look at the subprocess module – what it is and what it is used for. We have used it to list files in a directory, output that results to a file and read the contents of a file. We have also used subprocess to execute a program we created before and we have seen how we can run open programs using the Popen class of subprocess.