How To Read And Write To Files In Python

How To Read And Write To Files In Python

Beginner

Reading and writing is one of the most fundamental things you will do in Python. There are many variations of this of whether you are working with a text file, a comma separated file, or even image files. We will explore all of these and more in this ultimate guide to reading and writing files!

For a given file, regardless of the variation of what type of file we are working with, the fundamental is the same: we need to open it first, we perform the read/write operation, then we close the file. Hence, we typically have these three steps for any file interaction. Sometimes though, the close maybe automatic when the program finishes, but it is best practice to manually close the file each time.

1. Read and Write a Text File in Python

When accessing a file, we could do many things such as read from it, write from it etc. In Python this is specified by providing a File Access Modes. This is the list of the most commonly used access modes which you can use when you work with text files:

Read Only r
Read and Writer+
Write Onlyw
Append Onlya
Append and Writea+
Table – File Accessing Modes

a) Writing To A Text File In Python

Here we learn how to write data in the file

  • Open a file using the open() in w mode. If you have to read and write data using a file, then open it in an r+ mode.
  • Write the data to the file using write() or writelines() method.
  • Close the file.

Let us look at the given example to understand it better:

#for opening a file in 'w'
file= open('sample.txt','w')
# write() - for write direct text to the file
# writelines() - for write multiple lines or strings at a time 
file.write("How to program python.\nI am a programmer.")
file.close # closing the file

Now you will find sample.txt in your directory.

b) Reading From A Text File in Python

In the above section, we see how to write data to a file, Now we learn how to read data that we have written into a file.

  • Open a file using the open() in r mode. If you have to read and write data using a file, then open it in a r+ mode.
  • Read data from the file using read() or readline() or readlines() methods. Store the data in a variable.
  • Display the data.
  • Close the file.

Again let us look at the given example to understand it better:

Now our output display like this way;

# for opening a file in 'r'
file = open('sample.txt','r')
# read()- this reads all content from a file
# readline()- this reads one line by default or number of lines you specify
# readlines()-it used to read all the lines from a file, it returns a list 
data = file.read()

print(data) # for printing the data
file.close() # for closing the file

Output-

How to program python.
I am a programmer

2. Read & Write A JSON File

A JSON (or JavaScript Object Notation) format is a popular file format which can be extremely helpful to manage complex objects in Python and convert them to a human readable, hierarchical, text format. JSON format does represent data types in different ways which can be seen here:

Python objectJSON object
dictobject
list, tuplearray
strstring
int, long, floatnumbers
Truetrue
Falsefalse
Nonenull
Data type conversion from Python native format to JSON format

a) Writing an object to a JSON file in Python

We use json.dumps() method in python for writing JSON to a file in Python.

It takes 2 parameters:

  1. dictionary – the name of the dictionary which should be converted to a JSON object.
  2. file pointer – pointer of the file opened in write or append mode.
#python program to write JSON to a file
import json
#Data to be written
dictionary = {"name":"charleswhite",
              "zipcode" : 1230,
              "address" : "polland"
}
with open("sample.json","w") as outfile:
  json.dump( dictionary, outfile)
Output :
{ "name":"charleswhite","zipcode":1230,"address":"polland"}

b) Reading JSON from a file using python

The JSON package has json.load() function that loads the JSON content from a JSON file into a dictionary.

It takes one parameter:

  • File pointer: A file pointer that points to a JSON file.
# python program to read JSON from a file
import json
# opening JSON file
with open('sample.json','r') as openfile: 
  json_object=json.load(openfile)  # Reading from json file
  print(json_object)

print(type(json_object))
Output :

3. Reading And Writing Tab-Delimited File in Python

Tab-delimited files have traditionally been used to communicate between enterprise systems where both the structured nature of the data, and the small file size has been quite appealing in these scenarios. Typically each column is separated (or delimited) by a tab \t character, and each row is separated by a newline character \n.

myFile = open("somefile","r")
for aRow in myFile:
  print( aRow.split('\t') )
myFile.close()

Writing Tab-Delimited File: The writing case is just opposite to the reading scenario. we use a “\t”.join( someList ) to create the tab-delimited row. Just see the example :

list_header = ['id', 'customer_name', 'address']
list_items = [ [ 1, 'Homer Simpson', '1 Rodeo drive'],
               [ 2, 'Ned Flanders', '3 Rodeo drive'] ]

test = open("boats.tab","w")
test.write("\t".join( list_header ) )
test.write("\n")
for item in list_items:
  test.write( "\t".join( item ) )
  test.write("\n")
test.close()

4. Read and Write a Binary file in Python

For writing a binary file we use file.write()

In this scenario you must call open(file,mode) with mode as “wb” to open the file in write binary mode. Call file.write(str) with the opened file as the file to append a sequence of bytes str.

file = open("document.bin","wb")
file.write("This binary string will be written to document.bin")
file.close()

For reading the file we can take our document.bin file and used the “rb” mode. Here we use our read() function to read the document.bin file.  The read() method returns the specified number of bytes from the file.

Here we use print(file.read(4)) so it will read only four bytes.

file= open("document.bin","rb")
print(file.read(4))
file.close()

5. How to Read and Write an Image in Python

To read an image we can use the library cv2.imread() function. We provide a certain path for the image and that will read the file.

import numpy as np
import cv2
# Load an color image in grayscale
img = cv2.imread('Top-bike-wallpaper.jpg',0)

To display an image in a window, use cv2.imshow() function.

# Display the image
cv2.imshow('image',img)
# key binding function
cv2.waitkey(0)
# Destroyed all window we created earlier.
cv2.destroyAllwindows()

On running the following code, our image will look like this

kawasaki bike, Ninja H2, Rs 3330000/piece India Kawasaki Motors Pvt. Ltd |  ID: 19715432012
Image: Bike

For Writing an Image, we use a cv2.imwrite() function to save an image. Here we take the first argument as a file name and the second to save the image which I want.

cv2.imwrite('messigray.png',img)

6. Load a Dictionary or List to A File with Pickle in Python

Pickle is a library that preserves a given object’s structure when you save it to a file. You can think of it as a persistent storage of your data. To load a dictionary from a file with Pickle in Python, simply we use pickle.load(file) with the file object as a file to get the dictionary stored in it.

import pickle

file_to_read =  open("data.pickle","rb")
loaded_dictionary = pickle.load( file_to_read )

print( loaded_dictionary )

Output :

{'a': 1}

7. Reading and Writing Lists to A File in Python

This is similar to reading / writing text files. Here, the intention is to write all the items which are separated by each line. When the file is ready, you can then reverse the operation and read in the file.

# List
item_list =['Charles','White','Python']

with open('cwp.txt','wt') as f:    
    for items in item_list:
         f.write('%s\n' % items)   # write elements of list
    print("File written successfully")

f.close() # close the file

In the above example we see how to write a file. Now we see how to read a file by using r mode. The data read from the file is printed.

# open file in read mode
f= open('cp.txt','r')

# display content of the file 
print( f.read() )

#close the file 
f.close()

Output :

Charles
White
Python

8. Reading And Writing CSV Files in Python

A CSV (comma-separated values) file is a text file that has a specific format that allows data to be saved in a table structured format. To represent a CSV file, the convention is to name the file with the .csv file extension.

First we open CSV file using open() in r mode then use reader() method. Here is some syntax that will consider:

csv.reader(csvfile, dialect='excel', **fmtparams)

Let’s see it with an example :

A sheet
import csv

with open('Giants.csv', mode='r')as file:  # opening  the CSV file
  csvFile = csv.reader(file)  # reading the CSV file

# display the contents of the CSV file
for lines in csvFile:
  print(lines)

Output :

'Charles', 4, 'A' 
'Robert', 8, 'B' 
'Gravie', 5, 'A'
'Maggie', 4, 'C'

Here we use writer.csv to insert the data and some syntax which will consider as :

csv.writer(csvfile, dialect='excel', **fmtparams)

To write a single row, we use writerow() method was writing multiple rows we use writerows() function.

Let’s understand with an example :

# Python program to demonstrate writing to CSV
import csv

fields=['Name','Class','Grade','Year'] # field names

# data rows of csv file
rows = [['Charles','4','A','2015'],
       ['Robert','8','B','2020'],
       ['Gravie','5','A','2022'],
       ['Maggie','4','C','2021']],

filename="university_records.csv"   # name of csv file
  
# writing to csv file
with open(filename,'w') as csvfile:
  csvwriter = csv.writer(csvfile)  # creating a csv writer object
  csvwriter.writerow(fields)  # writing the fields
  csvwriter.writerows(rows)   # writing the data rows

Conclusion

Writing and reading of files is not a complex task and there are many ways to achieve the same thing either by using libraries, or by performing some of the transformation yourself at the point of file read/write.

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!
Use subprocess To Execute Commands In Python 3

Use subprocess To Execute Commands In Python 3

Intermediate

While working in Python you may come across cases where you want a quick way to run programs you have already written, without tampering with the source code – cases like this are where the subprocess module can come into play.

The subprocess module, present in a standard installation of Python is used to run new applications or programs through Python code by creating new processes.

It gives us the ability to:

  1. spawn new processes
  2. connect to their input, output and error pipes
  3. obtain their return codes

NOTE: The subprocess module works better on a Linux system as quite a few of the operations are Linux commands.

How to use subprocess to list files in a directory

A common use for subprocess would be to list files and folders in a directory. This makes use of the run method of subprocess.

The code will differ depending on what system you are on.

For Linux

import subprocess

subprocess.run('ls')

For Windows

import subprocess

subprocess.run("dir", shell=True)


In the code above we

  • import subprocess
  • call the run module from it
  • pass the list directory command based on your system (ls/dir)
    • if you are on Windows you will have to additionally pass shell=True because dir is a shell command and you need to tell the system that you want to use it.

We automatically got output in the terminal even though we did not print it ourselves. That is how the run command works – it does not capture our command output by default.

Let’s see what happens if we try to capture our command by placing it in a variable

import subprocess

result = subprocess.run("dir", shell=True)
print(result)

The output is not what you would expect. We get a message containing the arguments that were passed and the return code.

If we wanted to capture the output we would modify out code to this:

import subprocess

result = subprocess.run(["dir"], shell=True, capture_output=True, text=True)

print(result.stdout)

We have added:

  • capture_output=True: to capture the output
  • text=True: to decode the output to a readable format since it is captured as bytes

We then print result.stdout : the result as standard output

The same can be achieved with the follow code:

import subprocess

result = subprocess.run(["ls", "-la"], stdout=subprocess.PIPE, text=True)

print(result.stdout)

The difference is stdout=subprocess.PIPE This code performs the same function as capture_output=True

In addition, we use the ls command with the argument of -la both passed as a list

How to redirect subprocess output to a file

Having the output of our command in the console is not that useful. It would serve use better if we could store it in a file. This is possible by modifying our code to look like this:

import subprocess

with open("output.txt", "w") as file:
   result = subprocess.run(["ls", "-la"], stdout=file, text=True)

Instead of having the output display in the terminal, we set the stdout to our file.

When we run this code: python sub.py it does not look like anything happened. However, if we check our folder a text file, with the output we had prior has been created

How to read the contents of a file with subprocess

Now let’s see how we can use subprocess to read the contents of a file. Below is a text file with a line of text

This is how we would read the contents of it:

import subprocess

result = subprocess.run(["cat", "file.txt"], capture_output=True, text=True)

print(result.stdout)

In the above code, we pass in our list; the cat command and the file name. The rest is the same as before

NOTE: The cat command allows us to create files, view the contents of a file, concatenate them, and redirect their output.

Additionally, we can use this output as input to access specific parts of the file. We do this using the file we wrote to previosly

import subprocess

result = subprocess.run(["cat", "output.txt"], capture_output=True, text=True)

# print(result.stdout)
input = subprocess.run(
    ["grep", "-n", "sub.py"], capture_output=True, text=True, input=result.stdout
)

print(input.stdout)

After running the cat command on output.txt, we do not print that output. Instead, we use it as the input of the next command – input

In the second list of our input line of code:

  • we use grep to search for a string of text in the file
  • pass the -n flag to specify we are looking for the entire line of text
  • add “sub.py” as what we are searching for

Lastly, we print the output of this second command

We have successfully searched for a line of text in our file and output it.

How to call an external command using subprocess

Now that we have seen several ways we can make use of subprocess, let’s have a look at something more interesting…

If you remember, we built a script that gives us current weather in this article: Click Package To Create Command Line Interfaces in Python

We ran that program with this command

[program name] [location] -a [api_key]

Now, we are going to run it through subprocess.

In that same directory, create a new file and type the command we used to run the weather program in the run command.

For me:

import subprocess

subprocess.run("weather Harare -a [your api key]", shell=True)

After importing subprocess, we have added the same command we used to run our weather program the parentheses and quotation marks.

As you see, we are able to get the script to execute without touching weather.py

How to use subprocess.Popen to start programs

Another feature of subprocess is the Popen class. Yes, class. It gives us the ability to open files and or programs from Python.

Consider the code below:

import subprocess

subprocess.Popen("start excel", shell=True)

Executing the code above will start Excel. Alternatively, you could substitute excel with any other program on your system to open it with Python.

You could also choose to go further and create a CLI to run the command more efficiently.


In this article, we have taken a look at the subprocess module – what it is and what it is used for. We have used it to list files in a directory, output that results to a file and read the contents of a file. We have also used subprocess to execute a program we created before and we have seen how we can run open programs using the Popen class of subprocess.

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!
Click Package To Create Command Line  Interfaces in Python

Click Package To Create Command Line Interfaces in Python

Intermediate

There is a simpler way create command line interfaces (CLIs) in python using the click package. When we made CLIs in this article on how to parse arguments we made use of the argparse module. This worked fine but there is also another way we can accomplish this.

Today we will look at a package in Python that gives us the ability to build better more intuitive command line interfaces – the click package. The click package uses decorators as a mechanism to making arguments easier (see our article on the Simple Guide to Decorators)

This article assumes you have already read our articles on command line interfaces.

Overview of Click Command Line Argument Python Package

Click is a package for creating cleaner command line interfaces in a simple way with as little code as necessary. It makes use of the functionality of argparse and the efficiency of decorators.

Before we start using click we have to install it first – it does not come with the standard installation of Python. In your command prompt or terminal, you can install via pip:

pip install click

We will start with something simple – a script that outputs “Hello World!”

import click

@click.command()
def main():
    click.echo("Hello World from click")

if __name__ == "__main__":
    main()

In the code above we:

  • import click package
  • decorate the main function with @click.command() – this gives us access to the modules from click. This turns main into a click command
  • we input click.echo() to display output

When using click it is advisable to use click.echo for output instead of print. click.echo takes into account python 2 and python 3 compatibility

How to add options and arguments to click

A CLI that just outputs “Hello World” which is not that useful. We will look at how we add options and arguments to our program. Let’s make some changes to the code we had before

import click

@click.command()
@click.option('--string', '-s', default='World',
                help='what is to be greeted')
def main(string):
    '''program that greets the user'''
    print("Hello {} from click".format(string))

if __name__ == "__main__":
    main()

We added a click.option() decorator. In it, a flag for the option the CLI accepts (in this case string ), a default value if nothing is passed, and help text giving an explanation of what the option does.

Both the --string and -s flag do the same thing. If we did not use the flag the script would still run because options are optional.

Now, let’s look at an example that uses an argument instead of an option

import click

@click.command()
@click.argument('name')
def main(name):
    '''program that greets the user'''
    print("Hello {}!".format(name))

if __name__ == "__main__":
    main()

Have a look what happens when we run the above script but don’t pass anything

We get an error message telling us that we are missing an argument. An error similar to one we had when we were using argparse

NOTE: options have the choice to be passed or not whereas arguments are mandatory

How to Improve the Execution of the click CLI

The current way we are accessing and executing our CLI will become very tedious for a user – having to type the file extension every time. What if there was a way such that all we had to do was input the name of the file followed by the parameters without extension?

Luckily there is a way – by making our python script executable. It’s a simple process but has to be done accurately.

Start by creating a directory and navigating into it. In your terminal or command prompt

mkdir [directory_name]

cd [directory_name]

In that directory make two python files: setup.py and your primary file; in my case main.py

Add this to the main.py file

import click

@click.command()
def cli():
    """A script to greet a user."""
    click.echo('Hello World!')

Then this to the setup.py file

from setuptools import setup

setup(
    name='main',
    version='0.1',
    py_modules=['main'],
    install_requires=[
        'Click',
    ],
    entry_points='''
        [console_scripts]
        main=main:cli
    ''',
)


The important part is the entry_points :

The first main defines what our program will be run as which is equal to main the module we specified and that will access the function cli in our main.py file

Now back in your terminal type this and press enter

pip install --editable .

We can now run our script with just the name.

You can add the options and or arguments the same way we did before.

A Practical Use for Click – Current Weather Program

We are going to build a program that gives us the current weather for an inputted city.

First, you have get an API key from OpenWeather, it’s free. You just have to sign up for an account.

The program consists of two parts:

  • One that makes a call to the API
  • Another – our click command – that passes the parameters to the API so that we get the weather information.

We will add to the last program as it already has some of the functionality we need. Modify it to look like this:

import requests
import click

# MAKING API CALL

SAMPLE_API_KEY = 'b1b15e88fa797225412429c1c50c122a1'

# function for current weather that takes location and api key
def current_weather(location, api_key=SAMPLE_API_KEY):
    url = 'https://api.openweathermap.org/data/2.5/weather'

    query_params = {
        'q': location,
        'appid': api_key,
    }

    # request to get api data
    response = requests.get(url, params=query_params)

    # check if response was 200 - successful
    if response.status_code == 200:
        # Return the json data for weather description
        return response.json()['weather'][0]['description']

        
# EXECUTING PROGRAM

@click.command()
# click argument for location which must be passed
@click.argument('location')
# click option for the api key
@click.option('--api-key', '-a', help='your API key for the OpenWeatherMap API',)
def cli(location, api_key):
    """
    A little weather tool that shows you the current weather in a LOCATION of
    your choice. Provide the city name and optionally a two-digit country code.
    """
    # invocation of current weather function
    weather = current_weather(location, api_key)
    click.echo(f"The weather in {location} right now: {weather}.")

if __name__ == "__main__":
    cli()

First, we write a function that will make a call to the Openweather API – this function will be called in our click module.

For the click part of our program, we will have an argument for location(city) and an option for the api key. In it we call our current_weather function.

We run our program like this:

[program name] [location] -a [api_key]
NOTE: You will have to use your own api key when you run this program

In this article we have introduced the click module. What it is and how we use it to build better CLIs. With it we have also added a better way to run our CLI. With that knowledge we have built a program that gives us the current weather.

Get Notified Automatically Of New Articles

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!

Simple Guide To Decorators in Python 3 – When and How to Use Them

Simple Guide To Decorators in Python 3 – When and How to Use Them

Advanced

Python is a very versatile programming language – there is a great deal you can accomplish with it. Useful as it may be, that versatility has the potential to bring about a bit of confusion with some of its implementations. Decorators are one such implementation of Python that if not explained properly, will probably do more harm than good.

This article aims to explain decorators so that you fully understand them and are confident enough to utilize them in your programs where appropriate.

NOTE: An understanding of functions in Python is required to grasp the following concepts.

What is a Decorator?

A decorator is a feature in python that allows you to add extra capabilities to your function, without having to change the inner workings of your function.

Robotic Exoskeletons Are Changing Lives in Surprising Ways
Sort of an analogy is how exoskeletons help to enhance humans (image source)

Some examples of uses of decorators include: determining the execution time of a function, determine which URL call should result in this function being called, adding logging messages when a function starts/stops. There are many uses for this.

More technically, a decorator is a function that takes another function as an argument, adds some functionality then returns another function. This happens without altering the original source code of the function that was passed.

A Practical Example of a Decorator

An example code to measure runtime

Rather than go into the technical explanation, let’s work on a practical example of a decorator. Suppose you had the following code and you wanted to determine which function is faster:

#decorator.py
import math, functools

def sqrt_by_loop( count ):
	total = 0;
	for i in range(count):
		total = total + math.sqrt(i)
	print(f"Running function sqrt_by_loop: {total}")
	return total

def sqrt_by_reduce( count ):
	total = functools.reduce( lambda running_sum, next_item: running_sum + math.sqrt(next_item), range(count) )
	print(f"Running function sqrt_by_reduce: {total}")
	return total

sqrt_by_loop(100)
sqrt_by_reduce(100)

The output for the above is as follows:

Method 1: Traditional method to measure runtime

To determine which function is faster, you can change the function code to track the start and end time, and then subtract the two to determine the runtime. So your code could be like this:

#decorator.py
import math, functools
import time

def sqrt_by_loop( count ):
	start = time.time()	# performance time trackingcode
	total = 0;
	for i in range(count):
		total = total + math.sqrt(i)
	print(f"Running function sqrt_by_loop: {total}")
	end = time.time()	# performance time trackingcode
	print(f"Runtime for sqrt_by_loop: {end-start}")

	return total

def sqrt_by_reduce( count ):	
	start = time.time()	# performance time trackingcode
	total = functools.reduce( lambda running_sum, next_item: running_sum + math.sqrt(next_item), range(count) )
	print(f"Running function sqrt_by_reduce: {total}")
	end = time.time()	# performance time trackingcode
	print(f"Runtime for sqrt_by_loop: {end-start}")
	return total

sqrt_by_loop(9145000)

sqrt_by_reduce(9145000) 


Output:

We ran this with a much larger input of 9,145,000 to see which method was fastest to determine the sum of the square root of all sequence of numbers. So interestingly, the reduce function was a little bit slower.

So this works, however, the problem is that you have to edit your original functions.

Method 2: Measuring with a method

Another way to do this would be to use a method in between to do the measuring. There are several ways to do this, so this is one variation:

#decorator.py
import math, functools
import time

def measure_runtime( func, param ):
	start = time.time()		# performance time trackingcode
	func( param )
	end = time.time()		# performance time trackingcode
	print(f"Runtime for {func}: {end-start}")

def sqrt_by_loop( count ):
	total = 0;
	for i in range(count):
		total = total + math.sqrt(i)
	print(f"Running function sqrt_by_loop: {total}")
	return total

def sqrt_by_reduce( count ):	
	total = functools.reduce( lambda running_sum, next_item: running_sum + math.sqrt(next_item), range(count) )
	print(f"Running function sqrt_by_reduce: {total}")
	return total

measure_runtime( sqrt_by_loop, 9145000) 
measure_runtime( sqrt_by_reduce, 9145000) 

Output:

So this is an improvement in that the original functions dont need to be changed, however the problem is that the calling functions have to be changed quite a bit. It’s also quite inflexible in that as the parameters differ, the measure_runtime would have to be different.

The challenge with both Method 1 and Method 2 is also that if you have lots of functions, then this is a lot of extra code you have to include.

This is where decorators can help

Method 3: Using decorators to measure runtime

The above problems can be all resolved with a decorator. We can enhance the original functions, by adding a timer function, without having to edit the original function, and without having to edit the calling code.

Agile - what sorcery is this?

Yes, it is possible and one of the great things about python. The code looks like this:

#decorator.py
import math, functools
import time

def measure_runtime(original_func):
	def wrapper_func(*args, **kwargs):
		start = time.time()		 
		return_value = original_func(*args, **kwargs)
		end = time.time()		 
		print(f"Runtime for {original_func}: {end-start}")
	return wrapper_func
	
@measure_runtime
def sqrt_by_loop( count ):
	total = 0;
	for i in range(count):
		total = total + math.sqrt(i)
	print(f"Running function sqrt_by_loop: {total}")
	return total

@measure_runtime
def sqrt_by_reduce( count ):	
	total = functools.reduce( lambda running_sum, next_item: running_sum + math.sqrt(next_item), range(count) )
	print(f"Running function sqrt_by_reduce: {total}")
	return total

sqrt_by_loop( 9145000) 
sqrt_by_reduce( 9145000) 


Output:

As you can see, the only change was to add this @measure_runtime single line above each function, and you achieve the same thing!

Hallelujah squirrel! | Happy squirrel, Cute animals, Funny animals

Method 3: Using decorators to measure runtime – with averages

The useful thing about decorators, is that you can also do some advanced things such as run functions multiple times to get the averages as well. So with the decorator, we can make the call to the core function number of times and return the average speed since it may vary:

#decorator.py
import math, functools
import time

def measure_runtime(number_of_times=1):
	def decorator_func(original_func):
		def wrapper_func(*args, **kwargs):
			execution_time = 0
			for execution in range(number_of_times):
				start = time.time() 
				return_value = original_func(*args, **kwargs)
				end = time.time() 
				print(f"Runtime for [{execution}] for {original_func}: {end-start}")
				execution_time = execution_time + (end-start)
			execution_time = execution_time / float(number_of_times)
			print(f"***Average runtime for {original_func}: {execution_time} ***\n\n")

		return wrapper_func
	return decorator_func
		
@measure_runtime(3)
def sqrt_by_loop( count ):
	total = 0;
	for i in range(count):
		total = total + math.sqrt(i)
	print(f"Running function sqrt_by_loop: {total}")
	return total

@measure_runtime(3)
def sqrt_by_reduce( count ):	
	total = functools.reduce( lambda running_sum, next_item: running_sum + math.sqrt(next_item), range(count) )
	print(f"Running function sqrt_by_reduce: {total}")
	return total

sqrt_by_loop( 9145000) 
sqrt_by_reduce( 9145000) 

See how the decorator @measure_runtime was called with a parameter of ‘3’ which was to make the call to that function 3 times to get the average runtime.

The Concept Behind Decorators

So the above should explain the power of decorators. So how does it actually work? Well it’s a combination of a few interesting features in python:

  • Functions are First-Class Objects: this means you can assign functions to variables, pass them as arguments, and return them as values from other functions.
  • Scope / Closure: this specifically refers to how a nested function that has access to variable(s) from its enclosing function.

It’s these two things that make it possible to have decorators. This is a diagram that shows what happens conceptually. The actual call to the original function is circumvented to call the decorator which encapsulates the call to the original call with some additional functionality. That’s where the magic lies

How Function References Work

The above example for the performance measurement showed you how to call and crate a decorator, but let’s spend some time going through that in more detail so you can understand what is happening and decorators are even possible.

Here’s a simple example to a function call:

message = 'Hello world'
def main_func():
  print(message)
   
main_func()   
Output:
Hello World

The output will obviously show: ‘Hello world’. The reason is that the ‘main_func’ can see the variable ‘message’.

That’s easy enough. Now consider this example:

message = 'Hello world'
def main_func():
	print(message)
 
temp  = main_func
print("Temp:" + str(temp))
temp()   
Output:
Temp:<function main_func at 0x7f3d429551f0>
Hello World

What you see above is that the “main_func” is just a variable like any other variable (in fact, it’s an object), and that it has an address for the actual function. You can also assign it to other variables and call it.

This is a more complex example of using addresses inside a function:

def outer_func():
	message = 'Hello world'

	def inner_func():
		print(message)

	print( "1. Inside outer_func - address of inner_func:" + str(main_func) )
	return inner_func

print( "2. In main code - address of outer_func:" + str(outer_func))
my_func = outer_func() 
print( "3. In main code - address of what outer_func returns:" + str(my_func) + "\n\n")
print( "4. Now calling my_func()")
my_func()
Output:
2. In main code - address of outer_func:<function outer_func at 0x7f6d692d71f0>
1. Inside outer_func - address of inner_func:<function outer_func.<locals>.inner_func at 0x7f6d691deee0>
3. In main code - address of what outer_func returns:<function outer_func.<locals>.inner_func at 0x7f6d691deee0>


4. Now calling my_func()
Hello world

What’s happening here is the following:

  • The call to outer_func() returns a reference to the inner function called “inner_func()”. This is stored in the variable “my_func”
  • Next, when my_func is called through my_func() it will execute the code under “inner_func()”

And here’s a final example, but with data passed as variables:

def outer_func(msg):
    def inner_func():
        print("Inner Function:" + msg)
    return inner_func

hi_func = outer_func('Hi')
bye_func = outer_func('Bye')

hi_func()
bye_func()
Output:
Inner Function:Hi
Inner Function:Bye

As you can see, the messages were also accessible by the inner functions.

The code we have just seen is the basis of how you start to understand decorators.

How to Write a Decorator

Now let us look at an example that illustrates how a decorator works.

# Python - Decorators

# decorator function taking function as argument
def decorator_func(original_func):
    def wrapper_func():
        print( f'A. wrapper executed this before function [{original_func.__name__}]' )
        # return passed function and execute
        return original_func()
    # return wrapper function waiting to be executed
    return wrapper_func

def display_func():
    print('B. [display_func] function ran')

# decorate display function with decorator function 
decorated_display = decorator_func(display_func)

decorated_display()
Output:
A. wrapper executed this before function [display_func]
B. [display_func] function ran

The best way to explain this code is starting at the end:

  • We create a variable decorated_display and assign it decorator_func() with the display function passed to it
  • We execute decorated_display and that:
    • runs the wrapper_func() which prints out a statement and returns the display function we passed
    • we return wrapper_func which is waiting to be executed
      • its execution happens when we call decorated_display

Normally, you won’t find decorators written as in the code above. Instead, you will something like this which is the shorthand version:

# Python - Decorators

# decorator function taking function as argument
def decorator_func(original_func):
    def wrapper_func():
        print( f'A. wrapper executed this before function [{original_func.__name__}]' )
        # return passed function and execute
        return original_func()
    # return wrapper function waiting to be executed
    return wrapper_func

@decorator_func
def display_func():
    print('B. [display_func] function ran')

display_func()

The output is the same because @decorator_func is the same as decorated_display = decorator_func(display) . But note, that in this version, you just call the original function name “display_func()” – the @decorator_func does the reshuffling automatically. The code becomes much easier to read and maintain!

Adding Arguments to Decorators

The example we have is good for illustration but does not have any use beyond that. If we wanted to pass parameters to our decorated function it would not work and we would run into errors. Let’s how we can correct this with a different example

NOTE: When you want to pass arguments but don’t know how many they are going to be, you can use *args and **kwargs for the parameters of that function.

def decorator_func(original_func):
    def wrapper_func(*args, **kwargs):
        print('wrappper executed this before {}'.format(original_func.__name__))
        return original_func(*args, **kwargs)
    return wrapper_func

@decorator_func
def display_info(name, age):
    print('display_info ran with arguments ({}, {})'.format(name, age))


display_info('James', 23)

Since we want to pass parameters to the function we decorate we would add *args and **kwargs both to our wrapper function and the function returned within it.

We have managed to decorate display_info() and pass parameters to it

Decorators with parameters

In the above example, the decorator was used to change the behaviour of the original function. However, you can also pass parameters so that your decorator can be even more powerful.

Suppose in the previous example, you also wanted to add a custom title text each time the decorated function is called. You can do the following:

def decorator_func(header_message):
	def main_decorator(original_func):
		def wrapper_func(*args, **kwargs):
			print(f"##### {header_message} #####")
			print('A. wrapper executed this before {}'.format(original_func.__name__))
			return original_func(*args, **kwargs)
		return wrapper_func

	return main_decorator

@decorator_func(header_message='My Title')
def display_info(name, age):
	print('B. [display_func] function ran with arguments ({}, {})'.format(name, age))

display_info('James', 23)
Output:
##### My Title #####
A. wrapper executed this before display_info
B. [display_func] function ran with arguments (James, 23)

It does look rather complex due to the multiple nesting, but it’s there to capture the arguments.

  • The decorator_func is called with the argument, and it returns the main_dectorator decorate.
  • The main_decorator() when called, in fact returns a call to wrapper_func which encapsulates a call to the original function display_info()!

You can use the above template to guide your decorators.

Conclusion

Decorators are a powerful tool to help simplify your code and also make it very extendible. There are many use cases you can use this for – some examples include:

  • Measuring runtime of functions to assess performance (as above)
  • Automatic logging when a function is called and storing arguments to support debugging
  • Linking a website URL to execute a given function for that url (this is the famous @route decorator that you see in Django and Flask apps)
  • Processing command line arguments with click
  • Creating a plugin architecture for your code (see article here on Plugin articles with decorators)
  • And there are many other use cases..

They are a bit intimidating at first, but once you set them up, then you put your decorator code in different package files to help make your code cleaner.

How have you used decorators in the past? Share you comments below!

Get Notified Automatically Of New Articles

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!

How to Parse Arguments from the Command Line using argparse in Python 3

How to Parse Arguments from the Command Line using argparse in Python 3

Intermediate

Having command line parameters is an essential mechanism to manage the behaviour of your application but it can get complex to validate and extract the arguments. In our previous article on arguments (How to use argv parameters in Python), we went over how to use command line parameters with sys.argv. This is a useful module, but we will come across cases where we want to do more than just that. If you code enough in Python, you are most likely going to want to let users provide values for variables at runtime.

This is where argparse comes in. In this article, we are going to look into argparse – what it is. why we use it and how we parse arguments using it.

NOTE: This article assumes you have some Python knowledge and understand command line arguments.

argparse is a python module whose primary purpose is to create command line interfaces. This module was released as a replacement for the getopt and optparse modules because they lacked some important features.

The Python argparse module:

  • Allows the use of positional arguments
  • Allows the customization of the prefix chars
  • Supports variable numbers of parameters for a single option

What is a Command Line Interface (CLI)?

Before we start making use of argparse to build our own command line interfaces, let’s take a brief look at what a command line interface is. If you have used the terminal or command prompt you have probably already used a command line interface.

Below we open a folder and run the ls command to get the list of files in the directory we are in

There are some files in this directory but not a lot of information about them. Let’s add some arguments to ls. We will run the same command and parse -l to it.

Adding that argument gives us more information about the files. This works by accessing the options present in the ls command line interface. Now that we have a general idea of CLIs we can start making our own.

If your goal is to provide a user-friendly approach to your program, it is usually a good idea to implement a CLI

How to Parse Arguments with argparse

To understand how argparse works and how we go about making our own command line interfaces, we will begin with a simple example illustrated below:

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("-n", "--name", required=True)
args = parser.parse_args()
print(f'Hi {args.name} , Welcome ')

Confusing? Yes, I know. But let’s break this down line by line:

  • We initially import the argparse module
  • We then initialize the ArgumentParser object as parser
  • We add the only argument –name, here we must specify both the shorthand and longhand versions. Either can be used when the program is run
  • We stipulate that this argument is required
  • We execute the parse_args() method
  • We output the parsed argument

We get the output of our print statement plus the parsed argument. But what would happen if we ran the program but didn’t give any arguments? We will try that next:

We got an error message, explaining the usage of the program and the arguments that are required. But we never added that to our code, did we? Well, not explicitly. When we added arguments for the program, argparse generates the error message if those arguments ate not given.

Looking at the output once again, you will notice there is another argument, one that we did not give -h. This is an optional argument that the program accepts. It serves as a sort of help command to describe what is required by the program.

Let’s parse that argument to see what output we get:

As we see, parsing the -h argument gives a bit more detailed explanation of the program. Armed with a basic understanding of argparse we can look at another example.

We are going to code our own program that simulates the same operation as ls:

# Import the argparse library
import argparse

import os
import sys

my_parser = argparse.ArgumentParser(description='List all the contents of a folder')

my_parser.add_argument('Path',
                       metavar='path',
                       type=str,
                       help='the path to list')

args = my_parser.parse_args()

input_path = args.Path

if not os.path.isdir(input_path):
    print('The path specified does not exist')
    sys.exit()

print('\n'.join(os.listdir(input_path)))

We need to import a number of modules first namely os, sys, and argparse. After that we:

  • create the parser and assign it the name my_parser
  • add the necessary arguments – in this case, path. We also specify the type of the argument as string which tells argparse to check the type automatically
  • execute the parse_args() function
  • create a variable named input_path and assign it args.Path – calling path on the parser
  • run a check if the parsed directory does not exist and display an error message if it doesn’t
  • we output the inputted directory

We are running the program on the same directory as before:

We get an output with the same files.

How to Customize Your CLI with argparse

The argparse module also allows us to customize our CLI. We have already done something of the sort in the last example. When we created the parser object: my_parser = argparse.ArgumentParser(description='List all the contents of a folder')

In this case, description, which gives an explanation of the program which is outputted when we run the -h/--help argument. In addition to description, there are two other ways we can use to customize our parser:

  • prog: which gives our program a custom name replacing sys.argv[0]
  • epilog: text which displayed after help text

Let’s add these to our first example:

import argparse

parser = argparse.ArgumentParser(prog='greeting', 
                                description='Greets the user', 
                                epilog='Thank you for using the program. Have a nice day')

parser.add_argument("-n", "--name", required=True)
args = parser.parse_args()
print(f'Hi {args.name} , Welcome ')

We ran our program, parsed the –help argument and we the name we gave, description and epilog are all output.

Final example with argparse

Let’s say you wanted to make a script that would write text to a file, that would look something like this:

import argparse

parser = argparse.ArgumentParser(description='Write to file')


parser.add_argument('file', help='file to store text in')
parser.add_argument('text', help='text to be written to file')

args = parser.parse_args()


print('Writing to: {}'.format(args.file))
print('\t {}'.format(args.text))

with open(args.file, 'w') as my_file:
    my_file.write(args.text)

print('File has been written')

After importing argparse and creating our parser we:

  • add two arguments – one for the file, the other for the data to be written to it
  • execute parse_args()
  • invoke the with() funnction to open the file – parsed as the first argument – in write mode
  • write the second argument to the file
  • the print statements are so we see what is happening – they can be omitted

NOTE: If you surround arguments in quotation marks they are treated as though they are one argument.

IMPORTANT: When you add arguments to your parser they become positional arguments and must be parsed in that exact same order

Get Notified Automatically Of New Articles

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!