Advanced

Once your core application is complete, a plugin architecture can help you to extend the functionality very easily. With a plugin architecture, you can simply write the core application, and then extend the functionality in the future much more easily. Without a plugin architecture, it can be quite difficult to do this since you will be afraid that you will break the original functionality.

So why don’t do this all the time? Well it does take more planning effort in the beginning in order to reap the rewards in the future, and most of us (myself included) are often too impatient to do that. However, there are some methods that you can take in order to embed a plugin desirable to extend the functionality. Last time we looked at using importlib (see our previous article “A Plugin Architecture using importlib“), and this time we have an even simpler library called pyplugs.

When to use plugin architecture

So when should you use a plugin architecture? Here are several scenarios – they are all around separating the code from the core to the variations:

• Separate Functionality: When you can split the problem you’re trying to solve/application from core functionality (the main “engine”) to the variations: e.g. ranking cheapest flights where data is from different websites. The core application/engine is the ranking logic. The data extraction from different websites would each be a plugin – website 1 = plugin 1, website 2 = plugin2. When you want to add a new website, you just need to add a new plugin
• Distribute Development Effort: When you want to work in a team to easily separate the focus from core functionality to variations: e.g. suppose you have an application to do image recognition. Team 1 (e.g. data science team) can work on the core engine of doing the image recognition, while you can have Team 2-4 work on creating different plugins for different image formats (e.g. Team 2: read in JPG files, Team 3: read in PNG files, etc)
• Launch sooner and add functionality in future: When you want to launch an application as quickly as possible. e.g. Suppose you want to create an application to return the number of working days from different countries. To begin with, you can just start by launching this for United States and Australia. Then, you can add more countries in the future. Since you designed the plugin architecture from the start, it’ll be safer to add more countries.

There are many more, but the disadvantage is that you have to plan for it upfront. Invest now in a plugin architecture, and then reap the benefits in the future.

Invest now in a plugin architecture, and then reap the benefits in the future

Let’s explore this third example of a public holiday counter application and show how the pyplugs library can help.

Example Problem: Extracting Public Holidays

The application we’d like to create is a command line application that can be used to pass in a location (country and/or state), and then return the list of public holidays in 2020:

The pseudo-code will be as follows:

1. Get location  
2. If data for location not available, then error
3. Get the list of all holidays from the location
4. Return the list of working days  

As you probably guessed, it’s step 3 that can be converted into a plugin. However, let’s start without a plugin architecture and do this the normal way.

First let’s see where we can get the data from – for UK data you can get this from publicholidays.co.uk:

And then for Singapore data, you can get it from jalanow.com:

In both cases, the data is in a HTML Table view where the data is in a <td> tag. We will need to use regular expressions to extract the data.

Here’s the code for non-plugin approach:

#pubholiday.py
import argparse
import requests, re 

G_COUNTRIES = ['UK', 'SG']

def get_working_days(args):
	if args.countrycode =='UK':
		r = requests.get( 'https://publicholidays.co.uk/2020-dates/')
		m = re.findall('<tr class.+?><td>(.+?)<\/td>', r.text)
		return list(set(m))
	elif args.countrycode =='SG':
		r = requests.get('https://www.jalanow.com/singapore-holidays-2021.htm')
		m = re.findall('<td class\=\"crDate\">(.+?)<\/td>', r.text)
		return list(set(m)) 


def setup_args():
	parser = argparse.ArgumentParser(description='Get list of public holidays in a given year')

	parser.add_argument('-c', '--countrycode', required=True, type=str, choices=G_COUNTRIES, help='Country code') 
	return parser

if __name__ == '__main__':
	parser =  setup_args()
	args = parser.parse_args()
	print( get_working_days(args) )

Running the above with no arguments gives the following – the argparse is a useful library to create arguments very easily – see our other article How to use argparse to manage arguments.

Now, when we run the application with either UK or SG, we get the following data:

The way the code works is all from the function get_working_days:

def get_working_days(args):
	if args.countrycode =='UK':
		r = requests.get( 'https://publicholidays.co.uk/2020-dates/')
		m = re.findall('<tr class.+?><td>(.+?)<\/td>', r.text)
		return list(set(m))
	elif args.countrycode =='SG':
		r = requests.get('https://www.jalanow.com/singapore-holidays-2021.htm')
		m = re.findall('<td class\=\"crDate\">(.+?)<\/td>', r.text)
		return list(set(m)) 

The code for UK, for examples works the following way:

1. Get the data using the requests to the website.  All the data will be in a r.text
2. Next, run a regular expression to extract the date data from the <TD> tag
3. Finally, remove duplicates with the list(set(m)) code

The disadvantage with this code is that if we add more countries, the function get_working_days() will become longer and longer with complex IF statements. The other challenge is testing it, either manually or with pytest will become quite painful. We can always have it call a dynamic function, but then we end up having difficult to read code.

What we need is a dynamic way to call a function for each country so that it can be easily maintainable and extendible… this is where a plugin architecture will help.

Extracting Public Holidays with a plugin architecture using pyplugs

What we will do now is to separate the main core logic from the plugins. So the file structure will be as follows:

|--- pubholidays.py
|___ plugins\
|___________ __init__.py
|___________ reader_UK.py
|___________ reader_SG.py

So there will be the main functionality still in pubholidays.py, however all the country readers will all be in the plugins package (and subdirectory).

But first, let’s install the pyplugs library

Installing pyplugs

PyPlugs is available at PyPI. You can install it using pip:

 python -m pip install pyplugs  

Or, using pip directly:

 pip install pyplugs 

Pyplugs is composed of three levels:

• Plug-in packages: Directories containing files with plug-ins
• Plug-ins: Modules containing registered functions or classes
• Plug-in functions: Several registered functions in the same file

Core logic in plugin architecture

The core logic will be simplified to the following:

#pubholiday_pi.py
import argparse
import requests, re 
import plugins

G_COUNTRIES = ['UK', 'SG']

def get_working_days(args): 
	return plugins.read( 'reader_' + args.countrycode)

def setup_args():
	parser = argparse.ArgumentParser(description='Get list of public holidays in a given year')
	parser.add_argument('-c', '--countrycode', required=True, type=str, choices=G_COUNTRIES, help='Country code') 
	return parser

if __name__ == '__main__':
	parser =  setup_args()
	args = parser.parse_args()
	print( get_working_days(args) )

Now the get_working_days() function has been significant simplified. It calls the “read” function from the plugins/__init__.py package file. The ‘reader_’ + args.countrycode refers to the function and the module name.

Plugin logic

The plugsin/__init__.py is setup as follows:

# plugins/__init__.py
# Import the pyplugs libs
import pyplugs

# All function names are going to be stored under names
names = pyplugs.names_factory(__package__)

# When read function is called, it will call a function received as parameter
read = pyplugs.call_factory(__package__)  

The “read” is the same “read” that is referenced by get_working_days() function from the main pubholiday_pi.py files.

The plugin files/functions are each to be stored in files called “reader_<country code>.py”. The following is the UK file:

#plugins/reader_UK.py
import re, requests
import pyplugs

@pyplugs.register
def reader_UK():
	r = requests.get('https://www.jalanow.com/singapore-holidays-2021.htm')
	m = re.findall('<td class\=\"crDate\">(.+?)<\/td>', r.text)
	return list(set(m)) 

And then finally the SG file:

#plugins/reader_SG.py
import re, requests
import pyplugs

@pyplugs.register
def reader_SG():
	r = requests.get('https://www.jalanow.com/singapore-holidays-2021.htm')
	m = re.findall('<td class\=\"crDate\">(.+?)<\/td>', r.text)
	return list(set(m)) 

In Conclusion

So there is no change when you run the application – you still get the same output:

However, you have a much more maintainable application.

So we started with a monolithic file, and now we extended this to a plugin architecture where the variations are all stored in the “plugins/” folder. In order to add more country public holidays where the data may come from different websites, all that needs to be done is to: (1) add the country code into variable G_COUNTRIES to ensure the command line argument validation works, and (2) add the new file called reader_<country code>.py in the plugins directory with a function name also called reader_<country code>(). That’s it, everything else will work.

You can also see how we used importlib to achieve a similar outcome as well: A plugin architecture using importlib.

Get Notified Automatically Of New Articles