Beginner

Importing modules or packages (in other languages this would be referred to as libraries) is a fundamental aspect of the language which makes it so useful. As of this writing, the most popular python package library, pypi.org, has over 300k packages to import. This isn’t just important for importing of external packages. It also becomes a must when your own project becomes quite large. You need to make sure you can split your code into manageable logical chunks which can talk to each other. This is what this article is all about.

What’s the difference between a python package vs module

First, some terminology. A module, is a single python file (still with a .py extension) that contains some code which you can import. While a package, is a collection of files. In your project, a package is all the files in a given directory and where the directory also contains the file __init__.py to signal that this is a package.

What happens when you import a python module

There is nothing special in fact you need to do to make a module – all python files are by default a module and can be imported. When a file is imported, all the code does get processed – e.g. if there’s any code to be executed it will run.

See following example. Suppose we have the following relationship:

Code as follows:

#module1.py
print("module1: I'm in module 1 root section")

def output_hw():
	print("module1: Hello world - output_hw 1")

#module2.py
import module1
print("module2: I'm in root section of module 2")

def output_hw():
	print("module2: Hello world - output_hw 2")

#main_file.py
print("main_file: starting code")
import module1
import module2

print("main_file: I'm in the root section ")

if __name__ == '__main__':
	print("main_file: ******* starting __main__ section")
	module1.output_hw()
	module2.output_hw()
	print("main_file: Main file done!")

Output:

So what’s happening here:

1. The main_file.py gets executed first and then imports module1 then module2
2. As part of importing module1, it executes all the code including the print statements in the root part of the code. Similarly for module2
3. Then the code returns to the main_file where it calls the functions under module1 and module2.
4. Please note, that both module1 and module2 have the same function name of output_hw(). This is perfectly fine as the scope of the function is in different modules.

One additional item to note, is that the module2 also imports module1. However, the print statement in the root section print("module1: I'm in module 1 root section") did not get executed the second time. Why? Python only imports a given module once.

Now let’s make a slight change – let’s remove the references to module1 in the main_file, and in module2, import module1!

The updated code looks like this:

#module1.py
print("module1: I'm in module 1 root section")

def output_hw():
	print("module1: Hello world - output_hw 1")

#module2.py
import module1
print("module2: I'm in root section of module 2")

def output_hw():
	print("module2: Hello world - output_hw 2")

#main_file.py
print("main_file: starting code")
# import module1
import module2


print("main_file: I'm in the root section ")

if __name__ == '__main__':
	print("main_file: ******* starting __main__ section")
	module2.output_hw()
	# module2.output_hw()
	print("main_file: Main file done!")

Output:

Now notice that module1 gets imported and executed from module2. Notice that the first line is “module1: I’m in module 1 root section” since the very first line of module2 is to import module1!

How do you make a package in your python project

To create a package it’s fairly straightforward. You simply need to move all your files into a directory and then create a file called __init__.py.

This means your directory structure looks like this:

/main_file.py
└── package1/
    ├── __init__.py
    ├── module1.py
    └── module2.py

The above example, would now look like the following:

#__init__py
import package1.module1
import package1.module2

#module1.py
print("module1: I'm in module 1 root section")

def output_hw():
	print("module1: Hello world - output_hw 1")

#module2.py
import package1.module1
print("module2: I'm in root section of module 2")

def output_hw():
	print("module2: Hello world - output_hw 2")

#main_file.py
print("main_file: starting code") 
import package

print("main_file: I'm in the root section ")

if __name__ == '__main__':
	print("main_file: ******* starting __main__ section")
	package1.module1.output_hw()
	package1.module2.output_hw() 
	print("main_file: Main file done!")

So in the __init__.py file, it imports module1 & module2. The reason this is important is because so that when in main_file the package1 is imported, then it will have immediate access to module1 and module2. This is why the package1.module1 and package1.module2 works.

You cannot make the inclusion of modules automatic, and generally you shouldn’t as you may have name clashes which you can avoid if you do this manually.

Can you avoid typing the prefix of “package1” each time? Yes in fact if you use the “from”. See next section.

Only Import a part of a module

You can also import just either a class or a function of a given module if you prefer in order to limit what is accessible in your local code. However, it does still execute your whole module though. It is more a means to make your code much more readable. See the following example:

#module1.py
print("module1: I'm in module 1 root section")

def output_hw():
	print("module1: Hello world - output_hw 1")

#main_file.py
print("main_file: starting code") 
from module1 import output_hw

print("main_file: I'm in the root section ")

if __name__ == '__main__':
	print("main_file: ******* starting __main__ section")
	output_hw() 
	print("main_file: Main file done!")

Output

As can be seen in the above output, although just the output_hw() function is being imported, the statement “module1: Im in module1 root section” was still executed.

Note also, that you do not need to mention the module prefix in the code, you can just refer to the function as is.

So back to above, for the packages, instead of the following:

import package1.module1

you can instead use the “from” keyword but force to check local directory:

from .module1 import *

There’s a few things going on here. The '.' in front of module1 is referring to the current directory. If you wanted to check the parent directory then you can use two '.'s so the line looks like this: from ..module1 import *. The second item is that everything is being imported with the import * section.

Importing a module and applying an alias

In case you wanted to make your code easier to read, or you wanted to avoid any name clashes (see at the start of the article how module1 and module2 both had the same function name of output_hw() ), you can use the “as” keyword at the import statement to give an alternative name.

You can do the following:

#main_file.py
print("main_file: starting code") 
from module1 import output_hw as module1__output_hw

print("main_file: I'm in the root section ")

if __name__ == '__main__':
	print("main_file: ******* starting __main__ section")
	module1__output_hw() 
	print("main_file: Main file done!")

This can also be done with the module or package name as well, i.e.

import module1 as mod1

Importing modules outside your project folder

Modules can by default be imported from the sub-directories up to the main script file. So the following works:

/main_file.py
└── package1/
│   ├── __init__.py
│   ├── module1.py
│   └── module2.py
└── package2/
    ├── __init__.py
    └── pkg2_mod_a.py

Then in module1, you can import from pkg2_mod_2 with the following:

#module1.py
from package2.pkg2_mod_a import get_main_list

def output_hw():
	print("module1: List from pkg2 module A:" + str( get_main_list()) )

Just need to remember in package2/__init__.py that you have to import pkg2_mod_a.py

However, what if the code was outside your main running script? Suppose if you had the following directory structure:

/
└── server_key.py
/r1/
  └── main_file.py
  └── package1/
      ├── __init__.py
      └──  module1.py 

From any file in the /r1/ project, if you tried to import a file from server_key.py , you will get the error:

ValueError: attempted relative import beyond top-level package

To resolve this, you can in fact tell python where to look. Python keeps track of all the directories to search for modules under sys.path folder. Hence, the solution is to add an entry for the parent directory. Namely:

import sys
sys.path.append("..")

So the full code looks like the following:

#main_file.py
import sys
sys.path.append("..")
print("main_file: starting code")  
import package1

print("main_file: I'm in the root section ")

if __name__ == '__main__':
	print("main_file: ******* starting __main__ section")
	package1.module1.output_hw() 
	print("main_file: Main file done!")

#module1.py
from package2.pkg2_mod_a import get_main_list
from server_key import get_server_master_key

def output_hw():
	print("module1: List from pkg2 module A:" + str( get_main_list()) )
	print("module1: server key :" + get_server_master_key() )

#server_key.py
def get_server_master_key():
	return "AA33FF1255";

Output – The output is as follows:

How to import modules dynamically

All of the above is when you know exactly what the module name to import. However, what if you don’t know the module name until runtime?

This is where you can use the __import__ and the getattr functions to achieve this.

Firstly the getattr(). This function is used to in fact load an object dynamically where you can specify the object name in a string, or provide a default.

Secondly, the __import__() can be used to provide a module name as a string.

When you combine the two together, you first load the module with __import__, and then use getattr to load the actual function you want to call or class you want to load from the import.

See the following example:

/r1/
  └── main_file.py
  └── package1/
      ├── __init__.py
      └──  module1.py 

With the following code:

#module1.py

def output_hw():
	print("module1: take me to a funky town")
	

#main_file.py
if __name__ == '__main__':
	print("main_file: ******* starting __main__ section")
	
	module = __import__( 'package1.module1')
	func = getattr( module, 'output_hw', None)
	if func:
		func()
	print("main_file: Main file done!")

In the above code, we first load the module called “package1.module1” which only loads the module. Then the getattr is called on the module and then the function is passed as a string. You can also pass in a class name if you wish.

Conclusion

There are many ways to import files and to organize your projects into smaller chunks. The most difficult piece is to decide what parts of your code go where..

Get notified automatically of new articles

We are always here to help provide useful articles with usable ode snippets. Sign up to our newsletter and receive articles in your inbox automatically so you won’t miss out on the next useful tips.