Get Air Quality Data Globally in Python 3

Get Air Quality Data Globally in Python 3

Beginner

There are some interesting data on our environment such as air quality factors that can be available from public datasets. This can be useful to determine if there are health risks in your local weather environments and can be programatically tracked by using the right approach in Python.

What Comprises Air Quality?

There is no one thing called ‘air quality’ but it is a combination of different factors that are combined together to form an index. There is also no universal air quality index, but there are different standards across geographies as each locale has different factors they want to include. It is a typically a ranking system which makes it easier to grasp the amount of pollutants or quality in the air to indicate to people how much risk they could be at in a given location.

Air Quality Index

A scale used to indicate how toxic the air is, along with the risks associated with each ranking, is an air quality index. For the appropriate amounts of major air contaminants, an AQI is determined using proven criteria based on medical science.

There are several values of air quality indexes:

  • Informing the public in an understandable way about air quality so that they can take steps to protect their health
  • To support countries in designing and reviewing strategies for improved air quality
  • Ability to consolidate information across various factors to understand a single quantitative representation of air quality
  • Ability to compare different locations and/or trends in air quality over time

Air Quality Data Sources

The World Air Quality Index website (waqi.info) is a modestly funded platform which helps to aggregate air quality data from sensors across the world. The site was created by researchers in Beijing that wanted to make it easier to see a world view of air quality data. They have so far managed to get about 15,000 sensors across 130 countries around the world that provide air quality data in real-time.

The data is updated on an hourly basis where, for example, a 10AM reading means the measurement was performed between 9am-10am. Each city is searchable and data can be accessed through an API from an API key.

The platform is useful in that you can get a good coverage of the world’s air quality data in the one place.

Getting Air Quality From API

The waqi.info website offers an API service through an API key where the data is provided in JSON format. You can call the API a maximum of 1000 times a second.

Data format will be as follows:

Ok, so first step is to get the API key. You can get the token from the following page: https://aqicn.org/api/ where you can get a data-platform token. You’ll need to enter your email address and name to get the API token key.

Once you provide your email address and click on the confirmation link on your confirmation email, you will get the token as shown here:

You will need the API key in order to call the API shortly. As you can see, if you call the url with the API and the city, you can then get the json data. The API can be called with:

https://api.waqi.info/feed/<city>/?token=<API Token key>

In the above example, when you go to the url with the city ‘beijing’ and your token, you will see the following:

Air Quality API Format

The API provides a wealth of information – here are some key fields:

FieldDescriptionExample
data/aqiAir quality index from 0 – +300. The quality is as follows:
0-50 = Good
51-100 = Moderate
101-150 = Unhealthy for moderate groups
151-200 = Unhealthy
201-300 = Very Unhealthy
+300 = Hazardous
71
data/time/sTime in the local timezone where the station is located“2021-02-20 19:00:00”
data/time/tzTimezone of the field data/time/s“+08:00”
data/city/nameCity name“Chi_sp, Illinois”
data/city/geoLongitude and latitude in an array“41.913600”,”-87.723900″
Key fields from the API called – you can find the full documentation in the API docs

If you are struggling to get a given city name, you can also provide the ID of the station. You can find the ID of a given station by searching for the “idx” variable in the source code for a given city. For example, if you wanted to find the ID for Chicago. You can do the following:

Step1: Go to the city search page at https://aiqcn.org/city/all

Step 2: Click on the city page – in this case chicago. Next go to view source

Step 3: Search for the keyword “idx”. You can see here the id is 7595

Step 4 : Call the API with the id.

https://api.waqi.info/feed/@<ID>/?token=<API Token key>

Calling the Air Quality API from Python

import requests
city = 'kanpur'
url = 'http://api.waqi.info/feed/' + city + '/?token='
api_key = '4b64972ed75f8008b440319494e7b01f7c5c248c'

main_url = url + api_key
r = requests.get(main_url)
data = r.json()['data']
aqi = data['aqi']
iaqi = data['iaqi']
dew = iaqi.get('dew','Nil')
no2 = iaqi.get('no2','Nil')
o3 = iaqi.get('o3','Nil')
so2 = iaqi.get('so2','Nil')
pm10 = iaqi.get('pm10','Nil')
pm25 = iaqi.get('pm25','Nil')
pollen = iaqi.get('pol','Nil')

print(f'{city} AQI :',aqi,'\n')
print('Individual Air quality')
print('Dew :',dew['v'])
print('no2 :',no2['v'])
print('Ozone :',o3['v'])
print('sulphur :',so2['v'])
print('pm10 :',so2['v'])
print('pm25 :',pm25['v'])

Here we are going to use Python3 for pulling the data from a free source weather forecast website using tokens and APIs.

Output:

In the above code we simply requested the given website to share its weather forecast data with us and we put forward the data in an organised way in our console using python3.

Get Notified Automatically Of New Articles

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!

How To Create Getter Setter Class Properties in Python 3

How To Create Getter Setter Class Properties in Python 3

Beginner

We know everyone hates setting property fields in Python classes, here’s how you can create getter setter properties in python classes to do it for you. In this article, you’ll see how to create these above properties.

Basically, maintaining data encapsulation is the key objective of using getters and setters in object-oriented programs. However, getters and setters are not the same in Python as those in other programming languages that are object-oriented. Private variables in python are not hidden fields, but then the idea of encapsulation is simulated in ways.

Python encapsulation is the concept of packaging together variables and methods into a single object. A class is an illustration of programming that encapsulates all the variables and methods specified inside it.

Consider the volume control of your radio. You don’t update the volume setting directly on the circuit board in order to change the volume. You instead use the volume control to adjust the settings, and the internal values are updated on the circuit board chip. Here the setter and getter is the volume control that helps you set a new value, and you get to see the current volume on the dial on the display.

If fields are not hidden, why use Python’s Getters and Setters ?

Even though you don’t get the strict control and privacy from attributes compared to other languages (e.g. Java has the concept of ‘private’ variables) it is still very useful to use getters and setters. Some key reasons include:

  • To add validation logic around getting and setting a value
  • In order to prevent direct access to a class field where internal function requires some control
  • To provide some flexibility where the internal function can change without affecting how the class is accessed

How to Create Getter and Setter Properties in Python

There are several methods to create getter and setter functions:

  • Method 1: By using a normal function to achieve the actions of getters and setters.
  • Method 2: By using the property() function to implement the behaviour of getters and setters.
  • Method 3: By using @property decorators to accomplish the actions of getters and setters (see our other article on how decorators work).

Method 1: By using a function

If we define basic get() and set() methods to achieve the property of getters & setters, it is easy to follow but will require some discipline to be consistent. For example:

#A python programme that displays a use of the  methods get() and set()
#in normal function
class AgeSet:
	def __init__(self, age = 0):
		self._age = age

	# getter method
	def get_age(self):
		return self._age

	# setter method
	def set_age(self, x):
		self._age = x

pkj = AgeSet()

# setting the age using setter
pkj.set_age(int(input("set the age using setter: ")))

# retrieving age using getter
print(pkj.get_age())

print(pkj._age)

Output:

By using a normal function to achieve the actions of getters and setters.

According to the above code functions get_age() and set_age() functions as standard functions. You can also see that the private variable in the class was self._age. However, you can still access that without restriction. The underscore _ prefix is used as a convention to denote private variables but there is no restrictions, it is just a common practice.

Method 2: By using the property() function

The property() is a built-in function in Python that creates a property object and returns it. There are three methods for a property object, getter(), setter(), and delete (). The property() function has four arguments in Python: property(fget, fset, fdel, doc).

  • fget is an attribute value retrieval function.
  • fset is a function for setting a value for an attribute.
  • fdel is an attribute value removal function.
  • doc generates an attribute docstring.

The object property has three methods, getter(), setter(), and delete() to  explicitly state fget, fset, and fdel.

# Python program displaying a use of the property() function 

class AgeSet:
	def __init__(self):
		self._age = 0

	# function to get value of _age
	def get_age(self):
		print("getter method called")
		return self._age

	# function to set value of _age
	def set_age(self, a):
		print("setter method called")
		self._age = a

	# function to delete _age attribute
	def del_age(self):
		del self._age

	age = property(get_age, set_age, del_age)

pkj = AgeSet()

pkj.age = int(input("set the age using setter: "))

print(pkj.age)

Output:

By using the property() function to implement the behaviour of getters and setters.

There is only one print expression in the above code at line 25, but the output consists of three lines because of the setter method set_age() called in the input line, and getter method get_age() called in the print line. Therefore, age is a property object that helps to keep private variable access secure. As can be seen, within the class the property() function does the magic of assigning the getter and setter function.

Method 3: By using @property decorators

This is by far the easiest and the way that I recommend. In the previous step, we used the property() function to accomplish the action of getters and setters. However, as stated previously in this article, getters and setters are often used to validate the validity of attributes obtained and set. There is one more way to use the decorator to enforce the property function. One of the built-in decorators is python @property.

# Python program displaying the use of @property 

class AgeSet:
	def __init__(self):
		self._age = 0

	# using property decorator a getter function
	@property
	def age(self):
		print("getter method called")
		return self._age

	# a setter function
	@age.setter
	def age(self, a):
		if(a < 18):
			raise ValueError("Sorry your age is below eligibility criteria")
		print("setter method called")
		self._age = a

pkj = AgeSet()

pkj.age = int(input("set the age using setter: "))

print(pkj.age)

Output:

By using @property decorators to accomplish the actions of getters and setters.

In the above, it is clear how to use @property decorator in our python programming language to build getters & setters. You simply use the @property decorator to define the getter, and then use the @<variable>.setter to define the setter immediately above the relevant function. That’s it.

Conclusion

This is terrific, isn’t it? You can start with the easiest implementation imaginable, and without modifying the user interface, you are free to move to a property version later! So properties are not just a substitution for setters and getters!

Something else you may already have noticed: properties are syntactically similar to ordinary attributes for users of a class.

It is a common misconception that by using getters and setters, a proper Python class can encapsulate private attributes. As soon as a new attribute is added by one of these programmers, he or she will make it a private variable and create a getter and a setter “automatically” for these attributes. These programmers can even use an editor or an IDE, which automatically generates all private attributes with getters and setters. These tools also warn the programmer if a public attribute is used by her or him! When they read the following, Java programmers can wrinkle their brows, screw up their noses, or even yell in horror: The Pythonic way of adding attributes is to make them public.

Thanks for reading the full article, check out our other already uploaded articles if that comes to your intuitive attention.

Get Notified Automatically Of New Articles

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!

Getting Historical Stock Data Using Python 3

Getting Historical Stock Data Using Python 3

Intermediate

So in this tutorial, we are going learn about how to get Historical stock data using pandas_datareader from yahoo finance along with basics of pandas dataframe and moving average. We will also learn how to store the obtained data in a file, cache, cleaning the data by adding missing rows and at last how to visualise it by making a graph.

Installing Requirements

We need to install a few libraries before getting started.
pip install pandas_datareader
pip install pandas
pip install matplotlib
pip install requests_cache

Well if you don’t know about pandas or dataframes. Need not to worry. We will discuss them too.

Terminology

I will be using few technical terms in this tutorial so if you don’t understand you can refer here.

  • Dataframe: Just imagine data in an excel sheet with columns and rows. It’s somewhat similar to that. It stores data in the form of an object and has many predefined functions to perform various operations.
  • Ticker: Sometimes known as a “stock” or “sock code”, it is a unique identifier for each companies name. Every company has its own unique ticker. Like AAPL for Apple, FB for Facebook, AMZN for Amazon, etc.

Extracting data from yahoo with Pandas

We will be using Yahoo Finance for obtaining stock data. Firstly we will import pandas_datareader.data then send a request to yahoo finance with ticker , start and end date. The “pandas_datareader” module is a useful module that abstracts extracting web data very easily. You don’t have to bother with “requests” or “urllib” and parse the HTML data, this is all done for you! From the module in response we will get a dataframe containing our data in rows and columns . The columns are basically High , Low , Open , Close , Volume and Adj Close.

import pandas_datareader.data as web
import datetime as dt

ticker = "AAPL" # This is Apple's ticker
start = dt.datetime(2020,1,1) # Starting date in format (year,month,day)
end = dt.datetime(2020,12,1) # Ending date 

df = web.DataReader(ticker,"yahoo",start,end)
print(df.head()) # Prints first 5 columns of data
print(df.shape) # Prints the number of rows and columns
OUTPUT
  • df = web.DataReader(ticker,"yahoo",start,end)
    • This stores the dataframe obtained in the variable df. The first value passed here is ticker i.e. “AAPL” , second is the source name . As we are using yahoo finance here so it should be “yahoo”. Then the end and start date (in the form of datetime). You can see other data sources here.
  • print(df.head())
    • It prints the first 5 columns of the data obtained.
  • print(df.shape)
    • Prints the rows and columns in the dataframe . As seen in output there are 241 rows and 6 columns

You can also use end = dt.date.today() to set end as today.

Managing data with Dataframes

Few commands which you can use to extract custom data you want from the dataframe obtained in the above code.

  • df.head() : To get first 5 columns
  • df.tail() : To get last 5 columns
  • df.head(<int>) : To get first int no of columns . Like df.head(20) for first 20 columns
  • df.tail(<int>) : To get last int no. of columns.
  • df.shape : To get columns and rows
  • df.columns : To get no. of columns
  • df.loc["2020-01-06"] : To get data of exact date

How to store dataframe data to a file

After obtaining the data next task is to store it as the historical data is not going to change. It make our script fast as we can re-use the stored data instead of making api calls.

Storing dataframe data in a file using pickle

import pandas_datareader.data as web
import datetime as dt

ticker = "AAPL" 
start = dt.datetime(2020,1,1) 
end = dt.datetime(2020,12,1)  

df = web.DataReader(ticker,"yahoo",start,end)

# Now storing the data in csv file
df.to_csv("data.csv")

You can also use this to dump the object in a binary file.
df.to_pickle("data.bin") : Dumps in binary format.

Reading dataframe data from the file using pickle

Reading the data from file is as easy as storing it.

import pandas as pd

df = pd.read_csv("data.csv",parse_dates=True,index_col=0)
# or
df = pd.read_pickle("data.bin")

Caching the dataframe data to improve speed

DataReader also supports caching of data. Sending requests again and again may slow down your code and can also result in the ban of your IP.
You can create cache using request_cache by passing a request_cache.Session to DataReader using session parameter.

import pandas_datareader.data as web
import datetime as dt 
import requests_cache

ticker = "AAPL"
start = dt.datetime(2020,1,1)
end = dt.date.today()

expiry = dt.timedelta(hours=2)
session = requests_cache.CachedSession(cache_name="cache",backend="sqlite",expire_after=expiry)

df = web.DataReader(ticker,"yahoo",start,end,session=session)

When you will run this code from the second time onwards you will observe that the program runs very fast. The reason is it uses the cache to respond your requests. The cache is valid for the time given in expire_after parameter. In our case, it’s 2 hours.


Operating on pandas dataframe data

Now it’s time to perform some operations on the data and make it more appealing and meaningful. Starting with Cleaning the data.

Cleaning datareader data

If you clearly observe the data given below you will see that the data of some weekdays are missing. (Note: Stockmarket are closed on weekends)

Data of 20 Jan 2020 (Monday) is missing.
Like it there are many weekday’s data missing in the whole dataframe and if we perform any operation on this dataframe it won’t be perfect. So to solve this issue we need to create a column consisting of each weekdays then replace our previous dates column with this new one. This isn’t the only approach to handle “gap data” or “missing data”, but it is a simple method. Other approaches to take are to average the value between the missing row.

import pandas as pd
import datetime as dt

start = dt.datetime(2020,1,2)
end = dt.date.today()

df = pd.read_pickle("hello.bin")
alldays = pd.date_range(start=start, end=end, freq='B')
df = df.reindex(alldays)
df = df.fillna(method='ffill')
  • alldays = pd.date_range(start=start, end=end, freq='B')
    • This creates a DatetimeIndex (think it as a column) of all weekdays between start and end .
  • df = df.reindex(alldays)
    • This line replaces our previous Dates column with the new one that has all dates. All the dates that dont have a value are set as NaN meaning Not a Number.
  • df = df.fillna(method=”ffill”)
    • This replaces all the empty NaN data with the data of nearest date.

After performing this if we look at our new data we will see that. 20 Jan has been added and it’s data has been copied from 17 Jan.

Creating Moving Average with Pandas Dataframe

What is the moving average? Suppose if I say a moving average of 10 days data from today’s date of 2020-12-20. That means to get the average of all the data between 2020-12-11 to 2020-12-20. Then on 2020-12-21 the moving average would the be the average of all the data between 2020-12-12 to 2020-12-21 (the “window” then “moves” to the last 10 days). In stock analysis it helps to smoothen out the constant day to day changes to see if there is a general trending of upward or downward and to eliminate any fluctuations.

In this example we will create and a new column in our dataframe which will contain the moving average of 30 days.

import pandas as pd

df = pd.read_pickle("hello.bin")
df["ma30"] = df["Adj Close"].rolling(window=30,min_periods=0).mean()
print(df.tail())
OUTPUT

As you can see it created a new column in our dataframe as ma30 which has moving average of 30 days. We used used rolling() function for it.

  • window=30 : The no of Days in moving average
  • min_periods=0 : We used this option parameter to make sure our first 29 values aren’t empty . Because as they don’t have 30 days for first 29 values to take average. So this parameter makes sure that if they have less days than 30. No problem take average of the available days before it . Hence the moving average of first day is itself , the second day is average of first and second and so on till 30.

Visualize data with matplotlib

Looking at large texts doesn’t gives us much information. So let’s convert the obtained data to a graph. (Assuming the data is already stored in the file data.bin in binary format)

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_pickle("hello.bin")

df["Open"].plot()
df["Adj Close"].plot()
plt.legend()
plt.show()

Lets add some more code to improve the graph. Here we will be creating 2 sub-graphs embedded in one figure . The top one will be a line graph representing the Adj Close and below one is a filled polygon graph representing the volume. We will be using some advanced features of matplotlib.

subplot2grid is used to create graphs based on grid formats . Here we are creating a grid of 6 rows and 1 columns in which the first sub-graph will occupy 5 rows and second one will occupy 1 row. The size of first graph will be 5 rowspan and second’s will be 1 rowspan.

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_pickle("hello.bin")
fig = plt.subplots(figsize=(10,5))

ax1 = plt.subplot2grid((6,1),(0,0),rowspan=5,colspan=1)
plt.grid()
ax2 = plt.subplot2grid((6,1),(5,0),rowspan=1,colspan=1,sharex=ax1)

ax1.plot(df.index,df["Adj Close"])
ax2.fill_between(df.index,df["Volume"],0)

plt.show()
OUTPUT
  • fig = plt.subplots(figsize=(10,5))
    • Sets the window size. You can obiviously scroll the borders but initialising the window is recommended.
  • ax1 = plt.subplot2grid((6,1),(0,0),rowspan=5,colspan=1)
    • Creates space for first graph . (6,1) is the gridsize and (0,0) is the position from where the figure of this graph will start which is obiviously start point.
  • ax2 = plt.subplot2grid((6,1),(5,0),rowspan=1,colspan=1,sharex=ax1)
    • Creates space for second graph . (6,1) is the gridsize which will be same as given in the first one and (5,0) is the position which means from 5th row that is below the first graph.
  • ax1.plot(df.index,df["Adj Close"])
    • Plots the df.index i.e the Date on the x axis and "Adj Close" data on the y axis in the form of line graph .
  • ax2.fill_between(df.index,df["Volume"],0)
    • Plots the df.index i.e. the Date on x axis and "Volume" data on y axis in the form of filled polygons.

Conclusion

There are many types of analysis you can make on stock data such as: analysing if current value is at a week/month/annual high, if we have a trending upwards compared to the sector or the market as a whole, if the trend is due to announcements, and many more things. Here we’ve covered the very first and important step of extracting data and visualizing this which will be a great foundation to begin with.

Code away!

Get Notified Automatically Of New Articles

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!
Logging in Python 3, How To Output Logs to File and Console

Logging in Python 3, How To Output Logs to File and Console

Advanced

Logging is an essential tool for development as it helps us in keeping track of events, monitoring the program behavior, and debugging; and without it making big projects and running them is unimaginable. It is a vital tool to understand what’s happening in your code during development and testing, but also becomes an essential tool once your code goes into production

Using print() vs Logging

When we write a new application, we tend to always using print() to understand what’s happening in your code. You should stop doing that as soon as your code starts to becoming more than a quick hack. There are many advantages to using Logging vs print, and in fact logging does not take that much effort. Here are some quick points:

  • You generally only print out to the console and not to a file. So the output disappears after execution
  • There is no uniformity and each print statement is different – with logging, you can include time stamps, the calling function, the line of code. This is invaluable to debug issues
  • With Logging you can output to both console and a file, and have these appended over time so you get a history
  • You can turn on/off the level of information that is output for logging – e.g only DEBUG level, only ERRORs, etc

Basics of logging in python 3

Often we use the print() command for tracking events which is good for small programs but become difficult in complex programs. In python, we use logging module for logging.

Logging module comes built in with python so we don’t need to install it. Just import and use.

import logging

# The demo test code
logging.debug("This is debug")
logging.info("This is info")
logging.warning("This is warning")
logging.error("This is error")
logging.critical("This is critical")

The output logging.debug and logging.info is not shown due to the severity level. It’s a level which we use to define how important a line of the log will be. Only the logs of the level set and above are shown. There are 5 levels namely as DEBUG , INFO , WARNING , ERROR and CRITICAL.

  • DEBUG: Detailed information, typically of interest only when diagnosing problems.
  • INFO: Confirmation that things are working as expected.
  • WARNING: An indication that something unexpected happened, or indicative of some problem in the near future (e.g. ‘disk space low’). The software is still working as expected.
  • ERROR: Due to a more serious problem, the software has not been able to perform some functions.
  • CRITICAL: A serious error, indicating that the problem itself may be unable to continue running.

By default log level is set to WARNING so only logs of WARNING , ERROR and CRITICAL are shown we can change it by adding basicConfig command.

import logging

logging.basicConfig(level=logging.DEBUG)

# The demo test code
logging.debug("This is debug")
logging.info("This is info")
logging.warning("This is warning")
logging.error("This is error")
logging.critical("This is critical")

The function logging.DEBUG used in basicConfig does nothing than just returning a number which denotes the level. All the level have a number associated with them.

Level Numeric Value
CRITICAL 50
ERROR 40
WARNING 30
INFO 20
DEBUG 10
NOTSET 0

NOTE : While writing the logging command the level is always written in small letters . logging.info() not logging.INFO(). While declaring level we use capital letters basicConfig(level=logging.DEBUG) not basicConfig(level=logging.debug) .


Output logs to file

If you want to make the logs to be written in a file. This is easy to do. You just need to add filename="file.log" in the basicConfig file.

import logging

logging.basicConfig(filename="hello.log")

# The demo test code
logging.warning("This is warning")
logging.error("This is error")
logging.critical("This is critical")

Formatting the logging output

Now if you have closely observed you would have found that the logs are been written in specific formats i.e <level><name><message> . But we can change it as we want by adding the format parameters in basicConfig. eg:
logging.basicConfig(format="%(asctime)s:%(levelname)s:%(module)s:%(messages)s")
The name of the attribute (like asctime , levelname , message) written inside are case sensitive and each is separated by :

import logging

logging.basicConfig(filename="hello.log",format="%(asctime)s:%(levelname)s:%(message)s")

logging.warning("This is warning")
logging.error("This is Error")
logging.critical("This is Critical")

There are lots of attributes you can use. Listed Below

Attribute nameFormatDescription
args You shouldn’t need to format this yourself. The tuple of arguments merged into msg to produce message, or a dict whose values are used for the merge (when there is only one argument, and it is a dictionary).
asctime %(asctime)s Human-readable time when the LogRecord was created. By default this is of the form ‘2003-07-08 16:49:45,896’ (the numbers after the comma are millisecond portion of the time).
created %(created)f Time when the LogRecord was created (as returned by time.time()).
exc_info You shouldn’t need to format this yourself. Exception tuple (à la sys.exc_info) or, if no exception has occurred, None.
filename %(filename)s Filename portion of pathname.
funcName %(funcName)s Name of function containing the logging call.
levelname %(levelname)s Text logging level for the message ('DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL').
levelno %(levelno)s Numeric logging level for the message (DEBUG, INFO, WARNING, ERROR, CRITICAL).
lineno %(lineno)d Source line number where the logging call was issued (if available).
message %(message)s The logged message, computed as msg % args. This is set when Formatter.format() is invoked.
module %(module)s Module (name portion of filename).
msecs %(msecs)d Millisecond portion of the time when the LogRecord was created.
msg You shouldn’t need to format this yourself. The format string passed in the original logging call. Merged with args to produce message, or an arbitrary object.
name %(name)s Name of the logger used to log the call.
pathname %(pathname)s Full pathname of the source file where the logging call was issued (if available).
process %(process)d Process ID (if available).
processName %(processName)s Process name (if available).
relativeCreated %(relativeCreated)d Time in milliseconds when the LogRecord was created, relative to the time the logging module was loaded.
stack_info You shouldn’t need to format this yourself. Stack frame information (where available) from the bottom of the stack in the current thread, up to and including the stack frame of the logging call which resulted in the creation of this record.
thread %(thread)d Thread ID (if available).
threadName %(threadName)s Thread name (if available).

Now lets create something which is more comfortable and easy to understand using the Attributes mentioned above.

import logging
logging.basicConfig(format="%(asctime)s [%(levelname)s] - [%(filename)s > %(funcName)s() > %(lineno)s] - %(message)s",
                    datefmt="%H:%M:%S",
                    filename="sample.log")
def hellologs():
    logging.warning("Some wrong has occured")

hellologs()
OUTPUT

If you clearly observe you will easily understand how I created the format to look like this. Here what you will find is a new parameter which is datefmt and if you clearly observe the time in the logs you will observe that the date is not written its just time.
Well , datefmt parameter is passed to set the format of asctime . You can format it in you own way by using strings like.
%Y – Year, %m – Month, %d – Date, %H – Hour, %M – Minutes, %S – Second
Exmaples
“23/05/2020 21:52” > datefmt="%d/%m/%Y %H:%M"
“05-23 21:52:13” > datefmt="%m-%d %H:%M:%S"


Outputting Python Logs to Console and Files

By default, using the logging class you can only output the logging to just one channel. However, you can output to multiple streams using handlers. Handlers are useful flexible tools to output files, screen, email and others. You can do this by adding to your own logger instance which also has an advantage to be shared across multiple files. When your program has many python files and all using the logging module it’s preferred to use a logger object instance which is shared. A logger has many advantages like it avoids confusion since you can setup the configuration settings once and then reuse it everywhere.

import logging

mylogs = logging.getLogger(__name__)

file = logging.FileHandler("sample.log")
mylogs.addHandler(file)

# The demo test code
mylogs.debug("The debug")
mylogs.info("The info")
mylogs.warning("The warn")
mylogs.error("The error")
mylogs.critical("The critical")
OUTPUT
  • mylogs = logging.getLogger(__name__)
    • Creates an instance of the logger in variable mylogs . Now we will user mylogs instead of logging for storing logs.
  • file = logging.FileHandler("sample.log")
    • Now this is what we call a handler . The variable file is now a file handler . For creating a file handler you must pass a file name to it.
  • mylogs.addHandler(file)
    • Every Handler needs to be added to the logger.

So , now the reason that debug and info are not shown and the kind of weird format of output we got is because we have not set the level and format for the handler. So it’s on its default level (warning) and format.
Let’s change its format and level..

import logging

mylogs = logging.getLogger(__name__)
mylogs.setLevel(logging.DEBUG)

file = logging.FileHandler("sample.log")
file.setLevel(logging.INFO)
fileformat = logging.Formatter("%(asctime)s:%(levelname)s:%(message)s",datefmt="%H:%M:%S")
file.setFormatter(fileformat)

mylogs.addHandler(file)

# And all that demo test code below this
  • mylogs.setLevel(logging.DEBUG)
    • Although we need to set the level of the handler but before that we have to give a level to the logger . The level that is given to the logger should be as lowest as possible otherwise it will filter out rest levels of the logs.
  • file.setLevel(logging.INFO)
    • Pretty easy to set the level , isn’t it ?
  • fileformat = logging.Formatter("%(asctime)s:%(levelname)s:%(message)s",datefmt="%H:%M:%S")
    • Well, firstly we need to create a format that we will use for handlers. NOTE: This format can be used by any handler
  • file.setFormatter(fileformat)
    • Now setting the format of file handler to fileformat

Next we extend the handler to output to console as well

Outputting to Console and Files

Writing in the console is done by creating a Stream Handler next. The rest is as same as other handlers. You can configure the format, level, and all other properties as you want.

import logging

mylogs = logging.getLogger(__name__)
mylogs.setLevel(logging.DEBUG)

stream = logging.StreamHandler()
stream.setLevel(logging.INFO)
streamformat = logging.Formatter("%(asctime)s:%(levelname)s:%(message)s")
stream.setFormatter(streamformat)


mylogs.addHandler(stream)

# And all that demo code below this
OUTPUT

Colored Output

The traditional white output in the Console isn’t always helpful to find out what’s going on especially when you have long log files to investigate. To get logs output in different colours we will use coloredlogs module . Since its not preinstalled in python so you need to install it using pip.
pip install coloredlogs

So with all set let’s dive into the colorful console.

import coloredlogs, logging

mylogs = logging.getLogger(__name__)

coloredlogs.install(level=logging.DEBUG, logger=logger)
# Some examples.
mylogs.debug("This is debug")
mylogs.info("This is info")
mylogs.warning("This is warning")
mylogs.error("This is an error")
mylogs.critical("This is a critical message")
OUTPUT

coloredlogs creates a stream handler and attaches it to the logger passed. It has its own default format and colour settings which can be customised as per interest. Lets first look at the few important parameters we can pass in coloredlogs.install()

  • level
    • An integer to denote the level. (Remember logging.DEBUG returns an integer)
  • logger
    • Name of the logger in which this stream handler has to be attached. (Performs same act as logger.add_handler() )
  • fmt
    • A string denoting the format style.
  • datefmt
    • A string denoting the format of asctime .
  • level_styles
    • A dictionary containing the data of level and their colors
  • field_styles
    • A dictionary containing the data of field and their colors
import coloredlogs, logging

# Create a logger object.
mylogs = logging.getLogger(__name__)

fieldstyle = {'asctime': {'color': 'green'},
              'levelname': {'bold': True, 'color': 'black'},
              'filename':{'color':'cyan'},
              'funcName':{'color':'blue'}}
                                   
levelstyles = {'critical': {'bold': True, 'color': 'red'},
               'debug': {'color': 'green'}, 
               'error': {'color': 'red'}, 
               'info': {'color':'magenta'},
               'warning': {'color': 'yellow'}}

coloredlogs.install(level=logging.DEBUG,
                    logger=mylogs,
                    fmt='%(asctime)s [%(levelname)s] - [%(filename)s > %(funcName)s() > %(lineno)s] - %(message)s',
                    datefmt='%H:%M:%S',
                    field_styles=fieldstyle,
                    level_styles=levelstyles)
# Some examples.
mylogs.debug("This is debug")
mylogs.info("This is info")
mylogs.warning("This is warning")
mylogs.error("This is an error")
mylogs.critical("This is a critical message")

Example

Now we are going to create 3 handlers namely file , cric_file , stream .
file will be storing all the logs of level warning and above in the file sample.log
cric_file will be storing all the logs of critical level in the file Critical.log
stream will be showing all the logs in the console.
So, as I said about handlers they can be configured differently as you want; like having different levels and all other configs. Dont worry everything is explained below.

import logging

# Creating logger
mylogs = logging.getLogger(__name__)
mylogs.setLevel(logging.DEBUG)

# Handler - 1
file = logging.FileHandler("Sample.log")
fileformat = logging.Formatter("%(asctime)s:%(levelname)s:%(message)s")
file.setLevel(logging.WARNING)
file.setFormatter(fileformat)

# Handler - 2
cric_file = logging.FileHandler("Critical.log")
cric_file.setLevel(logging.CRITICAL)
cric_file.setFormatter(fileformat)
# format we can use it anywhere.

# Handler - 3
stream = logging.StreamHandler()
streamformat = logging.Formatter("%(levelname)s:%(module)s:%(message)s")
stream.setLevel(logging.DEBUG)
stream.setFormatter(streamformat)

# Adding all handlers to the logs
mylogs.addHandler(file)
mylogs.addHandler(stream)
mylogs.addHandler(cric_file)

# Some demo codes
mylogs.debug("debug")
mylogs.info("info")
mylogs.warning("warn")
mylogs.critical("critical")
mylogs.error("error")
  • import logging
    • Import the logging module
  • mylogs = logging.getLogger(__name__)
    • Defines a logger instance
  • file = logging.FileHandler("Sample.log")
    • Handler – 1
    • Creating the Handler command FileHandler and passing the filename for it.  
    • This handler will save all the logs of Warning and above in the file “sample.log”
  • fileformat = logging.Formatter("%(asctime)s:%(levelname)s:%(message)s")
    • Creating a format. (It’s independent. The file still hasn’t been set in this format. The format type is stored in the variable fileformat which can be used by any handler
  • file.setLevel(logging.WARNING)
    • Setting the level of this file handler
  • file.setFormatter(fileformat)
    • Now setting its format to fileformat defined just above.
  • cric_file = logging.FileHandler("Critical.log")
    • Handler – 2
    • This handler will save all the critical logs in the file “Critical.log”
    • Creating the Handler command FileHandler and passing the filename for it.
  • cric_file.setLevel(logging.CRITICAL)
    • Setting the level of this file handler
  • cric_file.setFormatter(fileformat)
    • Now setting its format to fileformat . Since we have already defined that format we can use it anywhere.
  • stream = logging.StreamHandler()
    • Handler – 3 
    • This handler is what will be shown in the console . Its called StreamHandler . And obivously we dont need a filename to pass.
  • streamformat = logging.Formatter("%(levelname)s:%(module)s:%(message)s")
    • Since we want a different format to what will be shown in the console so creating another formatter for it. Obiviously we can alos use the format defined before it i.e fileformat
  • mylogs.setLevel(logging.INFO)
    • Setting its level
  • stream.setFormatter(streamformat)
    • Setting its format 
  • mylogs.addHandler(file)
  • mylogs.addHandler(stream)
  • mylogs.addHandler(cric_file)
    • Now since we have defined all our handlers we added them to the mylogs logger.

Final Thoughts

Now that we have gone in depth into managing logs, you should be able to reuse this code in your future projects where you can save time and get more robustness to help you understanding what’s happening under the hood.

Subscribe to our newsletter

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!