How To Send Emails From Gmail Using Python 3

How To Send Emails From Gmail Using Python 3

Intermediate

Automating emails and integrating to gmail can be done fairly easily with Python 3 giving you a range of things that can make your applications richer. This will enable you to link email to one of your applications for things like weekly notifications, alerts, or even the logon confirmation. We will go through how to send emails using Gmail using Python 3 step by step.

How to access Gmail API

If this is you first time trying to use the Gmail API, we recommend you take a look into the our “How to access Gmail through API with Python 3” Blog in which we had learned how to use this API, it’s requirements and we have code sample in which we printed all email subjects. Either way, we will take a brief summary just to refresh

Pre-requisites

On our blog ” How to access Gmail through API with Python 3 ” we touched this topic in depth, so please refer to this blog if you get stuck setting the following pre-requisites

  • Python 2.6 or greater
  • The pip package management tool
  • A Google account with Gmail API enabled
  • OAuth 2.0 Client IDs credentials from Google API console
  • Google Client Libraries
    • pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib

Now that everything has been set and we know how to use the API we are going to dive a little bit more dipper, now we are going to learn how to write and send emails directly from our code. This will be very useful if we are going for example to automatize some alerts, or maybe some automatic replays with the customization that you want, your mind is your limit here

Gmail Access Scope

The scope within Gmail API dictates what permissions our application have, in other words, what is going to be able to perform with this API. As we are going to a reading and sending emails we are going to need the scope: compose

Scope Code Description Usage
https://www.googleapis.com/auth/gmail.composeCreate, read, update, and delete drafts. Send messages and drafts.Restricted

Creating an email

Emails are sent as base64url encoded strings within the raw property of a message resource. The high-level workflow to send an email is:

  1. Create the email content in some convenient way and encode it as a base64url string.
  2. Create a new message resource and set its raw property to the base64url string you just created.
  3. Call messages.send, or, if sending a draft, drafts.send to send the message.

Importing the necessary libraries for processing gmail

In order to create emails, we are going to need at least tow libraries. These are intended to handle the respective format of the content in different points of the process.

  • base64
    This module provides functions for encoding binary data to printable ASCII characters and decoding such encodings back to binary data
  • MIMEText
    A subclass of MIMENonMultipart, the MIMEText class is used to create MIME objects of major type text.
#import base64 lib in order to encode and decode
import base64 
# Import MIMEText in order to cast str into MIMEText object
from email.mime.text import MIMEText 

Creating email messages

In the following code, we are going to define a method that is going to receive all the parameters needed to build an email and will return an object which contains the message of the email.


Within our method, we create a variable called “message” which is a MIMEtext object, which contains our messages in plain text, and we add som values to handle the other parameters as the sender, to and the subject

 Args:

  • sender: Email address of the sender. 
  • to: Email address of the receiver. 
  • subject: The subject of the email message.
  • message_text: The text of the email message.

  Returns:    

  • A string containing an email.
def create_message(sender, to, subject, message_text):
  message = MIMEText(message_text,'plain')
  message['to'] = to
  message['from'] = sender
  message['subject'] = subject
  email = base64.urlsafe_b64encode(message.as_bytes()).decode()
  return email   

Before sending an email let’s understand a little bit the functions that we are going to use.

Sending an email through Gmail

Once we have a method that will create an object which will contain our different parameters of the email, we are ready to create another method which will be in charge of sending it.

This new method will receive and returns the following:

  • Args:
    • service: Authorized Gmail API service instance.
    • user_id: User’s email address. The special value “me” can be used to indicate the authenticated user. 
    • message: Message to be sent.
  •   Returns:
    •  Array with sent message information.

In addition to this, we are going to execute the send() method and store the returning value into a variable called message. This variable is going to be used to check if the email was sent, and identify the email’s id

def send_message(service, user_id, message):
    message = (service.users().messages().send( userId=user_id, body={'raw':message}).execute())
    return message 

Now that we have the two methods that we are going to need to send emails through Gmail API, we need to include some additional functions to complete the process.

Common functions for preparing the API calls for Gmail

If you are not familiar with the following code we encourage you to go to our ” How to access Gmail through API with Python 3 ” blog in which we explain everything in depth. These are helper functions that help to determine the security controls of access the emails.

# Import all needed libraries to use Gmail API

import pickle
import os.path
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
import base64
from email.mime.text import MIMEText

# Defines the permissions that are going to be requested to the end-user
# These permissions defines what our application can performe
SCOPES = ['https://www.googleapis.com/auth/gmail.compose']

def main():
    # Checking credentials 
    creds = get_permissions()
    
    #Declare Gmail API service
    service = build('gmail', 'v1', credentials=creds) 

# returns permissions granted by the end-user 
def get_permissions():
    creds = None
    if os.path.exists('token.pickle'):
        with open('token.pickle', 'rb') as token:
            creds = pickle.load(token)
    # If there are no (valid) credentials available, let the user log in.
    if not creds or not creds.valid:
        if creds and creds.expired and creds.refresh_token:
            creds.refresh(Request())
        else:
            flow = InstalledAppFlow.from_client_secrets_file('credentials.json', SCOPES)
            creds = flow.run_local_server(port=0)
        # Save the credentials for the next run
        with open('token.pickle', 'wb') as token:
            pickle.dump(creds, token)
    return creds

Complete script to send emails with gmail using python 3

And finally, after all the work is done, we are ready to execute our code and send emails from our application. All the code snippets seen above is merged to the below script

Here are the steps performed by the main function:

  • Stores credential’s data into the ‘creds’ variable executing the get_permissions() method.
  • Then use our ‘creds variable to initialize the Gmail API, and store it at the service variable.
  • At the following step, you will need to change the parameters to the ones needed, accordingly with the applications requirements, in this case, we are going to use test parameters .
  • Once the message is stored in our ‘msg’ variable we just call our send_message() so it sends the email, and at the same time we print the result so we know it has been sent successfully.

# Import all needed libraries to use Gmail API
from __future__ import print_function
import pickle
import os.path
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
import base64
from email.mime.text import MIMEText

SCOPES = ['https://www.googleapis.com/auth/gmail.compose']

def main():
    # Checking credentials
    creds = get_permissions()
    
    #Declare Gmail API service
    service = build('gmail', 'v1', credentials=creds)
    msg = (create_message('test@gmail.com','test@gmail.com','test','test_body'))
    print (send_message(service,"me",msg))


def get_permissions():
    creds = None
    if os.path.exists('token.pickle'):
        with open('token.pickle', 'rb') as token:
            creds = pickle.load(token)
    # If there are no (valid) credentials available, let the user log in.
    if not creds or not creds.valid:
        if creds and creds.expired and creds.refresh_token:
            creds.refresh(Request())
        else:
            flow = InstalledAppFlow.from_client_secrets_file('credentials.json', SCOPES)
            creds = flow.run_local_server(port=0)
        # Save the credentials for the next run
        with open('token.pickle', 'wb') as token:
            pickle.dump(creds, token)
    return creds

def create_message(sender, to, subject, message_text):
  message = MIMEText(message_text,'plain')
  message['to'] = to
  message['from'] = sender
  message['subject'] = subject
  email = base64.urlsafe_b64encode(message.as_bytes()).decode()
  return email 

def send_message(service, user_id, message):
    message = (service.users().messages().send( userId=user_id, body={'raw':message}).execute())
    return message

if __name__ == '__main__':
    main()  

Running the code

Congrats! Everything has been done, now we are ready to run our script and check the Gmail client to check our new incoming Email has been sent from our script. Just open a console/terminal an execute the script type:

# Replace script.py with the name given to your file
python script.py

After the code has been executed, you will get two results, one at the console/terminal and the other at your Gmail client account, your results will look similar to the following images

How to send emails using gmail using python 3 Console
Console / Terminal
How to send emails using gmail using python 3 Gmail client
Gmail client

Get Notified Automatically Of New Articles

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!
How To Access Your Gmail with Python 3

How To Access Your Gmail with Python 3

Intermediate

Have you ever wonder how to read and send an email, manage drafts and attachments, search threads and messages, work with labels, set up push notifications, and manage Gmail settings… Well you have come to the right place because now we are going to learn how to access Gmail through API with Python 3

What is Gmail API?

The Gmail API is a web service. It uses a RESTful API with JSON data which contains the data for the email (or sometimes referred to as “payload”). Gmail API can be used in a variety of different applications, some use cases are:

  • Read-only mail extraction, indexing, and backup
  • Label management (add/remove labels)
  • Automated or programmatic message sending
  • Migrating email accounts from other providers
  • Set standardized email signatures for users in a domain

Setting pre-requisites for Gmail API

First of all, we need to be sure that we accomplish the prerequisites so we can use the API, but don’t be scare, these requisites are very simple.

  • Python 2.6 or greater
    • Navigate to the Python downloads page: Python downloads.
    • Click on the link/button to download Python
    • Follow the installation instructions (leave all defaults as-is).
  • The pip package management tool
    • Follow this official guide: https://pip.pypa.io/en/stable/installing/
  • A Google account with Gmail API enabled
    • Log into your project at the Google Cloud Platform console
    • Go to the navigation bar, at the top left corner
    • Select API & services -> Library
    • At the search bar type “Gmail API”, then open the API
    • Click the “Enable” button if it is disabled

      How to access gmail through API with Python 3 - Gmail API enabled
  • Download OAuth 2.0 Client IDs credentials
    • Go to your project at the Google Cloud Platform console
    • Go to the navigation bar, at the top left corner, select API & services -> credentials
    • Click at “Create credentials” -> OAuth Client IDs
    • Download the document created at OAuth 2.0 Client IDs section
      How to access Gmail through API with Python 3 Credentials-min
    • Rename the file to credentials.json
  • Install the Google Client Library
    • pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib

Once you have done all the requisites, you are able to create a new python file, just be sure that the JSON file downloaded from the OAuth 2.0 Client IDs section (what we renamed to be credentials.json) at the Google Cloud Platform console is at the same folder that your python script.

Importing libraries

The following imports are needed to handle the Google API, as you can see the imports comes from google, but two of them, which are used to handle the permissions file. But let’s get some more information about each

  • pickle
    The pickle module implements binary protocols for serializing (converting an object into a string) and de-serializing (converting the string back to an object) to a Python object structure so you can save and restore
  • os.path
    This module implements some useful functions on pathnames, not files.
    The os module is used for accessing the filesystem.
  • The other 3 modules imported are used by Google API for Request handling and building and installation of the API
# Import all needed libraries to use Gmail API
import pickle
import os.path
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request

Defining scopes for Gmail API access

Scopes are intended to provide the right level access that our application, so based on the scope chosen, different permission levels will be prompted to the end-user and those permissions will be granted to our application after the end-user accept them.

The Gmail API scopes are the permissions granted by the end-user to the application itself, this means that the user will be asked to grant some permissions based on the scope. e.g: Scope: labels means that the application can create, update and delete labels only. By requesting the minimum level of access required, users feel more confident granting access.
The Gmail API supports the following scopes:

Scope CodeDescriptionUsage
https://www.googleapis.com/auth/gmail.labelsCreate, read, update, and delete labels only.Recommended
https://www.googleapis.com/auth/gmail.sendSend messages only. No read or modify privileges on the mailbox.Sensitive
https://www.googleapis.com/auth/gmail.readonlyRead all resources and their metadata—no write operations.Restricted
https://www.googleapis.com/auth/gmail.composeCreate, read, update, and delete drafts. Send messages and drafts.Restricted
https://www.googleapis.com/auth/gmail.insertInsert and import messages only.Restricted
https://www.googleapis.com/auth/gmail.modifyAll read/write operations except immediate, permanent deletion of threads and messages, bypassing Trash.Restricted
https://www.googleapis.com/auth/gmail.metadataRead resources metadata including labels, history records, and email message headers, but not the message body or attachments.Restricted
https://www.googleapis.com/auth/gmail.settings.basicManage basic mail settings.Restricted
https://www.googleapis.com/auth/gmail.settings.sharing Manage sensitive mail settings, including forwarding rules and aliases.

Note: Operations guarded by this scope are restricted to administrative use only. They are only available to Google Workspace customers using a service account with domain-wide delegation.
Restricted
https://mail.google.com/Full access to the account’s mailboxes, including permanent deletion of threads and messages This scope, should only be requested if your application needs to immediately and permanently delete threads and messages, bypassing Trash; all other actions can be performed with less permissive scopes.Restricted

Once the end-user accepts the scope requested, it will be stored on the pickle file, if any change is made to the scope, it will trigger a new request to the end-user and the token.pickle will be replaced with the new one

# Scope provide different levels of privileges, this example is with readonly privileges
# If modifying these scopes, delete the file token.pickle.
# token.pickle files are auto generated after first API usage
SCOPES = ['https://www.googleapis.com/auth/gmail.readonly']

Getting credentials for Gmail

The following code will trigger what we just discussed on the last paragraph, so in the previous code, we just declare the scope, now we are going to create a method to get the result of executing the request of permissions. This is intended to handle our authentication with the API. This method will trigger a new web browser windows/tab that will as the end-user for permissions within his Gmail account for our application, it will be triggered just the first time and then it will be stored at a new document called token.pickle which is automatically created.

The method itself go through the following steps:

  • Create ‘creds’ variable to store the result
  • Look for a file called ‘token.pickle’, if the file exists it is assigned to ‘creds’
  • If not exists it triggers a process to requests the permissions to the end-user through the API
  • If exists but it is expired, it will refresh the token
def get_permissions():
    creds = None
    if os.path.exists('token.pickle'):
        with open('token.pickle', 'rb') as token:
            creds = pickle.load(token)
    # If there are no (valid) credentials available, let the user log in.
    if not creds or not creds.valid:
        if creds and creds.expired and creds.refresh_token:
            creds.refresh(Request())
        else:
            flow = InstalledAppFlow.from_client_secrets_file('credentials.json', SCOPES)
            creds = flow.run_local_server(port=0)
        # Save the credentials for the next run
        with open('token.pickle', 'wb') as token:
            pickle.dump(creds, token)
    return creds

Getting messages from Gmail

Now we define a method which returns a list of emails which are called ‘messages’, the service variable is going to be declared at the end of the blog, within the main function, for now, let’s have in mind that we are calling the API and get the list of messages of the user ‘me’. The ‘me’ user means the end-user which is currently using our application

def get_messages(service):
    # Call the Gmail API, in order to get all messages
    return service.users().messages().list(userId='me').execute()['messages'] 

Getting email subjects

Once we have the list of emails from the API, we are able to request for the subjects of each email, so we are going to iterate into that list looking for some specific fields.

  • Define a list in which will be stored all the subjects found
  • We are looking for the id of the email, in order to identify each unique email.
  • After that, we call the API again in order to get all the information regarding that specific email and store the information on the ‘message’ variable
  • Once we have each email Id and its data, we are looking for the payload field due to it gathers the header’s value, both values are stored in individual variables but it can be collected in just one step.
  • Now we have the header of each email, we are going to iterate through it due we are looking for the Subject field, which is stored into a new variable and then appended to defined ‘subjects’ list
def get_subjects(emails, service):     
  subjects = []     
  for email in emails:         
    _id = email['id'] # get id of individual message         
    message = service.users().messages().get(userId='me', id=_id).execute() # fetch the message using API         
    payld = message['payload'] # get payload of the message          
    header = payld['headers'] # get header of the payload         

    for e_header in header:             
      if e_header['name'] == 'Subject':                 
        msg_subject = e_header['value']                 
        subjects.append(msg_subject)     
  return subjects 

Defining and calling main

It has been a lot of code and a lot of time to get into our final step. Here is where we are going to call all our predefined methods and finally print all the subjects collected.
In order to do that to following steps are performed

  • Declare a ‘creds’ variables to store our credentials (it include the scope)
  • Declare a ‘service’ variable which stored the service of the API, so the build method returns the service of Gmail with the respective credentials that were defined previously
  • Now call our predefined method to collect all the emails and store them into ’emails’ variable
  • Same procedure but now calling ‘get_subjects’ which returns a list with all the subjects of the emails provided as a parameter
  • And finally, just iterate through the list of subjects printing each subject.
def main():
    # Checking credentials
    creds = get_permissions()
    
    #Declare Gmail API service
    service = build('gmail', 'v1', credentials=creds)
    
    #Get all messages
    emails = get_messages(service)

    subjects = get_subjects(emails,service)
    
    for subject in subjects:
        print (subject)

if __name__ == '__main__':
    main() 
Output:
#Subjects:
10 destinos para visitar este verano
Developer .net en Praxis IT Costa Rica , nueva vacante de Developer en Heredia
Acabá el 2020 con COMPRAS INTELIGENTES

As can be seen on the following image, the subjects listed on the output can be found at the Gmail client as well

How-to-access-Gmail-through-API-with-Python-3-Gmail-client

Complete code

Here is all the code that we have developed through all the blog in just one block so you can analyze it in a complete file or maybe copy and paste if you prefer

# Import all needed libraries to use Gmail API
import pickle
import os.path
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request

# Scope provide different levels of privileges, this example is with readonly privileges
# If modifying these scopes, delete the file token.pickle.
# token.pickle files are auto generated after first API usage
SCOPES = ['https://www.googleapis.com/auth/gmail.readonly']

def main():
    # Checking credentials
    creds = get_permissions()
    
    #Declare Gmail API service
    service = build('gmail', 'v1', credentials=creds)
    
    #Get all messages
    emails = get_messages(service)

    #Get all subjects
    subjects = get_subjects(emails,service)
    
    #Printing all subjects
    for subject in subjects:
        print (subject)

def get_permissions():
    creds = None
    if os.path.exists('token.pickle'):
        with open('token.pickle', 'rb') as token:
            creds = pickle.load(token)
    # If there are no (valid) credentials available, let the user log in.
    if not creds or not creds.valid:
        if creds and creds.expired and creds.refresh_token:
            creds.refresh(Request())
        else:
            flow = InstalledAppFlow.from_client_secrets_file('credentials.json', SCOPES)
            creds = flow.run_local_server(port=0)
        # Save the credentials for the next run
        with open('token.pickle', 'wb') as token:
            pickle.dump(creds, token)
    return creds

def get_messages(service):
    # Call the Gmail API, in order to get all messages
    return service.users().messages().list(userId='me').execute()['messages']

def get_subjects(emails,service):
    subjects = []
    for email in emails:
        _id = email['id'] # get id of individual message
        message = service.users().messages().get(userId='me', id=_id).execute() # fetch the message using API
        payld = message['payload'] # get payload of the message 
        header = payld['headers'] # get header of the payload
        for e_header in header:
            if e_header['name'] == 'Subject':
                msg_subject = e_header['value']
                subjects.append(msg_subject)
    return subjects

if __name__ == '__main__':
    main() 

Resource types

The Gmail API provides several resource types, as they can be used in such different ways we will let you the following links so you can investigate deeper in the ones that you prefer

Sending emails

If you are interested in this API feature we invite you to read our upcoming blog: How to send emails using gmail using python 3 blog in which we will learn much more about this handy API

Get Notified Automatically Of New Articles

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!
Simple list comprehension examples in python 3

Simple list comprehension examples in python 3

Intermediate

Have you ever found yourself figuring out how to read lists of lists, filtering the list by element attributes, removing elements from a list, or flattening a matrix? Would you like to accomplish all this and write cleaner, comprehensive and much professional code at the same time?
Well once you have completely read this blog You will be not just able to do perform any task regarding lists but become a master handling list’s data and how list comprehension works in python 3.

List comprehensions

List comprehensions offer a concise way to create lists based on iterable objects such as list, str, and tuples or even some non-sequence types like dict, file objects, any object or method that implements sequence semantics like range() function.

What is an iterable object?

Iterable objects are known as Iterators and they are behind scenes when loops are used, Iterator in Python is simply an object that returns data, one element at a time.
Imagine an iterator like the structure of a list, so how a list delivers data is through the Iterator, which looks like to the picture below.

Iterators

If you want to go deeper on iterators you can watch the following video

List comprehension structure

Lists comprehensions consist of an expression which will be iterate followed by a for sentence (this for loop does not require the “:” at the end). It means that a list object can be created in just one line of code without making use of several loops. So the structure will look similar to this

List comprehension structure

Let’s go into a little example where a list of numbers has been given to us and we need to know the square value of each of the given numbers

# List of numbers given
numbers = [1,2,3,4,5,6,7,8,9,10]

# We are going to solve this without list comprehension
square_values = []
for number in numbers:
    square_values.append(number*number)

# Now we are going to solve with list comprehension
square_vaues_list = [number*number for number in numbers]

# Printing all values
print (numbers)
print (square_vaues_list)
print (square_values)
Output:
# Given numbers
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

# Without list comprehension solution
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

# List comprehension solution
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

As can be seen at the above code there is a list of numbers given from 1 to 10, we needed to know the square value of each element of this list so we iterate the list with a for loop in a way that each number is multiplied by itself on each iteration. A step by step will be like this:

  1. 1×1 = 1
  2. 2×2 =4
  3. 3×3 = 9
  4. 10×10 = 100

But this is just the beginning, multiplying a number by itself is not a big deal, so we are going to dig dipper into this and learn how list comprehension works in python 3 in much more detail so we can replace the traditional loops and even lambda functions, once you are able the replace some regular loops with list comprehension you can start using conditional within them to get much cleaner code but first let’s learn how to get conditional within a list conprehension

List comprehension with conditional statement

In the last example, a for loop was used, but now we are going to be using a different iterable object, now is the turn of range(). As You may know, range() will return a range object which is not a list but an iterable object, meaning that we can iterate each element one by one, keeping in mind that range() is a method it will receive some parameters that are shown on the following table

ParameterDescription
start (Optional)An integer number specifying at which position to start. Default is 0
stop (Required)An integer number specifying at which position to stop (not included).
step (Optional)An integer number specifying the incrementation. Default is 1

IF statement within list comprehension

IF list comprehension

The regular for vs list comprehension statements can be seen at the picture above where a 4 line code is translated into just 1 line and the if statement is added the the formular, if statement is optional but powerful, being added after the input sentence where we declare the scope of the loop.
The if statement filters our list, returning just the values that accomplish the statement

Now that we know how the structure works and how to replace a regular for loop and filter elements with the if statement, we are going a step forward on the following example.
Let’s imagine that we are looking for factors of 100, being a factor a number that divides 100 without a remainder. e.g: 100/2 = 50
To archive this we will need to use the operator modulo (%) which returns the remainder of dividing two numbers. e.g: 100%2 = 0

# We set a range from 1 to 101 so it starts on 1 due to we cannot divede by zero
# Then in the if statement we use the % operator to know if the current number is a factor of 100
factors = [num for num in range(1,101) if 100%num==0]

print (factors)
Output:
# All numbers are 100's factors
# List have been reduce from a range of 100 elements to a list of 9
[1, 2, 4, 5, 10, 20, 25, 50, 100]

Until now we have navigated into very nice features that can be useful in your every single day as a program due to iterators are everywhere but let´s know how to handle some headaches that each developer ever have at least once, let{s now learn how to handle a nested for loop, in other words, a for within another loop

Nested for loops

Nested loops have been the developer´s headache since memorial times but with the following examples, you will master these issues.

First of all, let’s understand what nested for loops are and how it works in a regular way, and how to handle list comprehension. Nested loops are just a loop within a loop meaning that the code will iterate an entire structure for each element of the first object, these kinds of loops could be a matrix or might be different structures look similar to the following image:

But let’s get our hands on it, lets first understand how nested loops work and how can we iterate on them with regular nested for loops

# creation of matrix
matrix = [] 
# iteration from 0 to 3 so 4 vectors are going to be created
for i in range(4): 
    # append an vector to the matrix
    matrix.append([]) 
    for j in range(4): 
    # append element on vector
        matrix[i].append(j) 

# print complete matrix
print(matrix)

# print all vectors of the matrix
for vector in matrix:
    print (vector)

# print every element of the matrix
for vector in matrix:
    for element in vector:
        print (element)
output: 
#Printing matrix
 [[1, 2, 3], [4, 5, 6], [7, 8, 9]]  

# printing each vector of the matrix
 [1, 2, 3]
 [4, 5, 6]
 [7, 8, 9]

# printing each element of the matrix
 1
 2
 3
 4
 5
 6
 7
 8
 9 

This seems to be a little bit confusing maybe, but ensure to understand how it works before jumping into list comprehension.
We are going to perform the same matrix creation but now in just one line

# creation of matrix
matrix = [[element for element in range(4)] for i in range(4)]

# following the structure of list comprehension, a list comprehension has been added at the outout section getting as result the list created by the second list comprehension

# now same prints than before to compare results

# print complete matrix
print(matrix)

# print all vectors of the matrix
for vector in matrix:
    print (vector)

# print every element of the matrix
for vector in matrix:
    for element in vector:
        print (element)
t: 
#Printing matrix
 [[1, 2, 3], [4, 5, 6], [7, 8, 9]]  

# printing each vector of the matrix
 [1, 2, 3]
 [4, 5, 6]
 [7, 8, 9]

# printing each element of the matrix
 1
 2
 3
 4
 5
 6
 7
 8
 9 

If you like functional programming

With list comprehension we can replace the use of functional methods like map(), filter(), and reduce()

Here is a video where you can go deeper on this extra topic: https://www.youtube.com/watch?v=JEapLNRuzVo

Some useful tips

  • We should avoid writing very long list comprehensions in one line to ensure that code is user-friendly.
  • Remember, every list comprehension can be rewritten in for loop, but every for loop can’t be rewritten in the form of list comprehension.

Conclusion

  • List comprehension is an elegant way to define and create lists based on existing lists.
  • List comprehension is generally more compact and faster than normal functions and loops for creating a list.

Subscribe to our newsletter

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!

Write Multiple JSON Objects To A File In Python 3

Write Multiple JSON Objects To A File In Python 3

Intermediate

Whether you have experience in coding or you are just starting to learn how to code, you must have heard of JSON data files, and it’s commonly used in coding that you found it very common. It’s a great way to store complex structures and also keep them in persistent storage such as in files as it becomes very readable to edit outside of your application.

What are JSON files and how do you use them

Well JSON is just an abbreviation of (JavaScript Object Notation). It’s a javascript programming language based type of file that is easy to read and understood by humans, and computers, and can be used in python, and other languages as well. The main use of JSON files is to transmit data between a server and an application. In the old days XML files were a popular alternative for human readable files, but they tended to have larger files due to the opening and closing tags, and were more complex to process. JSON files are much simpler and there are many languages with built in methods to process them easily. Python is no exception, there are several ways to process JSON files quite easily.

Let’s dive deeper and find out How to write multiple JSON objects to file in python 3. To answer this question we would like first to demonstrate the steps of opening JSON type files in python and then how to write JSON objects in python.

Reading a JSON File

A JSON files has the following format:

{
  "vehicle_type": "car",
  "make": "Toyota",
  "model": "Prius",
  "year": "2011",
  "license_plate" : "LX4590",
  "previous_owners": [ 
                       {"name" : "richard kimble",
                        "gender" : "male",
                        "age" : 42 },
                       {"name" : "michael scofield",
                        "gender" : "male",
                        "age" : 36 } ]
}

What you see in the above is a combination of single fields and an array with elements in there. A JSON dictionary would have a similar structure.

To open a JSON file in python is relatively easy, first we need to import the json module:

import json

and then to load it in we would

json.load(open('mydatafile.txt'))

Now that we loaded the file in, and if we need to open the file with the open method notice that it opens the file read mode, so no need to put the (‘r’) after the file. However if you want to make changes in the file, will need to add the (‘w’) or (‘a’) .

Converting to a JSON from Python objects

Moving on to the next step, where we need to write JSON objects in python, we use the json.dumps()

Let’s assume that we have a list its name is example_list example_list = ['e', 'o', 1 , 2, 3] and we would like to add it in a JSON file that we already have, this will be by using the json.dumps() like that

json.dumps(example_list)

And this will be the same with any type of objects that you would like to add in JSON strings and will be the same for dictionaries, lists, tuples, strings, int, float, True, False, and None.

However, we need first to use the dumps() to convert any of these types into JSON objects (JavaScript) and to know what are the equivalent types in JSON files check Figure 2

Figure 2

The output is fairly intuitive except the last few that can be a bit surprising. This means if you run print(json.dumps(True)) the out put will be true you can see this in Figure 3

Figure 3

and we get this out put as Figure 4 shows

Figure 4

So, when we use the json.dumps(example_list) that we used earlier the output will change the list to an array so we can add it to JSON file.

Writing JSON to a file

Now we know what .dumps() do, let’s add the object into JSON file and to do so we can use the open() function, but we prefer to use the with : statement for better handling for the JSON file, and it also closes the file and saves the added data, and it’s syntax:

with open ('mydatafile.txt') as j_file:

Keep in mind that if you don’t add the ‘w’ after the file it will automatically be read only and you will not be able to modify or add anything to it. And if we use the normal .open() statement we will have to close the file with .close() statement, now lets create a JSON file and add multiple objects into it.

Creating JSON file and adding multiple objects

To create a JSON file with the with : statement, first we will create a list and a dictionary to help us demonstrate the whole idea:

add_list = [0, 5, 9]
add_dict = { 'word':50, 'line':10, 'page':5 }

Let’s use the same file as the one that we used before mydatafile.txt. For the code that we need to write the add_list and add_dict into the our JSON file and this code will be as follows:

import json
with open ('mydatafile.txt', 'w') as jf:
     list = json.dumps(add_list)
     dict = json.dumps(add_dict)
     jf.write(list)
     jf.write('\n')
     jf.write(dict)

This will create a file with add_list and add_dict as an array and an object to be used as JSON file and we can see the code and the out put in the figures 5 and 6

Figure 5
Figure 6

So, to add multiple objects into JSON file we need to convert it’s type and this can be done easily by json.dumps(), but what if we want to add something like another list instead of the list that’s already there, for instance, lets say we want to change the add_list with the new_list = [1, 6, 7] we will have to brake the JSON file into lines by .readlines will look like this

import json
with open('mydatafile.txt', 'r') as jf:
    lines = j.readlines()

Then we will work on the line that we want to change and in this case we want to change the first line so we will work on lines[0]

lines[0] = new_list + '\n'

Now we open the file normally and add the new lines by .writelines method, just like that

with open('mydatafile.txt', 'w') as jf:
   jf.writelines(lines)

And the output will be the modified JSON file that we wanted to create as shown in Figure 7

Figure 7

With the same concept we can read lines or reach multiple objects on JSON file, it’s by using lines[ ]after reading the file and if there is nested lists or dictionaries we can reach all the objects we want by reading the file first and then lines [ ] [ "str"].

Conclusion

We have covered what is JSON files and that they are display as dictionaries in python, then we moved into how to open a JSON file and how to create one and how to modify and add into the file using different kind of methods. We know for a fact that JSON is a serialized data structure, it is not plain text. So the easy answer for our main question of ‘How to write multiple JSON objects to file in python 3’ is that you need to break the JSON file and to know the index of the place you want to add the object in because as we mentioned before that JSON files are structured, unless you want to add at the end of the JSON or in the beginning.

Subscribe to our newsletter

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!

How To Print String in Color In Python 3

How To Print String in Color In Python 3

Beginner

Python offers an option to print strings in color so that information in the console can be much easier to comprehend. This can be incredible useful when reading long log files, seeing console output, or even creating your own text based adventure games. By using this feature we can print string in different colors and highlights. In this article, we will see this concept thoroughly.

Normally, you can give color to string in python using some prefix in postfix to string. But if you have a facility like different packages or libraries then you can save a lot of time.

Let’s Look At Our Options

There are in fact quite a few packages which are available from python pip which can help you colourize the console. Here is a list of common ones that I’ve found especially useful to add some personality to my console output:

  • sty
  • colorama

How To Colour Console Output With Python Package: sty

To begin with you can install the package sty as follows:

pip install sty
installation of sty using pip

I am supposing you have any mentioned IDE already installed. So, we are moving towards the practical part.

A sty is a library which is helpful in customizing or decorating string. A sty is short of style and by using this package, you can colorize the foreground and background both. Additionally, you can give some effect to string such as make them italic or bold.

There are mainly four primitives which is used while customizing string which is as follows:

  1. ef(effect-register)
  2. fg(foreground-register)
  3. bg(background-register)
  4. rs(reset-register)

Each of these primitives has different kinds of default attributes and by using that we can give style to the text. Let’s look at a small example: in which we list out default attributes of ef and fg.

First import the library

from sty import fg, ef, rs, Style, RgbFg,bg, RgbBg

ef_dir=dir(ef)
fg_dir=dir(fg)
print("ef_dir: ",ef_dir)
print("fg_dir: ",fg_dir)

If you have correctly all the things then this thing will work fine and now let’s look at the output of this code snippet. As you can see we have got all default attributes present in both of that module. In ef_dir we have, italic, bold, inverse, and fg has different enlisted colors like blue, black, and many more which we can directly use.

So, yes now we know what’s inside this package. Thus, it’s time to get your hands dirty or colorful using sty.

First, we learn about how to colorize the text and then we focused on the background. Last, we will talk about how to give both effects simultaneously.

To begin with, we will give a red color to your name then we give a yellow background to the same string.

name="Enter your name here" #enter your name here

red_color_string = fg.red + name + rs.fg
print(red_color_string)
red-colored string output

As you can see, it’s fairly straight forward code where we gave the red foreground flag then we printed out the text. We also signalled to end this red-colored flag using reset primitive which signals to end whatever we signal earlier as here we give instruction to end foreground color. Otherwise, the red color will be included in all subsequent text.

Let’s add a yellow background now.

yellow_backgroud_string = bg.yellow + name + rs.bg
print(yellow_backgroud_string)
Yellow backgrounded string

Now, I am putting both of these things together. But with a slight change, as I am changing string color to dark red.

both_effect = fg.da_red + bg.yellow + name + rs.all 
print(both_effect)
Both effects in one string

Yes, your question is correct. Red and yellow are pretty common colors and what if I want to choose some different colors. What about your other color choices. W e can check our color choices too easily using the dir function as we already attached in the output. but we will see that thing again here.

print(dir(fg))
print(dir(bg))

As you can see, you have a wide range as green, blue, light blue, dark blue, cyan, magenta, white present on both of the list. Still not excited? Well, you can input the RGB value and it will give you that color for background and foreground. Let’s try that out.

What you are actually going to do is adding your own color in the fg or bg dictionary and using it later. First, pick some color using google color picker. Just search the color picker in your browser and select any color.

We are choosing color between aqua and green with some dark shade and background as dark pink or maybe you will see when you try it. So, it’s time to look at the code.

fg.text_color = Style(RgbFg(25,209,111)) #RgbFg is foregroundcolor
bg.background_color = Style(RgbBg(204,42,87)) #RgbBg is backgroundcolor

final_output_string = fg.text_color + bg.background_color+ name + rs.all
print(final_output_string)
custom colorized string

As you can see the library is pretty flexible since you can even give out a favorite color for text or as background using any RGB values. Thus, you can do so many things using the sty package. The final task we are going to do is give some effects to these texts like making the text bold or underlined.

So, Let’s explore ef primitive and by using that we are going to generate some bold and underlined text as mentioned

bold_text = "This is bold text"
underlined_text = "This is underlined text"

bold_output = ef.bold + fg.text_color + bg.background_color + bold_text + rs.all
underlined_output = ef.underl + fg.da_red + underlined_text + rs.all

print(bold_output,underlined_output,sep='\n')
effect primitive

I hope you all understand all of these three tags completely. You can explore more function of ef primitive using dir and you can generate your own color to customize these strings. So, try out all of these codes and try to minor changes in this code as it will give you a better understanding.

How To Colour Log Output With Python Package: colorama

Colorama works similarly to sty in that you put a start and end tag to colour your console output. In this section, I’ll go one step further and show how it can be setup to also output your log lines in color (you can in fact do the same thing with sty).

So in normal operation, you can colour your text with colorama as follows:

import colorama
from colorama import Fore, Style

print(Fore.BLUE + "Hello world + Style.RESET_ALL)

Similarly to sty, this will enable you to set the text to BLUE, and then reset the text at the end in order to ensure future text is with the default setting.

You can chose the following foregrounds:

Fore: BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, WHITE, RESET.

And the following background colors:

Back: BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, WHITE, RESET.

And also the following styles:

Style: DIM, NORMAL, BRIGHT, RESET_ALL

So it is pretty straight forward to work with. You will notice that Style has a constant called “RESET_ALL” which can be used to revert and of foreground background or the style.

When it comes to log files, this is the usual setup:

    logInst = logging.getLogger("mylog")

    logFormatter = logging.Formatter("%(asctime)s %(message)s")
    logInst.setLevel(logging.DEBUG)
    consoleHandler = logging.StreamHandler(sys.stdout)
    consoleHandler.setFormatter(logFormatter)
    logInst.addHandler(consoleHandler)

What this does is to ensure that any calls to logging (e.g. logging.debug) will result in the output going to the console.

Hence, to add color to the console, you simply have to edit the formatter:

 logInst = logging.getLogger("mylog")

    logFormatter = logging.Formatter(Fore.BLUE + "%(asctime)s [%(filename)s:%(lineno)s - %(funcName)5s() ] [%(levelname)-5.5s] " + Style.RESET_ALL + " %(message)s"  )
    logInst.setLevel(logging.DEBUG)
    consoleHandler = logging.StreamHandler(sys.stdout)
    consoleHandler.setFormatter(logFormatter)
    logInst.addHandler(consoleHandler)

The output will show the output as follows – here we’ve added some more useful information in the debug output to make it more readable.

Final Thoughts

All in all, the coloriziation of console is an immensely useful feature and can save some considerable time when looking through log files. Hence, it’s well worth the extra few minutes to set this up.

Subscribe to our newsletter

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!