How to Get FX Currency Rates using Python 3

How to Get FX Currency Rates using Python 3

Intermediate

Dealing with multiple currencies or FX rates in python is a common challenge for many applications such as for ecommerce sites fintech applications, or simply some clever bots that analyze the markets. It is fairly straight forward to get currency data and to manipulate them using python.

The versatility of the Python language allows us to not only get the rates but also create a currency converter with the aid of APIs that are readily available.

In this article, we are going to look at currency rates APIs, how we can use them, and build our own currency converter.

What is an API?

API stands for application programming interface. It allows programs to communicate and share data. It is like a railroad track using which data can be transferred from one terminal to another terminal.

API data is often transmitted in JSON (JavaScript Object Notation) format which is a human readable format that helps to manage data easily. Python comes with modules that are designed to handle that operation – namely requests and json.

Choosing an API

There are numerous currency rate APIs – some you can use right away and others requiring you to get an API key. The API you decide to use is completely up to you. For the purpose of this article, we will use these two APIs to illustrate an issue you may come across.

How to Build a Currency Converter using RestAPI.IO

We will start with a program that converts only two currency pairs – in this case, the United States Dollar (USD) and the Euro (EUR). The RestAPI.io service provides a straightforward mechanism to get the conversion by simply sending the base currency (the currency you are converting from) to the target currency (the currency you are converting to). Here is the sample code:

import requests
import json

def main():
    payload={"base":"USD","symbols":"EUR"}
    response=requests.get('https://api.ratesapi.io/api/latest', params=payload)
    if response.status_code !=200:
        print("Status Code :",response.status_code)
        raise Exception("There was an error accessing the web")
    
    print("Header:", response.headers['Content-Type'])
    data=response.json()
    print("JSON data",data)
      
if __name__ == "__main__":
    main()

In this program, we import requests and json then create a function main():

  • We then create a variable that sets the base currency and the currency we want to convert to (“payload”) in a dictionary format
  • We use the get method of requests and pass the API URL and the variable we created earlier – notice we send the dictionary data which gets converted to URL query string. e.g. in the above example, the URL gets sent as: https://api.ratesapi.io/api/latest?base=USD&symbols=EUR. You can even test the url out where you can get the result through your browser:
  • We have to check if we got a response from the API – 200 code – if we don’t, print the status code and an error message. The 200 code is the success code returned by web browsers.
  • If we got a response, print out the header information we fetched
  • We use json to convert the JSON data to a dictionary and output that as well

The program works well, but if we wanted to compare different currencies we would have to modify the code. We need a way to choose what currency rate we want to compare against the base we would have set.

NOTE: A 200 response code indicates successful communication to the API

Let’s see how we can make something a bit better:

import requests

def currency_converter():
        old_or_latest = input("Do you want the latest or historical value?\n"
        "if latest, please type 'latest' or else type historical value in '%Y-%m-%d' format:   ")
        base_URL_new = 'https://api.ratesapi.io/'+old_or_latest 
        
        response = requests.get(base_URL_new, params='base=USD')
           
        if response.status_code == 200:
            data,base_value, Date = response.json().items()
            print('Available Currency\n>>', data[1].keys())
            desired_currency = input("Please provide the currency code\n>>>").upper()
            print(f'{desired_currency}:{data[1][desired_currency]} | base:{base_value[1]} | date:{Date[1]}')

            convert = input('Do you want to convert the '+ desired_currency+ ' value to USD?\t  yes or no: ')
            if convert == 'yes' or convert == 'y':
                value = float(input("value\n>>>"))
                USD_value = value/float(data[1][desired_currency])
                print(f'The value is:\t ${USD_value}')
            else:
                print("\nThank you")
        else:
            print ("\nPlease give the correct data")
    
currency_converter()

This program is more functional than the previous one. Unlike the first, it asks the user if they want the latest rates data or from a specific point in time.

  • Using the get method, we pass the new API URL and the base currency. When you call restAPI with just the base currency code, the API will in fact return all the rates that currency can convert to. See this example of what happens when the API is called with just USD:
  • We then make the check to see if the response we receive is 200
  • If it was, we fetch the JSON data, convert it to a dictionary, and output the dictionary keys – the currency code
  • We ask the user to select a currency they would like to compare against the base – that rate is outputted along with the date
  • In addition, we ask if they would like to convert a specific amount – we perform that calculation and output that as well

We are able to get more information from the API and perform additional tasks with it. A better converter, yes? There is something else we can do to our code, specifically to handle errors that may arise when we make API calls.

How to Handle Errors from API calls

If an error occurs when we try to get the API data, we do not want to display an error message a user would not understand. We avoid that by nesting the line of code when we make the call in a try/except block. In this example, we will use exchangerates.io

We will make slight modifications to our converter:

import requests
import sys

def currency_converter():
        old_or_latest = input("Do you want the latest or historical value?\n"
        "if latest, please type 'latest' or else type historical value in '%Y-%m-%d' format:   ")
        base_URL_new = 'https://api.ratesapi.io/'+old_or_latest 
    
        try:
            response = requests.get(base_URL_new, params='base=USD')
        except:
            print("Unable to reach API")
            sys.exit()
        
           
        if response.status_code == 200:
            data,base_value, Date = response.json().items()
            print('Available Currency\n>>', data[1].keys())
            desired_currency = input("Please provide the INR or other currency value\n>>>").upper()
            print(f'{desired_currency}:{data[1][desired_currency]} | base:{base_value[1]} | date:{Date[1]}')

            convert = input('Do you want to convert the '+ desired_currency+ ' value to USD?\t  yes or no: ')
            if convert == 'yes' or convert == 'y':
                value = float(input("value\n>>>"))
                USD_value = value/float(data[1][desired_currency])
                print(f'The value is:\t ${USD_value}')
            else:
                print("\nThank you")
        else:
            print ("\nPlease give the correct data")
    
currency_converter()

We have additionally imported sys. Instead of outright making the get request, we have nested that in a try block. The way that works is:

  • We will try to
  • If that is successful the rest of the program executes
  • If not, the except block will run – where we output a message and exit the program

To show the output if such an error occurs, I have intentionally modified the API URL so the get request fails.

This is a much better error message that doesn’t leave the user confused.

NOTE: Some currency rate APIs hace a limit to the number of calls you can make to them per day.

We have seen what APIs are, used two such APIs to fetch currency rate data. We have also used that data to create our own currency converters.

Get Notified Automatically Of New Articles

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!
How to Use argv Command Line Parameters in Python

How to Use argv Command Line Parameters in Python

Beginner

The command line is a big part of using and understanding programming in Python. In this article, we will cover the command line parameters argv and argc – their origin and how you can use them.

Command line parameters or arguments are a way to externally input data into your program(script) while it is executing – at runtime. This is achieved by providing the script name and parameters to be inputted. We will look at this in-depth later in this article.

Python command line arguments are inherited from the C programming language – that is where argv and argc originate from.

A lot of you would be familiar with how command line parameters work in C. Below is one such example:

#include <stdio.h>

int main(int argc, char *argv[]) {
    printf("Arguments count: %d\n", argc);

    for (int i = 0; i < argc; i++) {
        printf("Argument %6d: %s\n", i, argv[i]);
    }
    return 0;
}

In the code above, main()defines the execution point of the C program. It has the parameters:

  • argc – the number of arguments in the program
  • argv – an array with the name of the program as the first element followed by any additional arguments that may have been passed.

Running the code we get this output:

NOTE: argc stands for argument count, while argv stands for argument vector.  

Command Line Parameters in Python

To use argv in Python, you have to first import the sys module that comes built into Python. After import, it would be accessed with the sys.argv command.

The next question would be what is sys.argv? Let’s have a look at that, shall we? Consider the code below:

import sys

print(sys.argv)
print(type(sys.argv))

Here is what happened:

  • The first line imports the sys module
  • The third line prints out the value of sys.argv
  • The fourth line prints out the type of sys.argv

When we run this program in the command line with python <filename> we get the following output:

The output of sys.argv is the name of the file back in list format. We also see that the type of sys.argv is a list.

Now we will rewrite the code that was shown in C but in Python this time:

import sys

if __name__ == '__main__':
    print(f"Arguments count: {len(sys.argv)}")
    for i, arg in enumerate(sys.argv):
        print(f"Argument {i:>6}: {arg}")

NOTE: f-strings provide a way to embed expressions inside string literals, using a minimal syntax.

After running this program: python <filename> we see that the output we get is the same as the C program:

Taking a closer look we see that have one argument and that argument is the file itself.

Python does not have an argc variable because sys.argv is more than sufficient. If we wanted to get the number of parameters we can call the len() function as illustrated above.

Adding Parameters to sys.argv

I mentioned earlier that command line parameters are accomplished by giving the script name and parameters to be inputted: <script name> <parameter 1> <parameter 2>

Let’s add some parameters to the previous program. We are not changing anything in the code just adding arguments when we invoke it.

python <filename> Python Command Line Arguments

Each of the parameters is outputted as an element of the sys.argv list. Given that, you can specify what you want to be displayed from the list, and list operations can be performed on it.

import sys

print(f'Name of the script: {sys.argv[0]=}')
print(f'Arguments of the script: {sys.argv[1:]=}')

This is how the code works after import of the sys module:

  • The third line accesses the first element of the list – the name of the script – and extracts it.
  • The fourth line outputs the remaining elements of the list – these are the command line arguments.

About now, you may be asking what is the point of all this? Let’s take a look at some ways we can use sys.argv

How to Use sys.argv

As stated prior, command line arguments like sys.argv allow the input of data into a script externally. We do this because sometimes we do not want to hard code data to our program. sys.argv simplifies the user input process. For example, a simple calculator in Python would look something like this:

# Python program for simple calculator 

# Function to add two numbers 
def add(num1, num2): 
	return num1 + num2 

# Function to subtract two numbers 
def subtract(num1, num2): 
	return num1 - num2 

# Function to multiply two numbers 
def multiply(num1, num2): 
	return num1 * num2 

# Function to divide two numbers 
def divide(num1, num2): 
	return num1 / num2 

print("Please select operation -\n" 
		"1. Add\n" 
		"2. Subtract\n"  
		"3. Multiply\n"  
		"4. Divide\n") 

# Take input from the user 
select = int(input("Select operations form 1, 2, 3, 4 :")) 

number_1 = int(input("Enter first number: ")) 
number_2 = int(input("Enter second number: ")) 

if select == 1: 
	print(number_1, "+", number_2, "=", add(number_1, number_2)) 

elif select == 2: 
	print(number_1, "-", number_2, "=", subtract(number_1, number_2)) 

elif select == 3: 
	print(number_1, "*", number_2, "=", multiply(number_1, number_2)) 

elif select == 4: 
	print(number_1, "/", number_2, "=", divide(number_1, number_2)) 
    
else: 
	print("Invalid input") 

The code consists of four fuctions, each performing a different mathematical operation. A function is called based on a choice made by the user.

The issue, as we see in the output, is that the program asks for user input multiple times. This has the possibility of leading to input errors.

We could eliminate the number of inputs the user has to enter by allowing them to input all of the required data all at once:

import sys 

def main():
    if len(sys.argv) == 4:
        if sys.argv[1] == 'add':
            print(int(sys.argv[2]) + int(sys.argv[3]))
        elif sys.argv[1] == 'sub': 
            print (int(sys.argv[2]) - int(sys.argv[3])) 
        elif sys.argv[1] == 'mult': 
            print (int(sys.argv[2]) * int(sys.argv[3])) 
        elif sys.argv[1] == 'div': 
            print (int(sys.argv[2]) / int(sys.argv[3])) 
        else:
            print("INVALID ARGUMENTS")

if __name__ == '__main__': 
	main()       

This program:

  • Checks if the length of sys.argv is equal to 4
  • Based on what is inputted in the second list element ( sys.argv[1] ), it performs the operation corresponding to that input on the third ( sys.argv[2] ) and fourth ( sys.argv[3] ) elements
  • Ending by outputting that result

We get the same result but with only one user input. Let’s have a look at another example:

import sys 

add = 0.0

# Getting the length of command line arguments 
num = len(sys.argv) 

for i in range(1, num): 
	add += float(sys.argv[i]) 

print ("the sum is :", add) 

In this program:

  • An initial variable of add is declared and assigned the value 0.0
  • The length of sys.argv is stored in another variable
  • We iterate over the values from 1 to the length of sys.argv incrementing each value to add
  • The new value of add is then output.

By using the length of sys.argv this program is able to add every argument inputted regardless of their number.

These are some of the ways you can utilise command line parameters in Python.

Subscribe to our newsletter

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!

How to Read And Update Smartsheets With Python 3

How to Read And Update Smartsheets With Python 3

Intermediate

Smartsheet is a cloud-based management tool used by over 7000 companies which include Fortune 500 companies like Netflix. This system helps the user share documents, assign tasks, and much more. Smartsheet has its API available in Python which allows users to update and read smartsheets. This API also manages folders and administrator accounts. Python also has small-scale modules in which API isn’t needed.

In this article we shall look at how to setup smartsheet SDK, accessing smartsheet files, and updating them using different methods.

Preparation

1. Installing Prerequisites Modules

Before installing the smartsheet SDK please note having the modules listed below is compulsory. The installation method is also given.

1- requests
2- requests-toolbelt
3- six
4- certifi
5- python-dateutil

pip install requests
pip install requests-toolbelt
pip install six
pip install certifi
pip install python-dateutil

2. Installaing Smartsheet SDK

To install the Smartsheet SDK module in python can be done using the package manager (pip). The installation method is:

pip install smartsheet-python-sdk

If you already have Smartsheet SDK installed please remember to upgrade it using the following method:

pip install smartsheet-python-sdk --upgrade

Getting API Access Token from Smartsheet

The first step in getting started with Smartsheet Python SDK is generating the access token. To generate the access token follow the following steps:

Step 1:

Login to your smartsheet account and if you do not have an account on Smartsheet you can create one for free.

Step 2:

Go to your account and then “Apps and Integration”.

Step 3:

Click “Generate new access Token” and give your access token a name as you can see below.

Step 4:

A token will be generated against the name. Don’t forget to copy it as it won’t be available later on.

Finally, when you will see the screen given below it means your access token has been created successfully.

Adding Access Token to your Program

They are two ways of adding the access token in your code.

1. First Method

If you are using the Pycharm IDE the token needs to be configured in the Environment Variables through Edit Configuration. The same goes for Anaconda Spyder IDE.

2. Second Method

The access token can be added directly in the code and a connection can be made. This method isn’t secure in the long run since anyone can see your code.

Creating a Sample Smartsheet

In this article, we shall be working with a sample smartsheet that I have created in excel.

After creating the sample excel file to the Smartsheet homepage and then from the Create dropdown select import excel file.

Your smartsheet is successfully imported if you see the following screen. Click import and you will have a smartsheet ready to use for this article.

Creating a Successful Connection

After creating a smartsheet access token and smartsheet let’s see if our connection is successful. To check if a connection is successful we need to first initialize a client that uses the access token in its API call.

import smartsheet

SMARTSHEET_ACCESS_TOKEN = "08nyhgggzczqlwbdlr62ru8jrh"
# Initialize client
smartsheet_client = smartsheet.Smartsheet(SMARTSHEET_ACCESS_TOKEN)
print(smartsheet_client)
# Check for any errors
smartsheet_client.errors_as_exceptions(True)

Output:
If the code executes successfully that means connection is successful.

Using GetSheet Method to connect to a single worksheet

First, we need to note the sheet id of the sample sheet just imported. Open the smartsheet and open the File dropdown and select Properties. Copy the sheet id and save it in a variable.

You can create an API Call using the get sheet method with the sheet id parameter. The API call returns a JSON response. We will be printing the sheet name for now.

#Sheet_id gotten from Smartsheet
sheet_id = 2503335061284740

sheet = smartsheet_client.Sheets.get_sheet(sheet_id)
print("Loaded Sheet: " + sheet.name)

Output:

Reading Cell Values from the Smartsheet

First, we need to understand the structure of JSON which is returned when a connection is made with a sheet. Within rows we have cells and in the cell array, we have keys by the name, value given. So our code will be structured within two loops as given.

import smartsheet

SMARTSHEET_ACCESS_TOKEN = "08nyhgggzczqlwbdlr62ru8jrh"
# Initialize client
smartsheet_client = smartsheet.Smartsheet(SMARTSHEET_ACCESS_TOKEN)
#Sheet_id gotten from Smartsheet
sheet_id = 2503335061284740
sheet = smartsheet_client.Sheets.get_sheet(sheet_id)

#Reading Row in Loop
for row in sheet.rows:
   #For each row read cell value
   for cell in row.cells:
       if cell.value!= None:
           print(cell.value)

Output:

Understanding the Smartsheet Structure

Before learning how to update a cell in a smartsheet, we need to understand the structure of our smartsheet. A smartsheet has three major components:

  1. Rows
  2. Columns
  3. Cells

Now, each row and column has a unique id that we use to distinguish them. We will learn how to map rows and columns to their IDs further on.

1. Map Columns to Column ID

In smartsheets, each column has a column id which is then used for updating cells or rows which we will see further on in this article. We can create a dictionary that maps the column value to the column id.

#Empty Dictionary
column_map ={}
for column in sheet.columns:
 #Map Values
 column_map[column.title] = column.id
print(column_map)

2. Map Rows to Row IDs

As columns have column ids that can be updated rows in the smartsheet also can be mapped to row id which is further used to update the specific row. We can create an empty dictionary and map the row id to the row number using a counter variable.

row_map = {}
i=0 #counter
for rows in sheet.rows:
 #Map Row ID
 row_map[i]=rows.id
 i=i+1
print(row_map)

Output:

Creating a new Cell to Update

Updating a cell requires three things. First is to build a new cell with values needed and then build an existing row in which the cell exists. Finally calling the update row method to update the smartsheet.

1. Build a Cell

To update a particular cell we need to first build a new cell that will access the column id using the dictionary built above. Then we need to assign an updated value. To create a new cell the method used is:

smartsheet_object_variable.models.Cell()

After assigning this method to a variable the properties can be assigned meaningful values to it.

#Build the cell you are updating with
new_cell = smartsheet_client.models.Cell()
#Update column Remaining
new_cell.column_id = column_map["Remaining"]
new_cell.value = 0
new_cell.strict = False

2. Build a Row

The second step in updating a cell is building the row according to its column id. For this example, we shall be updating the first row. To build a row the method used is:

smartsheet_object_variable.models.Row()

After assigning this method to a variable the properties can be assigned meaningful values to it. The cells append method is used to update the row value.

# Build the row to update
get_row = smartsheet.models.Row()
get_row.id = row_map[0]
get_row.cells.append(new_cell)

3. Using Update Row Method

Finally after building a new cell and row to update the update row method is called with sheet id as parameter and the row variable that we built as shown below.

updated_row = smartsheet_client.Sheets.update_rows(sheet_id,[get_row])

The code will have run successfully if there are no errors and you can reload your smartsheet to see the updated value in the cell.

Update an Image in a Cell

Smartsheet also allows you to update a given cell with an image. To do this the method “add_image_to_cell” is used. The add image method takes the parameters sheet id, row id, column id, picture, and the file type as parameters.


You need to add the specified image in your current directory and use the “add_image_to_cell” function which will upload the file to smartsheet automatically.

import smartsheet

SMARTSHEET_ACCESS_TOKEN = "08nyhgggzczqlwbdlr62ru8jrh"
# Initialize client
smartsheet_client = smartsheet.Smartsheet(SMARTSHEET_ACCESS_TOKEN)

#Empty Dictionary
column_map ={}
for column in sheet.columns:
 #Map Values
 column_map[column.title] = column.id
print(column_map)

row_map = {}
i=0 #counter
for rows in sheet.rows:
 #Map Row ID
 row_map[i]=rows.id
 i=i+1
print(row_map)

column_id = column_map["Remaining"]
row_id = row_map[4]
picture = "flower.jpg"
file_type = "jpg"
smartsheet_client.Cells.add_image_to_cell(sheet_id, row_id, column_id, picture, file_type)

The code above successfully adds an image to the cell specified.

Output:

Conclusion

Smartsheet is a very versatile tool which can be used to act as a powerful end user database. With the power of automating data reads or data writes with python 3, you have the option to provide some powerful solutions to users.

Subscribe to our newsletter

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!

How To Decode And Encode In Base64 In Python 3

How To Decode And Encode In Base64 In Python 3

Intermediate

Are you working with an application that needs to transmit and store the binary data over to a network that can only handle only textual form of data? Well then encoding your binary data to Base64 is your solution because this scheme can be used to format data into a string of ASCII characters.

Base64 encoding scheme is important because it makes sure that the data doesn’t get modified during the process of transferring or storage. Base64 tells us that we can represent 64 different characters and numbers using this base system that means at least 6 bits are required to represent a single character.

In this article we shall look at the process of how we can encode or decode strings, images, urls and binary data into Base64 using Python and explore the libraries that are involved in the process.

Encoding Strings using Base64

In python3 we have a module called base64 which allows us to encode data. To import the module we use the following syntax:

import base64

In this module we have a function called b64encode() which helps perform base64 encoding. This function takes a bytes object  as a parameter and returns a Base64 bytes object. The format of using b64encode() is :

base64.b64encode(variable_bytes_object)

The variable should always be a bytes like object but what if we need to encode a string to Base64. To encode strings in Base64 the process is first converting the string using encode() method into bytes and then using b64encode() method. Let’s look at an example of converting string to base64:

import base64

str_example = "Python is the best programming language"
str_to_bytes = str_example.encode("ascii") #we have an encoded string
bytes_to_base64= base64.b64encode(str_to_bytes)#encoding bytes to base64
print(bytes_to_base64)
#returns decoded string of base64
base64_output = bytes_to_base64.decode('ascii')
print(base64_output)

Output:

Since we know that b64encode returns a byte-like object which is shown in the first print statement. To convert it to a string we can use the decode() method with the parameter we want to decode it into. To prevent any data loss or corruption the main thing to remember is that the encoding and decoding format should always be the same.

Encoding URL/FileNames to Base64

Base64 provides another method to ensure safe URL encoding which can also be used for naming files. As we know that the characters like “\”, “<”, ”>”, “{“ and ”}” are the unsafe characters for naming files or using in a url. The method used for encoding urls is:

base64.urlsafe_b64encode(variable_bytes)

This method converts the “+” and “/” sign to “-” and “_” respectively. Let’s understand this with an example:

import base64

str_url = "abc9+"

url_bytes = str_url.encode("utf-8")
#URL to base64 encoding
url_base64 = base64.urlsafe_b64encode(url_bytes)
print(url_base64)
#base64 to url
base64_str = url_base64.decode('utf-8')
print(base64_str)

Output:

Encoding Image in Base64

Using the base64 module we can also encode an image. You can download a sample image from google and use it. We will first open the file in “rb” mode instead of “r” which is necessary because this opens the file in binary format. Let’s look at the code for encoding an image in Base64 using python.

import base64

#open file in binary read mode
with open("flower.jpg", "rb") as image:
   encoded_image = base64.b64encode(image.read())

print(encoded_image)

Output:

The output may be different according to the image chosen. Please note this can also be used for text or pdf files.

Encoding binary data to Base64

The b64encode() method takes binary as an input so we have one step less to do while encoding binary data to Base64. Let’s take a look at an example of encoding binary data to base64.

import base64

binary_data = b'\x00\xFF\x00\xFF\x00\xFF'
data_base64 = base64.b64encode(binary_data)
base64_string = data_base64.decode('utf-8')

print(base64_string)

Output:

Encoding JSON in Base64

JSON API is the designated way used for exchanging data between websites, mobile applications or servers. Python3 has a builtin module “json” that helps us to convert python objects into JSON. If in any case you want to encode the json object to a base64 encoded string, there is no straightforward way to do that, First you have to convert the JSON object into string by using the json.dumps method and then finally encode that string into base64. Let’s look at it through an example.

import json
import base64

dict_data = {"language": "Python"}
json_data = json.dumps(dict_data)
print(json_data)
print(type(json_data))
json_b64 = base64.b64encode(json_data.encode('utf-8'))
print(json_b64)

Output:

Before encoding the data to base64 converting it into bytes is necessary that’s why data is passed into encode function first and converted into the utf-8 format and then the resulting bytes object is passed to the base64.

Encoding Zip Files into Base64

Python3 is extremely flexible in encoding and decoding of data. It’s possible to encode zip files using the b64encode method. We can use the same methods to encode and decode zip files as used earlier only difference is that we need to use python’s open method to read the zip file in binary mode and then after converting it to base64 using the same method we have to re-write the converted output into the new file so let’s take a look on how we can do it.

import base64

with open('file_input.zip', 'rb') as file_input, open('file_output.zip.b64', 'w') as file_output:
    base64.encode(file_input,file_output)

Output:

The output is an encoded base64 zip file.

File needs to be read in binary mode as the encode method accepts binary data as input. Please note that multiple context support is only available in Python 3.x. This code is efficient since it doesn’t store data read in memory.

Decoding Base64 to string

Decoding is basically the reverse of what happens in encoding. The base64 string is decoded into bytes which is then converted to strings. The decode method is available in the “base64” module. The method used for decoding is as follows :

base64.b64decode(variable_bytes_type)

Let’s now take a look at a real example using the b64decode method. The string we shall use shall be the output encoded string of the first example.

import base64
#encoded string from first example
b64_str = "UHl0aG9uIGlzIHRoZSBiZXN0IHByb2dyYW1taW5nIGxhbmd1YWdl"
b64_str = b64_str.encode('ascii')
b64_bytes = base64.b64decode(b64_str)
decode_str = b64_bytes.decode('ascii')
print(decode_str)

Output:

As you can see in the output the string was correctly decoded. Please note that method “b64decode” takes a bytes like object as input.

Base64 URL FileNames Decoding and using UTF-8

The method “urlsafe_b64decode()” is similar to the default decoding method in the base64 module. The difference is that it uses this character “_” instead of “/” and “-” character instead of “=”. The method syntax is as follows:

base64.urlsafe_b64decode(b64_encoded_str)

The urlsafe method takes an encoded base64 string as parameter and returns a byte object. Now let’s implement this using an example.

import base64

b64_str = "aGVsbG8gd29ybGQ="
url_bytes_b64 = base64.urlsafe_b64decode(b64_str)
str_64 = str(url_bytes_b64, "utf-8")
print(str_64)

Output:

Subscribe to our newsletter

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!

How To Convert Bytes To String And String To Bytes In Python 3

How To Convert Bytes To String And String To Bytes In Python 3

Intermediate

One of the most important features that have been introduced in Python3 is that the string type is now Unicode and there is a significant difference between text and binary data. When a computer receives data in the form of characters/text it converts it into Unicode. The format of the string encoding is always provided with the string, but if it is missing the interpreter won’t be able to understand it. The standard encoding method is Unicode Transformation Format – 8-bit (UTF-8) decided worldwide but other encoding methods will definitely be encountered.

According to programming concepts byte is a group of 8 bits and these 8 bits represent 256 values. A string is a group of characters and these characters are then encoded to bytes for the machine i.e computer to understand.

In this article, we will learn how to create byte objects in Python3. We shall also look at different methods for encoding and decoding bytes and string. Lastly, we will understand the importance of the conversion of bytes to string and vice versa.

Creating Bytes Object in Python

First, let’s see how to create a bytes object in python. Python has two built-in methods for creating byte objects which are bytes() and bytearray(). The difference between these two methods is that the bytes() method is immutable which means it cannot be changed while bytearray() method is mutable, let us look at some examples for better understanding.

bytes() method

bytes(x, encoding, error)

Let’s look at an example of how to use bytes() method first. If we pass this method an integer as a parameter it returns a byte object with the specified size while if we pass it a string, it returns the string bytes object.

#using string
str = "Bytes Object in String"
convert = bytes(str,'utf-8')
print(convert)
print(type(convert))

#using integer
convert = bytes(8)
print(convert)
print(type(convert))

Output:

bytearray() method

bytearray() method requires a source value which can be an integer or a string as a parameter. other parameters can be the encoding format you want to use ( UTF-8 is a default format in case you don’t provide this parameter) and error which tells what to do if encoding fails. This method returns a new bytes array object. The syntax for using bytearray() method is as follow:

bytearray(x, encoding, error)

Let’s look at an example using the bytearray() method. If we use an integer it returns a byte array with the specified size while string returns the byte array string.

#using string
str = "Bytes Object in String"
convert = bytearray(str,'utf-8')
print(convert)
print(type(convert))

#using integer
convert = bytearray(8)
print(convert)
print(type(convert))

b’str’ format

Using b before a string also creates a byte object in python. The b that is prefixed before the string stands for byte and is the designated way to represent byte object in python. The syntax of creating a byte object this way is simple:

b"mystring"
b'''
mystring
'''

Using b we can create byte objects using triple or double quotations. Let’s look at an example of how to use this.

print(b"Encoding byte string")
print(b'''
Encoding byte string
''')

Output:

encode() method

encode() method in python can also be used to convert string to bytes. It encodes the string to the specified encoding. By default “UTF-8” is used and encode method returns a byte object. The parameters used in the encode method are optional. The syntax for encode() method is:

str.encode(encoding=encoding, errors=errors)

For the first example let’s suppose we have a string that we need to convert to bytes. We will see simple conversion,UTF-8, and UTF-16.

#Simple encoding to bytes
str = "$c6"
str = str.encode()
print(str)

#special characters to utf-8
encoded = 'τoρνoς'.encode('utf-8')
print(encoded)

#special characters to utf-16
encoded = 'τoρνoς'.encode('utf-16')
print(encoded)

Output:

decode() method

The decode() method in python converts bytes to the original string.  The default encoding assumed is  “UTF-8”. Decode() method is the opposite of encode() method. The syntax used for decode() method is:

str.decode(encoding,error)
#Simple decoding to string
byte = b'$c6'.decode()
print(byte)

#utf-8 to string
decoded = b'\xcf\x84o\xcf\x81\xce\xbdo\xcf\x82'.decode('utf-8')
print(decoded)

#utf-16 to string
decoded = b'\xff\xfe\xc4\x03o\x00\xc1\x03\xbd\x03o\x00\xc2\x03'.decode('utf-16')
print(decoded)

Encoding Newline in Bytes

We have seen how encoding works when we have a string input. Suppose the string we have has multiple words on new lines so let’s look at how encoding shall work in this scenario.

str_multiline = '''
Python 2.7
Python 3.6
Python 3.7
'''
print(str_multiline.encode('utf-8'))

Output:

As you can see in the output the newline is shown using “\n”.

Splitting Bytes in Newlines

In the example above we saw a string that had newlines been converted to bytes. Let’s now take a look at how we can convert bytes to string keeping the formatting intact.

bytes_multiline = b'\nPython 2.7\nPython 3.6\nPython 3.7\n'

bytes_multiline_str = bytes_multiline.decode('utf-8')
print(bytes_multiline_str)
print(type(bytes_multiline_str))

Output:

Decoding bytes using UTF-8 parameter replaces the “\n” with a newline also the type returned using the decoded method is a string. This is one way to split bytes in newlines let’s take a look at another way this can be achieved.

bytes_multiline = b'\nPython 2.7\nPython 3.6\nPython 3.7\n'

for word in bytes_multiline.decode('utf-8').split("\n"):
   print(word)

Output:

This method loops over each character and splits each word on the basis of “\n” and prints it accordingly. Decoding is done on a character level.

File Handling

Writing Bytes to File

First, we need to understand what the “File I/O Operations” actually work like. There are different access modes in the open() method which includes r,rb+,w,wb+. The r and rb mode means that file is opening in read-only mode while rb means that file is opening in binary format in read-only mode.

Suppose we want to write bytes to a file in python. The mode for writing is “w” but since we want to write in binary format the mode “wb” is used.

bytes_input = b"0x41x420x43xcfx84oxcf"
file = open("bytes.txt", "wb") #opening file in binary form
file.write(bytes_input)
file.close()

Output:

Reading Bytes from File

As we saw in writing bytes it is important to understand “File I/O Operations”. In this example, we shall assume we have a text file that will be read in binary mode. The sample.txt file contains a single string “Hello”. Let’s see how to output each character as a byte.

with open("sample.txt", "rb") as file:
 byte = file.read(1)
 while byte:
   print(byte)
   byte = file.read(1)

Output:

Each character is read and printed as a byte from the text file until it reaches the end of the line.

Importance of Bytes and String Conversion

As we know bytes can be understood by computers directly. This helps in saving data on the disk immediately unlike strings that require being encoded and then saved.

One of the most popular applications nowadays of conversion of strings to bytes is in Machine Learning. Machine Learning models are stored in pickle files that are encoded in bytes.

Subscribe to our newsletter

Join the Python Insiders Group and get FREE tips in your inbox
Also, when you subscribe, we will send you a list of the most useful python one liners which will help you save time, make your code more readable, and which you can use immediately in your code! Subscribe to our email list and get the list now!