How To Decode And Encode In Base64 In Python 3

by Charles White | Input Output, Intermediate

Are you working with an application that needs to transmit and store the binary data over to a network that can only handle only textual form of data? Well then encoding your binary data to Base64 is your solution because this scheme can be used to format data into a string of ASCII characters.

Base64 encoding scheme is important because it makes sure that the data doesn’t get modified during the process of transferring or storage. Base64 tells us that we can represent 64 different characters and numbers using this base system that means at least 6 bits are required to represent a single character.

In this article we shall look at the process of how we can encode or decode strings, images, urls and binary data into Base64 using Python and explore the libraries that are involved in the process.

Encoding Strings using Base64

In python3 we have a module called base64 which allows us to encode data. To import the module we use the following syntax:

import base64

In this module we have a function called b64encode() which helps perform base64 encoding. This function takes a bytes object  as a parameter and returns a Base64 bytes object. The format of using b64encode() is :

base64.b64encode(variable_bytes_object)

The variable should always be a bytes like object but what if we need to encode a string to Base64. To encode strings in Base64 the process is first converting the string using encode() method into bytes and then using b64encode() method. Let’s look at an example of converting string to base64:

import base64

str_example = "Python is the best programming language"
str_to_bytes = str_example.encode("ascii") #we have an encoded string
bytes_to_base64= base64.b64encode(str_to_bytes)#encoding bytes to base64
print(bytes_to_base64)
#returns decoded string of base64
base64_output = bytes_to_base64.decode('ascii')
print(base64_output)

Output:

Since we know that b64encode returns a byte-like object which is shown in the first print statement. To convert it to a string we can use the decode() method with the parameter we want to decode it into. To prevent any data loss or corruption the main thing to remember is that the encoding and decoding format should always be the same.

Encoding URL/FileNames to Base64

Base64 provides another method to ensure safe URL encoding which can also be used for naming files. As we know that the characters like “\”, “<”, ”>”, “{“ and ”}” are the unsafe characters for naming files or using in a url. The method used for encoding urls is:

base64.urlsafe_b64encode(variable_bytes)

This method converts the “+” and “/” sign to “-” and “_” respectively. Let’s understand this with an example:

import base64

str_url = "abc9+"

url_bytes = str_url.encode("utf-8")
#URL to base64 encoding
url_base64 = base64.urlsafe_b64encode(url_bytes)
print(url_base64)
#base64 to url
base64_str = url_base64.decode('utf-8')
print(base64_str)

Output:

Encoding Image in Base64

Using the base64 module we can also encode an image. You can download a sample image from google and use it. We will first open the file in “rb” mode instead of “r” which is necessary because this opens the file in binary format. Let’s look at the code for encoding an image in Base64 using python.

import base64

#open file in binary read mode
with open("flower.jpg", "rb") as image:
   encoded_image = base64.b64encode(image.read())

print(encoded_image)

Output:

The output may be different according to the image chosen. Please note this can also be used for text or pdf files.

Encoding binary data to Base64

The b64encode() method takes binary as an input so we have one step less to do while encoding binary data to Base64. Let’s take a look at an example of encoding binary data to base64.

import base64

binary_data = b'\x00\xFF\x00\xFF\x00\xFF'
data_base64 = base64.b64encode(binary_data)
base64_string = data_base64.decode('utf-8')

print(base64_string)

Output:

Encoding JSON in Base64

JSON API is the designated way used for exchanging data between websites, mobile applications or servers. Python3 has a builtin module “json” that helps us to convert python objects into JSON. If in any case you want to encode the json object to a base64 encoded string, there is no straightforward way to do that, First you have to convert the JSON object into string by using the json.dumps method and then finally encode that string into base64. Let’s look at it through an example.

import json
import base64

dict_data = {"language": "Python"}
json_data = json.dumps(dict_data)
print(json_data)
print(type(json_data))
json_b64 = base64.b64encode(json_data.encode('utf-8'))
print(json_b64)

Output:

Before encoding the data to base64 converting it into bytes is necessary that’s why data is passed into encode function first and converted into the utf-8 format and then the resulting bytes object is passed to the base64.

Encoding Zip Files into Base64

Python3 is extremely flexible in encoding and decoding of data. It’s possible to encode zip files using the b64encode method. We can use the same methods to encode and decode zip files as used earlier only difference is that we need to use python’s open method to read the zip file in binary mode and then after converting it to base64 using the same method we have to re-write the converted output into the new file so let’s take a look on how we can do it.

import base64

with open('file_input.zip', 'rb') as file_input, open('file_output.zip.b64', 'w') as file_output:
    base64.encode(file_input,file_output)

Output:

The output is an encoded base64 zip file.

File needs to be read in binary mode as the encode method accepts binary data as input. Please note that multiple context support is only available in Python 3.x. This code is efficient since it doesn’t store data read in memory.

Decoding Base64 to string

Decoding is basically the reverse of what happens in encoding. The base64 string is decoded into bytes which is then converted to strings. The decode method is available in the “base64” module. The method used for decoding is as follows :

base64.b64decode(variable_bytes_type)

Let’s now take a look at a real example using the b64decode method. The string we shall use shall be the output encoded string of the first example.

import base64
#encoded string from first example
b64_str = "UHl0aG9uIGlzIHRoZSBiZXN0IHByb2dyYW1taW5nIGxhbmd1YWdl"
b64_str = b64_str.encode('ascii')
b64_bytes = base64.b64decode(b64_str)
decode_str = b64_bytes.decode('ascii')
print(decode_str)

Output:

As you can see in the output the string was correctly decoded. Please note that method “b64decode” takes a bytes like object as input.

Base64 URL FileNames Decoding and using UTF-8

The method “urlsafe_b64decode()” is similar to the default decoding method in the base64 module. The difference is that it uses this character “_” instead of “/” and “-” character instead of “=”. The method syntax is as follows:

base64.urlsafe_b64decode(b64_encoded_str)

The urlsafe method takes an encoded base64 string as parameter and returns a byte object. Now let’s implement this using an example.

import base64

b64_str = "aGVsbG8gd29ybGQ="
url_bytes_b64 = base64.urlsafe_b64decode(b64_str)
str_64 = str(url_bytes_b64, "utf-8")
print(str_64)

Output:

Subscribe to our newsletter

How To Convert Bytes To String And String To Bytes In Python 3

by Charles White | Input Output, Intermediate

One of the most important features that have been introduced in Python3 is that the string type is now Unicode and there is a significant difference between text and binary data. When a computer receives data in the form of characters/text it converts it into Unicode. The format of the string encoding is always provided with the string, but if it is missing the interpreter won’t be able to understand it. The standard encoding method is Unicode Transformation Format – 8-bit (UTF-8) decided worldwide but other encoding methods will definitely be encountered.

According to programming concepts byte is a group of 8 bits and these 8 bits represent 256 values. A string is a group of characters and these characters are then encoded to bytes for the machine i.e computer to understand.

In this article, we will learn how to create byte objects in Python3. We shall also look at different methods for encoding and decoding bytes and string. Lastly, we will understand the importance of the conversion of bytes to string and vice versa.

Creating Bytes Object in Python

First, let’s see how to create a bytes object in python. Python has two built-in methods for creating byte objects which are bytes() and bytearray(). The difference between these two methods is that the bytes() method is immutable which means it cannot be changed while bytearray() method is mutable, let us look at some examples for better understanding.

bytes() method

bytes(x, encoding, error)

Let’s look at an example of how to use bytes() method first. If we pass this method an integer as a parameter it returns a byte object with the specified size while if we pass it a string, it returns the string bytes object.

#using string
str = "Bytes Object in String"
convert = bytes(str,'utf-8')
print(convert)
print(type(convert))

#using integer
convert = bytes(8)
print(convert)
print(type(convert))

Output:

bytearray() method

bytearray() method requires a source value which can be an integer or a string as a parameter. other parameters can be the encoding format you want to use ( UTF-8 is a default format in case you don’t provide this parameter) and error which tells what to do if encoding fails. This method returns a new bytes array object. The syntax for using bytearray() method is as follow:

bytearray(x, encoding, error)

Let’s look at an example using the bytearray() method. If we use an integer it returns a byte array with the specified size while string returns the byte array string.

#using string
str = "Bytes Object in String"
convert = bytearray(str,'utf-8')
print(convert)
print(type(convert))

#using integer
convert = bytearray(8)
print(convert)
print(type(convert))

b’str’ format

Using b before a string also creates a byte object in python. The b that is prefixed before the string stands for byte and is the designated way to represent byte object in python. The syntax of creating a byte object this way is simple:

b"mystring"
b'''
mystring
'''

Using b we can create byte objects using triple or double quotations. Let’s look at an example of how to use this.

print(b"Encoding byte string")
print(b'''
Encoding byte string
''')

Output:

encode() method

encode() method in python can also be used to convert string to bytes. It encodes the string to the specified encoding. By default “UTF-8” is used and encode method returns a byte object. The parameters used in the encode method are optional. The syntax for encode() method is:

str.encode(encoding=encoding, errors=errors)

For the first example let’s suppose we have a string that we need to convert to bytes. We will see simple conversion,UTF-8, and UTF-16.

#Simple encoding to bytes
str = "$c6"
str = str.encode()
print(str)

#special characters to utf-8
encoded = 'τoρνoς'.encode('utf-8')
print(encoded)

#special characters to utf-16
encoded = 'τoρνoς'.encode('utf-16')
print(encoded)

Output:

decode() method

The decode() method in python converts bytes to the original string.  The default encoding assumed is  “UTF-8”. Decode() method is the opposite of encode() method. The syntax used for decode() method is:

str.decode(encoding,error)

#Simple decoding to string
byte = b'$c6'.decode()
print(byte)

#utf-8 to string
decoded = b'\xcf\x84o\xcf\x81\xce\xbdo\xcf\x82'.decode('utf-8')
print(decoded)

#utf-16 to string
decoded = b'\xff\xfe\xc4\x03o\x00\xc1\x03\xbd\x03o\x00\xc2\x03'.decode('utf-16')
print(decoded)

Encoding Newline in Bytes

We have seen how encoding works when we have a string input. Suppose the string we have has multiple words on new lines so let’s look at how encoding shall work in this scenario.

str_multiline = '''
Python 2.7
Python 3.6
Python 3.7
'''
print(str_multiline.encode('utf-8'))

Output:

As you can see in the output the newline is shown using “\n”.

Splitting Bytes in Newlines

In the example above we saw a string that had newlines been converted to bytes. Let’s now take a look at how we can convert bytes to string keeping the formatting intact.

bytes_multiline = b'\nPython 2.7\nPython 3.6\nPython 3.7\n'

bytes_multiline_str = bytes_multiline.decode('utf-8')
print(bytes_multiline_str)
print(type(bytes_multiline_str))

Output:

Decoding bytes using UTF-8 parameter replaces the “\n” with a newline also the type returned using the decoded method is a string. This is one way to split bytes in newlines let’s take a look at another way this can be achieved.

bytes_multiline = b'\nPython 2.7\nPython 3.6\nPython 3.7\n'

for word in bytes_multiline.decode('utf-8').split("\n"):
   print(word)

Output:

This method loops over each character and splits each word on the basis of “\n” and prints it accordingly. Decoding is done on a character level.

File Handling

Writing Bytes to File

First, we need to understand what the “File I/O Operations” actually work like. There are different access modes in the open() method which includes r,rb+,w,wb+. The r and rb mode means that file is opening in read-only mode while rb means that file is opening in binary format in read-only mode.

Suppose we want to write bytes to a file in python. The mode for writing is “w” but since we want to write in binary format the mode “wb” is used.

bytes_input = b"0x41x420x43xcfx84oxcf"
file = open("bytes.txt", "wb") #opening file in binary form
file.write(bytes_input)
file.close()

Output:

Reading Bytes from File

As we saw in writing bytes it is important to understand “File I/O Operations”. In this example, we shall assume we have a text file that will be read in binary mode. The sample.txt file contains a single string “Hello”. Let’s see how to output each character as a byte.

with open("sample.txt", "rb") as file:
 byte = file.read(1)
 while byte:
   print(byte)
   byte = file.read(1)

Output:

Each character is read and printed as a byte from the text file until it reaches the end of the line.

Importance of Bytes and String Conversion

As we know bytes can be understood by computers directly. This helps in saving data on the disk immediately unlike strings that require being encoded and then saved.

One of the most popular applications nowadays of conversion of strings to bytes is in Machine Learning. Machine Learning models are stored in pickle files that are encoded in bytes.

Subscribe to our newsletter