Intermediate

Working with binary data is one of those skills that feels intimidating until you understand the two types Python gives you: bytes (immutable) and bytearray (mutable). Whether you are reading a binary file format, sending raw bytes over a network socket, working with image data, or packing integers into a compact binary protocol, these two types are what you reach for. The confusion usually comes from mixing up strings (Unicode text) and bytes (raw binary data) — a distinction Python 3 enforces strictly.

Both bytes and bytearray are built into Python 3 — no imports needed for the basic operations. The struct module (also built-in) handles packing and unpacking binary data formats like those used in network protocols and file headers. The examples below all run with the standard library only.

This article covers: creating bytes and bytearrays, encoding and decoding strings, slicing and searching binary data, modifying bytearrays in place, using the struct module for binary packing, reading binary files, and a real-life binary file parser that reads PNG header metadata.

Bytes and Bytearray: Quick Example

Here is the core distinction in 15 lines — bytes is immutable, bytearray is mutable:

# bytes_quick.py
# bytes: immutable
data = b"Hello, World!"
print(type(data), data)
print("First byte:", data[0])          # Integer (72 = ord('H'))
print("Slice:", data[0:5])             # Still bytes

# bytearray: mutable
buf = bytearray(b"Hello, World!")
buf[0] = 104                           # Change 'H' to 'h' (ASCII 104)
buf.extend(b" Goodbye!")
print("Modified:", buf)
print("As string:", buf.decode("utf-8"))

Output:

<class 'bytes'> b'Hello, World!'
First byte: 72
Slice: b'Hello'
Modified: bytearray(b'hello, World! Goodbye!')
As string: hello, World! Goodbye!

Indexing a bytes or bytearray object with a single integer returns an int (the byte value 0-255), not a single-character string. Slicing returns a new bytes or bytearray object. This is the most common source of confusion for developers coming from Python 2.

Creating bytes and bytearray

There are several ways to create byte sequences in Python, each suited to different situations:

MethodExampleUse case
Literal prefix b""b"hello"Hard-coded ASCII byte strings
str.encode()"hello".encode("utf-8")Converting text to bytes
bytes(n)bytes(10)Zero-filled buffer of n bytes
bytes(iterable)bytes([72, 101, 108])From list of integers (0-255)
bytes.fromhex()bytes.fromhex("48656c6c6f")From hex string
bytearray(source)bytearray(b"hello")Mutable copy of bytes
 
# bytes_creation.py

# From ASCII literal
a = b"Python"
print(a, a.hex())

# From UTF-8 string
b = "Python -- binary".encode("utf-8")
print(b)

# Zero-filled buffer
buf = bytearray(8)
print("Zeroed buffer:", buf)

# From integer list
c = bytes([80, 121, 116, 104, 111, 110])
print("From ints:", c.decode())

# From hex string
d = bytes.fromhex("deadbeef")
print("From hex:", d, d.hex())

Output:

b'Python' 507974686f6e
b'Python \xe2\x80\x94 binary'
Zeroed buffer: bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x00')
From ints: Python
From hex: b'\xde\xad\xbe\xef' deadbeef

Notice how "Python -- binary".encode("utf-8") shows the em-dash encoded as three UTF-8 bytes (\xe2\x80\x94). This is why Python 3 makes the string/bytes distinction explicit — text and binary data have completely different internal representations.

Encoding and Decoding Strings

The most frequent use of bytes is converting between text (strings) and binary data. Always specify the encoding explicitly — never rely on the default, which varies by platform.

# bytes_encoding.py

text = "Cafe -- served with care"

# Encode to bytes
utf8_bytes = text.encode("utf-8")
utf16_bytes = text.encode("utf-16")
ascii_bytes = "Hello".encode("ascii")

print("UTF-8: ", utf8_bytes[:20], "...", len(utf8_bytes), "bytes")
print("UTF-16:", utf16_bytes[:20], "...", len(utf16_bytes), "bytes")

# Decode back to string
decoded = utf8_bytes.decode("utf-8")
print("Decoded:", decoded)

# Handling errors gracefully
mixed = b"Hello \x80 World"  # Invalid UTF-8 byte \x80
safe = mixed.decode("utf-8", errors="replace")
print("With replace:", safe)
strict_safe = mixed.decode("utf-8", errors="ignore")
print("With ignore: ", strict_safe)

Output:

UTF-8:  b'Cafe \xe2\x80\x94 served wi' ... 26 bytes
UTF-16: b'\xff\xfec\x00a\x00f\x00e\x00' ... 52 bytes
Decoded: Cafe -- served with care
With replace: Hello  World
With ignore:  Hello  World

UTF-8 is the right choice for almost everything — it is ASCII-compatible, compact for English text, and the universal standard for the web. UTF-16 uses 2 bytes minimum per character and adds a byte-order mark. The errors parameter to .decode() controls what happens with bytes that are not valid in the given encoding: "replace" substitutes the Unicode replacement character, "ignore" silently drops the bad byte, and "strict" (default) raises a UnicodeDecodeError.

Searching and Slicing Binary Data

Because bytes supports the same sequence operations as str, you can search, slice, split, and join binary data with familiar method names.

# bytes_searching.py

data = b"Content-Type: application/json\r\nContent-Length: 42\r\n\r\n{}"

# Find the header separator
sep_pos = data.find(b"\r\n\r\n")
headers_raw = data[:sep_pos]
body = data[sep_pos + 4:]

print("Headers block:", headers_raw.decode())
print("Body:", body.decode())

# Split headers into individual lines
for line in headers_raw.split(b"\r\n"):
    if b":" in line:
        key, value = line.split(b": ", 1)
        print(f"  {key.decode()}: {value.decode()}")

# Check content type
if data.startswith(b"Content"):
    print("Starts with Content -- yes")

print("Position of 'json':", data.find(b"json"))

Output:

Headers block: Content-Type: application/json
Content-Length: 42
Body: {}
  Content-Type: application/json
  Content-Length: 42
Starts with Content -- yes
Position of 'json': 24

This pattern — splitting HTTP headers on \r\n and splitting each header on ": " — is essentially what Python’s http module does internally. Working directly with bytes is necessary here because HTTP headers are specified as ASCII bytes, not Unicode strings.

Packing Binary Data with struct

The struct module converts between Python values and packed binary data using C-style format strings. This is how you read binary file formats, network protocol headers, and hardware data formats.

# bytes_struct.py
import struct

# Pack: Python values -> bytes
packed = struct.pack(">IHH", 1234567, 100, 200)
print("Packed bytes:", packed.hex(), "-- length:", len(packed))

# Unpack: bytes -> Python values
value1, value2, value3 = struct.unpack(">IHH", packed)
print("Unpacked:", value1, value2, value3)

# Format string key: > = big-endian, I = unsigned int (4 bytes), H = unsigned short (2 bytes)
print("\nFormat sizes:")
for fmt, name in [("B", "unsigned byte"), ("H", "unsigned short"), ("I", "unsigned int"), ("Q", "unsigned long long")]:
    print(f"  {fmt} ({name}): {struct.calcsize(fmt)} bytes")

# Pack multiple fields into a fixed-size record
RECORD_FMT = ">HHI"  # port, flags, timestamp
record = struct.pack(RECORD_FMT, 8080, 0b11, 1713484800)
print("\nRecord hex:", record.hex())
port, flags, ts = struct.unpack(RECORD_FMT, record)
print(f"Port: {port}, Flags: {flags:02b}, Timestamp: {ts}")

Output:

Packed bytes: 0012d68700640c8 -- length: 8
Unpacked: 1234567 100 200

Format sizes:
  B (unsigned byte): 1 bytes
  H (unsigned short): 2 bytes
  I (unsigned int): 4 bytes
  Q (unsigned long long): 8 bytes

Record hex: 1f9000021ef34c00
Port: 8080, Flags: 11, Timestamp: 1713484800

The format string prefix controls byte order: > means big-endian (network byte order, most significant byte first), < means little-endian, and = uses the native byte order of the current machine. Always specify byte order explicitly when interoperating with external systems — never rely on native order for data you will send over a network or save to a file.

Reading and Writing Binary Files

Open files in binary mode with "rb" or "wb" to read and write raw bytes. This is essential for images, PDFs, audio files, and any non-text format.

# bytes_binary_file.py

# Write binary data
with open("data.bin", "wb") as f:
    # Write a simple custom binary format: 4-byte magic + 2-byte version + data
    header = struct.pack(">4sH", b"PYDT", 1)
    payload = b"Sample binary payload data\x00" * 3
    f.write(header)
    f.write(struct.pack(">I", len(payload)))  # payload length
    f.write(payload)

import struct  # ensure import at top in real scripts

# Read it back
with open("data.bin", "rb") as f:
    magic, version = struct.unpack(">4sH", f.read(6))
    payload_len = struct.unpack(">I", f.read(4))[0]
    payload = f.read(payload_len)

print("Magic:", magic.decode())
print("Version:", version)
print("Payload length:", payload_len)
print("Payload preview:", payload[:27])

Output:

Magic: PYDT
Version: 1
Payload length: 81
Payload preview: b'Sample binary payload data\x00'

Always open binary files with "rb"/"wb" — never with just "r"/"w". On Windows, text mode performs newline translation (\r\n becomes \n), which silently corrupts binary files. The pattern of writing a magic number at the start of a file is used by virtually every binary format — PNG starts with \x89PNG\r\n\x1a\n, PDF starts with %PDF, ZIP starts with PK.

Real-Life Example: PNG Header Parser

PNG files start with a fixed 8-byte signature followed by mandatory chunks. This parser reads the PNG signature and the IHDR chunk (which contains image width, height, and color mode) without loading the entire image into memory.

# png_header_parser.py
import struct
import urllib.request

def parse_png_header(data: bytes) -> dict:
    """Parse PNG signature and IHDR chunk from raw bytes."""
    PNG_SIGNATURE = b"\x89PNG\r\n\x1a\n"

    if data[:8] != PNG_SIGNATURE:
        raise ValueError("Not a PNG file -- magic bytes do not match")

    # IHDR chunk starts at byte 8
    # Format: 4-byte length + 4-byte type + data + 4-byte CRC
    chunk_len = struct.unpack(">I", data[8:12])[0]
    chunk_type = data[12:16]

    if chunk_type != b"IHDR":
        raise ValueError("Expected IHDR chunk")

    # IHDR data: width(4) + height(4) + bit_depth(1) + color_type(1) + ...
    ihdr = data[16:16 + chunk_len]
    width, height, bit_depth, color_type = struct.unpack(">IIBB", ihdr[:10])

    color_modes = {0: "Grayscale", 2: "RGB", 3: "Indexed", 4: "Grayscale+Alpha", 6: "RGBA"}
    return {
        "width": width,
        "height": height,
        "bit_depth": bit_depth,
        "color_mode": color_modes.get(color_type, "Unknown"),
        "raw_signature": data[:8].hex(),
    }

# Fetch a real PNG image (Python logo)
url = "https://www.python.org/static/img/python-logo.png"
with urllib.request.urlopen(url) as resp:
    raw = resp.read(200)  # Only need the first 200 bytes

info = parse_png_header(raw)
for key, val in info.items():
    print(f"{key:20s}: {val}")

Output:

width               : 290
height              : 82
bit_depth           : 8
color_mode          : RGBA
raw_signature       : 89504e470d0a1a0a

We only read the first 200 bytes of the file — enough to get the header without downloading the full image. The IHDR chunk always appears first in any valid PNG file, making this approach reliable. This same technique works for any binary format with a fixed header structure: BMP files, WAV audio, ZIP archives, ELF executables.

Frequently Asked Questions

What is the difference between bytes and str in Python 3?

str is a sequence of Unicode code points — it represents text and knows nothing about how that text is stored on disk or sent over a network. bytes is a sequence of integers (0-255) — it represents raw binary data. Converting between them requires an explicit encoding (UTF-8, ASCII, Latin-1, etc.). You cannot concatenate a str and a bytes object; Python will raise a TypeError. This strict separation prevents the encoding bugs that were common in Python 2.

When should I use bytearray instead of bytes?

Use bytearray when you need to modify the data in place — for example, building a binary packet by writing fields one by one, or patching specific bytes in a buffer. Use bytes when the data should be immutable — for function arguments, dictionary keys, and data you will not modify. bytearray is also useful for large binary buffers where creating a new bytes object for every modification would be wasteful.

How do I print bytes as hexadecimal?

Use data.hex() to get a lowercase hex string, or data.hex(" ") (Python 3.8+) to insert a space between each byte for readability. To go the other way, call bytes.fromhex("deadbeef"). For debugging, repr(data) shows the b"..." form with \xNN escapes for non-printable bytes. Use int.from_bytes(data, byteorder="big") to convert a byte sequence to a Python integer.

What is memoryview and when do I need it?

A memoryview provides a view into the underlying buffer of a bytes or bytearray object without copying data. Slicing a memoryview returns another memoryview pointing into the same memory — very efficient for large buffers where you want to process different sections without allocating new objects. Use it when you are working with large binary streams (network buffers, numpy arrays, audio data) and performance matters. For most everyday work, slicing bytes directly is fine.

What are the most useful struct format characters?

The most commonly used format characters are: B (unsigned byte, 1 byte), H (unsigned short, 2 bytes), I (unsigned int, 4 bytes), Q (unsigned long long, 8 bytes), f (float, 4 bytes), d (double, 8 bytes), s (char array, used as Ns for N bytes), and x (padding byte). Prefix with > for big-endian (network order) or < for little-endian (most Intel/x86 data). Use struct.calcsize(fmt) to check the byte size of a format string before use.

Conclusion

Python’s bytes and bytearray types give you precise control over binary data. You have covered the full toolkit: creating byte sequences from literals, strings, integers, and hex values; encoding and decoding with explicit error handling; slicing and searching binary buffers; using the struct module to pack and unpack binary formats; reading binary files; and the PNG header parser that shows these concepts working together in a real format.

A great next project is to extend the PNG parser to scan all chunks in a PNG file (not just IHDR) and print a manifest of chunk types and sizes — PNG files can contain text metadata (tEXt chunks), color profiles (iCCP), and transparency data (tRNS) beyond the image pixels. That project will give you practice with the chunk-based parsing pattern used by many binary formats.

For the complete API reference, see the Python binary sequence types documentation and the struct module documentation.