Python Await Async Tutorial with Real Examples and Simple Explanations

by Pubs | Advanced, Management, Web Scraping

Last Updated: June 01, 2026

Advanced

The python await and async is one of the more advanced features to help run your programs faster by making sure the CPU is spending as little time as possible waiting and instead as much time as possible working. If ever you see a capable chef, you’ll know what I mean. The chef is not just following a recipe step by step (i.e. working synchronously), the chef is boiling water to cook the pasta , measuring the amount of pasta, chopping tomatoes for the pasta sauce until the water boils etc (i.e. the chef is working asynchronously). The chef is minimizing the time they are waiting idle and always working on a task. That’s the same idea with async and await.

For this tutorial, we will focus on python 3.7 as it has some of the more modern features of await and async. We will call out some of the differences for python 3.4 – 3.6.

Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

What is async await in Python?

The async await keywords help to define in your program which parts need to run sequentially, and which parts may take sometime but other parts of the program can execute while this step completes. A modern example of this is that if you’re downloading a web page it may take a few seconds, while the download is happening you can execute other parts of your program.

How does async await work in Python?

Sometimes the best way to explain something is to show how you would achieve the same thing without the feature.

Continuing with the restaurant theme, suppose you are running a hamburger stall (you’re the waiter and the chef) and it is almost instant to collect payment for a customer and serve the final hamburger, but the most time consuming task is to cooking the beef patty which takes 2 seconds (one could only wish!).

See the below diagram:

In the above diagram:

• Step 1: you would first get the order and collect the money from Customer 1
• Step 2: you would then put a beef patty on the cook top and then wait for 2 seconds for the beef patty to cook. At the same time, Customer 1 is also waiting for 2 seconds.
• Step 3: when the beef patty is cooked, you can then plate this onto a hamburger bun
• Step 4: pass the final hamburger to Customer 1
• Step 5: You would then start to serve Customer 2 (who has already been waiting 2 seconds for you to serve Customer 1). You can then repeat steps 2-4

With the above approach, Customer 1 would have their burger in about 2 seconds, Customer 2 approx 4 seconds, and then Customer 3 approx 6 seconds.

The equivalent code would be as follows:

import time, datetime, timeit

customer_queue = [ "C1", "C2", "C3" ]

def get_next_customer():
    return customer_queue.pop(0)    #Get the first customer from list

def cook_hamburger(customer):
    start_customer_timer = timeit.default_timer()
    print( f"[{customer}]: Start cooking hamberger for customer")
    time.sleep(2)   # It takes 2 seconds to cook the hamburger
    end_customer_timer = timeit.default_timer()
    print( f"[{customer}]: Finish cooking hamberger for customer.  Total {end_customer_timer-start_customer_timer} seconds\n")

def run_shop():
    while customer_queue:
        curr_customer = get_next_customer()
        cook_hamburger(curr_customer)

def main():
    print('Hamburger Shop')
    start = timeit.default_timer()
    run_shop()
    stop = timeit.default_timer()
    print(f"** Total runtime: {stop-start} seconds ***")

if __name__ == '__main__':
    main()

The code above is fairly straightforward. We have a list of customers that are queuing in the list customer_queue which are being looped under the def run_shop(). For each customer (get_next_customer()), we call cook_hamburger() to cook the hamburger for 2 seconds and wait for it to complete.

Running this code you would get the following output:

As expected, the total runtime for 3 customers is 6 seconds since each customer is served sequentially.

Cooking Hamburgers Asynchronously and coding the event loop manually

Instead of serving the customer and cooking the hamburger for each customer, you can obviously do some of the tasks asynchronously, meaning you can start the task but you don’t have to sit and wait, you can do something else. See the following diagram where the chef/waiter is serving multiple customers and cooking at the same time. It’s not explicitly shown here, but the chef/waiter is constantly checking on the status of the next task and if a task doesn’t require his/her attention they’ll move on to the next task. This process of always looking for something to do is the equivalent of the “event loop”. The Event Loop is a programming construct where the logic is to always look for a task to execute and if there’s a task which will take some time it can release control to the next task in the loop.

In the above example, the following is happening:

• Step 1: you would first get the order and collect the money from Customer 1
• Step 2: you would then put a beef patty on the cook top and then let it cook, then immediately move on to the next customer while the patty is cooking.
• Step 3: you would first get the order and collect the money from Customer 2. You would also check if the first beef patty has completed cooking yet.
• Step 4: you would then put another beef patty on the cook top and then let it cook, then immediately move on to the next customer while the patty is cooking.
• …
• Step 5: When any of the beef patties are done, you would plate it
• Step 6: Pass the plated hamburger to the respective customer. Note, in the above example we’ve assumed it to be Customer 1, but it could be any customer depending on which beef patty cooked fully first.
• Step 7: When any of the beef patties are done, you would plate it, and server

This is the equivalent of the event loop. The chef/waiter is constantly checking if it needs to serve the customer or check on the hamburgers which are cooking. When there’s a hamburger is placed on the stove and we need to wait 2 seconds, the chef/waiter moves to the next task and does not wait for the 2 seconds to complete. When the hamburger is done, it is then served to the customer.

How can this be done programatically? Glad you asked:

import time ,datetime, timeit

customer_queue = [ "C1", "C2", "C3" ]
hamburger_queue = []

def get_next_customer():
    if customer_queue: return customer_queue.pop(0)    #Get the first customer from list
    return None 

def start_cooking_hamburger(customer):
    print( f"[{customer}]: Start cooking hamberger for customer")
    hamburger = { "customer":customer, "start_cooking_time": timeit.default_timer(), "cooked":False}
    hamburger_queue.append( hamburger )

def check_hamburger_status():
    curr_timer = timeit.default_timer()

    #Check if it's cooking, but release control
    for index, hamburger in enumerate(hamburger_queue):         
        elapsed_time = curr_timer-hamburger['start_cooking_time']
        if elapsed_time > 2: #2 second has passed for hamrburger to cook
            print( f"[{hamburger['customer']}]: Finish cooking hamberger for customer.  Total {elapsed_time} seconds\n")
            del hamburger_queue[ index].  #delete from list to mark as done

def run_shop():
    while customer_queue or hamburger_queue:        #Event loop
        curr_customer = get_next_customer()
        if curr_customer: start_cooking_hamburger(curr_customer)
        check_hamburger_status()

def main():
    print('Hamburger Shop')
    start = timeit.default_timer()
    run_shop()
    stop = timeit.default_timer()
    print(f"** Total runtime: {stop-start} seconds ***")

if __name__ == '__main__':
    main()

The output of the code is as follows:

So there’s a few things happening here:

• There’s a new list called hamburger_queue[] which is keeping track of each hamburger that is being cooked
• The event loop is the while customer_queue or hamburger_queue within the run_shop() function
• We have a new function called start_cooking_hamburger() which helps to keep track of the task to cooking starting. Why is this needed? Well in the past we would simply wait for a given task. Now, since we are doing something else while we wait, we need to remember a few things to come back to the task
• We also have a new function called check_hamburger_status() which checks the status of each hamburger being cooked (i.e. item in hamburger_queue[]), and if it is cooked (i.e. 2 seconds have passed), then it is considered complete

You may notice in the output that Customer 3 was in fact served before Customer 2. This is because that the execution order is not guarantee.

How To Use MarkItDown to Convert Documents to Markdown for LLMs

by Pubs | Jul 25, 2026 | Data Processing, Intermediate

Intermediate

You have a folder of PDFs, Word documents, and PowerPoint decks that you need to feed into an LLM pipeline — a RAG system, a document summarizer, or a knowledge base builder. The problem is that LLMs work best with plain text, and most documents are packed with binary formatting, embedded fonts, and layout metadata that the model cannot interpret. Sending a raw PDF to an LLM is like handing someone a ZIP file and asking them to read it.

Microsoft’s markitdown library solves this by converting dozens of file formats — PDF, DOCX, PPTX, XLSX, HTML, CSV, EPUB, images, and audio — into clean Markdown that any LLM can process. It is a single pip install, works from Python code or the command line, and handles the conversion pipeline so your code does not have to. The Markdown output preserves headings, tables, and code blocks in a structured format that models handle especially well.

This article covers everything you need to get documents into your AI pipelines with MarkItDown. You will learn how to install it, convert individual files and entire directories, work with the Python API, handle different file types, and build a real document-processing pipeline that prepares files for a RAG system. By the end you will have a working utility that accepts any folder of mixed-format documents and outputs a structured Markdown dataset ready for embedding.

MarkItDown: Quick Example

Before diving into the details, here is the fastest path from a file to Markdown text you can feed into an LLM. This example converts an HTML page to Markdown in four lines of Python.

# quick_markitdown.py
from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("https://example.com")
print(result.text_content[:500])

Output:

Example Domain
==============

This domain is for use in illustrative examples in documents. You may use this
domain in literature without prior coordination or asking for permission.

[More information...](https://www.iana.org/domains/reserved)

The MarkItDown() constructor creates a converter instance, and convert() accepts a file path, a URL, or a file-like object. The result is a DocumentConverterResult object — result.text_content holds the clean Markdown string ready to pass into any LLM or text processing pipeline.

The real power emerges when you need to handle PDFs, Office files, and mixed directories at scale. The sections below cover all of that, starting with installation.

What Is MarkItDown and Why Use It?

MarkItDown is an open-source Python library from Microsoft that acts as a universal document-to-Markdown converter. Think of it as a universal adapter — on one end you plug in a document in almost any format, and on the other end you get clean, structured Markdown text. The library was built specifically with LLM use cases in mind: the Markdown output is structured in a way that helps models understand document hierarchy (headings, tables, code blocks) rather than receiving a flat blob of text.

The key difference from alternatives like pypdf2 or python-docx is breadth and consistency. Those libraries require a different API for each format, handle only one format each, and return raw extracted text with no structure. MarkItDown gives you one API that handles all formats and preserves semantic structure as Markdown.

Format	MarkItDown	pypdf2 / python-docx / openpyxl
PDF	Yes	pypdf2 only
DOCX	Yes	python-docx only
PPTX	Yes	python-pptx only
XLSX	Yes	openpyxl only
HTML / URL	Yes	No
EPUB	Yes	No
CSV / JSON	Yes	No
Images (OCR)	Yes (optional)	No
Audio (transcription)	Yes (optional)	No
Unified output format	Markdown	Raw text / format-specific objects

The optional image OCR and audio transcription require extra dependencies (and an LLM API key for the best results), but the core document converters — PDF, Office, HTML — work completely offline with no API key required. That makes MarkItDown an excellent fit for any pipeline where you control the infrastructure.

Installing MarkItDown

MarkItDown is on PyPI. A basic install that handles PDFs, Office documents, and HTML covers most use cases and has minimal dependencies.

# Install the base package
pip install markitdown

# For PDF support (pdfminer.six is included in most installs)
pip install markitdown[pdf]

# For all optional extras (OCR, audio, Azure Document Intelligence)
pip install markitdown[all]

Output:

Successfully installed markitdown-0.1.1 pdfminer.six-20221105 ...

The [pdf] extra pulls in pdfminer.six for text-based PDF extraction. If you need image OCR within PDFs, you will also need an LLM plugin (covered in the advanced section). For most document-to-Markdown pipelines the base install is all you need — it handles DOCX, PPTX, XLSX, HTML, CSV, JSON, and EPUB without any extras.

Converting Files, URLs, and Streams

The MarkItDown class is the main entry point. It auto-detects the file format based on extension or MIME type, so you call the same method regardless of what you are converting.

Converting a Local File

Pass any local file path as a string. MarkItDown detects the format automatically and returns a DocumentConverterResult.

# convert_file.py
from markitdown import MarkItDown

md = MarkItDown()

# Convert a DOCX file
result = md.convert("report.docx")
print(f"Title: {result.title}")
print(f"Characters: {len(result.text_content)}")
print()
print(result.text_content[:300])

Output:

Title: Q3 Sales Report
Characters: 4821

# Q3 Sales Report

**Prepared by:** Finance Team
**Date:** September 30, 2025

## Executive Summary

Revenue grew 12% year-over-year driven by the enterprise segment...

The result.title property extracts the document title from metadata where available (DOCX, PDF). The text_content property always holds the full Markdown string. Notice that the headings from the Word document are preserved as Markdown heading levels — this is exactly the structural information that LLMs use to understand document hierarchy.

Converting a URL

Pass a URL string and MarkItDown fetches the page and converts the HTML to Markdown. This works especially well for documentation pages, Wikipedia articles, and any page with semantic HTML structure.

# convert_url.py
from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("https://docs.python.org/3/library/json.html")

print(result.text_content[:600])

Output:

json --- JSON encoder and decoder
=================================

**Source code:** [Lib/json/__init__.py](https://github.com/python/cpython/tree/3.13/Lib/json/__init__.py)

JSON (JavaScript Object Notation), specified by [RFC 7159](https://datatracker.ietf.org/doc/html/rfc7159.html)...

## json.dumps(obj, *, skipkeys=False, ...)

Serialize obj to a JSON formatted str...

Navigation menus, footers, and sidebars are typically stripped out, leaving the actual content. This makes URL conversion a fast way to pull documentation into a RAG corpus without building a custom scraper for every site.

Converting from a File-Like Object

When you receive a file as bytes (from an API response, an email attachment, or a web upload), you can convert it directly from a BytesIO object without writing it to disk first.

# convert_stream.py
import io
from markitdown import MarkItDown
import requests

md = MarkItDown()

# Download a PDF into memory and convert without saving to disk
response = requests.get("https://www.w3.org/WAI/WCAG21/wcag21.pdf")
pdf_stream = io.BytesIO(response.content)

# Must pass extension hint when using streams -- no filename to detect from
result = md.convert(pdf_stream, file_extension=".pdf")
print(f"Converted {len(result.text_content)} characters from in-memory PDF")
print(result.text_content[:400])

Output:

Converted 87432 characters from in-memory PDF

Web Content Accessibility Guidelines (WCAG) 2.1
================================================

W3C Recommendation 05 June 2018

Abstract
--------

Web Content Accessibility Guidelines (WCAG) 2.1 covers a wide range of...

The file_extension parameter is required when passing a stream because there is no filename to inspect. Always include it for streams to ensure the correct converter is selected.

Format-Specific Conversion Options

Most formats work out of the box, but a few have options worth knowing about. PowerPoint files get per-slide conversion, Excel files convert to Markdown tables, and PDFs expose page-level control.

Excel and CSV to Markdown Tables

Excel sheets are converted to Markdown tables — one table per worksheet. This is particularly useful when feeding structured data to LLMs that need to reason about tabular information.

# convert_excel.py
from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("sales_data.xlsx")
print(result.text_content)

Output (example with a two-sheet workbook):

## Sheet1

| Month | Revenue | Units |
|-------|---------|-------|
| Jan   | 84000   | 420   |
| Feb   | 91200   | 456   |
| Mar   | 78500   | 392   |

## Sheet2

| Region | Manager | Target |
|--------|---------|--------|
| East   | Alice   | 100000 |
| West   | Bob     | 95000  |

Each worksheet becomes an H2 section followed by a Markdown table. LLMs handle this format well for question-answering tasks — the model can identify column headers, filter by region, and compute simple aggregations when the data is in this structured Markdown table format rather than raw CSV text.

PowerPoint Slide Extraction

PPTX files are converted slide by slide. Each slide becomes a section with its title as a heading and the body text extracted below it. Speaker notes are included when present.

# convert_pptx.py
from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("product_roadmap.pptx")

# Show first 500 chars to see the structure
print(result.text_content[:500])

Output:

## Slide 1: Product Roadmap 2026

Vision: Ship the AI-native workflow layer by Q3.

- Q1: Core infrastructure and auth
- Q2: Integration layer + partner APIs
- Q3: Public launch
- Q4: Enterprise tier

**Notes:** Emphasize the partnership angle -- this is key differentiator vs competitors.

## Slide 2: Q1 Milestones

- Authentication service -- 95% complete
- Database migration -- in progress
...

Speaker notes appear as bold “Notes:” blocks after each slide’s content. For meeting-notes pipelines or sales-deck summarizers this is especially useful — you get the full context that the presenter intended to convey, not just the bullet points on the slide.

Batch Converting a Directory

Real pipelines rarely deal with a single file. Here is a pattern to convert every supported file in a directory, log any failures, and save all results to a single Markdown file for easy embedding.

# batch_convert.py
import os
from pathlib import Path
from markitdown import MarkItDown

SUPPORTED = {".pdf", ".docx", ".pptx", ".xlsx", ".html", ".csv", ".json", ".epub"}

def batch_convert(input_dir: str, output_path: str) -> dict:
    """Convert all supported files in input_dir to a single Markdown file."""
    md = MarkItDown()
    results = {"success": [], "failed": []}
    output_parts = []

    for fpath in sorted(Path(input_dir).rglob("*")):
        if fpath.suffix.lower() not in SUPPORTED:
            continue

        print(f"Converting: {fpath.name}")
        try:
            result = md.convert(str(fpath))
            # Separate each document with a clear section header
            section = f"\n\n---\n## Document: {fpath.name}\n\n{result.text_content}"
            output_parts.append(section)
            results["success"].append(str(fpath))
        except Exception as exc:
            print(f"  FAILED: {exc}")
            results["failed"].append({"file": str(fpath), "error": str(exc)})

    with open(output_path, "w", encoding="utf-8") as f:
        f.write("# Document Corpus\n")
        f.write(f"Converted {len(results['success'])} documents.\n")
        f.write("".join(output_parts))

    return results


if __name__ == "__main__":
    stats = batch_convert("./docs", "corpus.md")
    print(f"\nDone: {len(stats['success'])} converted, {len(stats['failed'])} failed")

Output:

Converting: annual_report.pdf
Converting: onboarding.docx
Converting: roadmap.pptx
Converting: q3_data.xlsx
  FAILED: File appears to be encrypted

Done: 3 converted, 1 failed

The try/except around each conversion is essential — encrypted PDFs, corrupted files, or unsupported subtypes will raise exceptions rather than silently producing empty output. The pattern above logs the failure and continues with the rest of the batch, which is the right behavior for any automated pipeline. The output corpus.md contains all documents separated by clear section dividers that chunk-based RAG systems can split on.

Using MarkItDown with LLM Pipelines

The most common use case is feeding converted documents into an LLM. Here is how to wire MarkItDown into an OpenAI-compatible pipeline for a simple document Q&A pattern.

# doc_qa.py
from markitdown import MarkItDown
from openai import OpenAI

def answer_from_document(file_path: str, question: str, api_key: str) -> str:
    """Convert a document to Markdown and answer a question about it."""
    md = MarkItDown()
    result = md.convert(file_path)
    doc_text = result.text_content

    # Truncate if the document exceeds a safe context window size
    max_chars = 80000  # ~20K tokens for most models
    if len(doc_text) > max_chars:
        doc_text = doc_text[:max_chars] + "\n\n[Document truncated...]"

    client = OpenAI(api_key=api_key)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": "You are a document analyst. Answer questions based only on the provided document.",
            },
            {
                "role": "user",
                "content": f"Document:\n\n{doc_text}\n\nQuestion: {question}",
            },
        ],
    )
    return response.choices[0].message.content


# Example usage
answer = answer_from_document(
    "contract.pdf",
    "What is the termination clause?",
    api_key="your-openai-api-key"
)
print(answer)

Output:

Section 12.3 of the contract states that either party may terminate this agreement
with 30 days written notice. Termination for cause requires only 5 business days
notice and must be accompanied by written documentation of the breach.

This pattern works with any OpenAI-compatible API — swap the base URL and API key for Anthropic, Gemini, or a local Ollama endpoint and the pattern is identical. The key insight is that MarkItDown’s Markdown output is substantially better context than raw extracted text because the model can use heading levels and table structure to locate specific sections of long documents.

Real-Life Example: Building a Document Corpus Preparer for RAG

This project ties together everything from this article. It accepts a folder of mixed-format documents, converts all of them to Markdown, chunks the output into LLM-ready segments, and saves a JSON file that a vector database like Chroma or Pinecone can ingest directly.

# rag_preparer.py
import json
import os
from pathlib import Path
from markitdown import MarkItDown

SUPPORTED = {".pdf", ".docx", ".pptx", ".xlsx", ".html", ".csv", ".epub"}
CHUNK_SIZE = 1500   # characters per chunk (~375 tokens)
CHUNK_OVERLAP = 150 # overlap so context is not lost at boundaries


def chunk_text(text: str, size: int = CHUNK_SIZE, overlap: int = CHUNK_OVERLAP) -> list:
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + size
        chunk = text[start:end]
        # Try to end on a paragraph boundary
        boundary = chunk.rfind("\n\n")
        if boundary > size // 2:
            chunk = chunk[:boundary]
        chunks.append(chunk.strip())
        start += len(chunk) - overlap
    return [c for c in chunks if len(c) > 50]  # drop tiny trailing chunks


def prepare_rag_corpus(docs_dir: str, output_path: str) -> None:
    md = MarkItDown()
    corpus = []
    chunk_id = 0

    for fpath in sorted(Path(docs_dir).rglob("*")):
        if fpath.suffix.lower() not in SUPPORTED:
            continue

        print(f"Processing: {fpath.name}")
        try:
            result = md.convert(str(fpath))
            chunks = chunk_text(result.text_content)
            for i, chunk in enumerate(chunks):
                corpus.append({
                    "id": f"chunk_{chunk_id:05d}",
                    "source_file": fpath.name,
                    "source_title": result.title or fpath.stem,
                    "chunk_index": i,
                    "total_chunks": len(chunks),
                    "text": chunk,
                })
                chunk_id += 1
            print(f"  -> {len(chunks)} chunks")
        except Exception as exc:
            print(f"  FAILED: {exc}")

    with open(output_path, "w", encoding="utf-8") as f:
        json.dump(corpus, f, indent=2, ensure_ascii=False)

    print(f"\nCorpus saved: {len(corpus)} chunks from {chunk_id} total across all docs")
    print(f"Output: {output_path}")


if __name__ == "__main__":
    prepare_rag_corpus("./knowledge_base", "rag_corpus.json")

Output:

Processing: company_handbook.pdf
  -> 47 chunks
Processing: product_specs.docx
  -> 18 chunks
Processing: roadmap_2026.pptx
  -> 9 chunks
Processing: pricing.xlsx
  -> 3 chunks

Corpus saved: 77 chunks from 77 total across all docs
Output: rag_corpus.json

The output rag_corpus.json is ready to embed. Each chunk carries its source file and title as metadata, so your vector database can filter by document when answering queries that should be scoped to a specific source. The overlapping chunks prevent context loss at boundaries — a sentence that straddles a chunk boundary will appear in both adjacent chunks, so the model always sees complete context around any retrieved passage. To extend this project, add an embedding step using sentence-transformers or the OpenAI Embeddings API and pipe the corpus directly into your vector store of choice.

Frequently Asked Questions

Can MarkItDown convert password-protected PDFs?

No — password-protected PDFs raise an exception during conversion because the underlying pdfminer.six library cannot read encrypted content without the password. You will see a PDFPasswordIncorrect or PDFEncryptionError. The fix is to pre-decrypt the PDF before conversion using a library like pikepdf: open with the password and save a decrypted copy, then convert the decrypted file. Always wrap conversion in try/except in batch pipelines so a single locked file does not abort the entire run.

Does MarkItDown extract images from PDFs?

With the base install, MarkItDown extracts text from PDFs but skips embedded images. If your PDFs are scanned documents (images of text, not actual text layers), the base converter will return little or no content. For scanned PDFs you need to add an LLM plugin — MarkItDown supports Azure Document Intelligence and OpenAI Vision as optional backends that can OCR images within documents. Pass an llm_client and llm_model to the MarkItDown constructor to enable this. The optional dependency install is pip install markitdown[all].

How should I handle very large documents?

For documents that produce more than 100,000 characters of Markdown, pass the whole text to an LLM in a single call only if your model supports a large context window (128K+). Otherwise, use the chunking pattern from the real-life example above — split the Markdown at paragraph boundaries with a 10% overlap and embed the chunks rather than the whole document. MarkItDown itself handles arbitrarily large files without issue; the bottleneck is always the downstream LLM context window, not the conversion step.

How well does MarkItDown handle complex tables?

Simple flat tables — with consistent column counts and no merged cells — convert reliably to Markdown tables. Complex tables with merged headers, multi-row cells, or heavily styled cells may produce irregular Markdown. Excel files generally convert better than Word or PDF tables because the underlying data is already structured. If you find table output unreliable for a specific document type, consider using openpyxl or python-docx for that format and MarkItDown for everything else in the same pipeline.

Can I stream the conversion output?

MarkItDown does not support streaming output — the convert() call blocks until the full document is converted and returns the complete result. For large documents this can take several seconds, especially for multi-hundred-page PDFs. If you need non-blocking behavior in an async pipeline, wrap the conversion in asyncio.run_in_executor() to run it in a thread pool and await the result without blocking your event loop.

Can I add a custom converter for a format MarkItDown does not support?

Yes — MarkItDown has a plugin system. You can register a custom DocumentConverter subclass for any file extension. Implement the convert() method that accepts a file path and returns a DocumentConverterResult, then register it via md.register_converter(MyConverter()). This is useful for proprietary formats specific to your industry — medical records in HL7, CAD files, or internal XML schemas — where you have the parsing logic but need to plug it into the same batch pipeline that handles standard formats.

Conclusion

MarkItDown removes the most painful part of building LLM document pipelines: writing and maintaining a different parser for every file format. A single MarkItDown().convert() call handles PDFs, DOCX, PPTX, XLSX, HTML, CSV, EPUB, and more, returning clean Markdown that LLMs can reason over rather than raw binary or poorly-structured text. The real-life example above gives you a complete RAG corpus preparer that you can extend — add an embedding step, plug in a vector store, or swap in a different chunking strategy depending on your document types.

The next step is to test it on your actual document set. Run the batch converter, inspect the output Markdown for any formatting issues specific to your files, and tune the chunk size to match your embedding model’s token limit. For OCR-heavy or audio transcription pipelines, explore the optional llm_client parameter to unlock MarkItDown’s full feature set.

Official documentation and source: github.com/microsoft/markitdown. PyPI package: pypi.org/project/markitdown/.

Related Articles

• How To Use Narwhals for DataFrame-Agnostic Python Code
• How To Use Python Box for Dot-Notation Dict Access
• How To Use PyArrow for Parquet File Processing in Python
• How To Use Python Dask for Parallel Data Processing
• How To Use Python more-itertools for Advanced Iteration
• How To Use Python DuckDB for Analytical SQL

« Older Entries

Next Entries »

Async Await Code Example in Python

In the previous section we created an asynchronous version manually. Here’s the same outcome but written with the async await syntax. As you’ll notice it is very similar to the original synchronous version:

import time, datetime, time
import asyncio

import time, datetime, timeit

customer_queue = [ "C1", "C2", "C3"  ]

def get_next_customer():
    return customer_queue.pop(0)    #Get the first customer from list

async def cook_hamburger(customer):     
    start_customer_timer = timeit.default_timer()
    print( f"[{customer}]: Start cooking hamberger for customer")
    await asyncio.sleep(2)   # Sleep but release control
    end_customer_timer = timeit.default_timer()
    print( f"[{customer}]: Finish cooking hamberger for customer.  Total {end_customer_timer-start_customer_timer} seconds\n")

async def run_shop():
    cooking_queue = []

    while customer_queue:
        curr_customer = get_next_customer()
        cooking_queue.append(  cook_hamburger(curr_customer)  )   #this returns a task only

    #cooking_queue[] has all the async tasks
    await asyncio.gather( *cooking_queue )      #Run all in parallel

def main():
    print('Hamburger Shop')
    start = timeit.default_timer()

    asyncio.run( run_shop() )           #Start the event loop

    stop = timeit.default_timer()
    print(f"** Total runtime: {stop-start} seconds ***")

if __name__ == '__main__':
    main()

Output as follows:

Let’s walk through the code:

• Firstly, the async await is available from the library asyncio hence the import asyncio
• There’s funny set of async keywords which precede the def run_shop() and the def cook_hamburger(customer) functions. In addition the run_shop() is no longer called directly, instead it is called with a asyncio.run( run_shop() ) function call. So here’s what is happening:
  • The asyncio.run() function is the trigger for the so-called event loop. It continues to run forever until all the tasks given to it are completed. You must pass it a function with the async def... prefix hence why run_shop() has the async prefix
  • In the async def run_shop() function call, the code iterates while there are customers in the queue to process, and then there’s a call to cook_hamburger(curr_customer) for each customer. A direct call to the customer does not actually call the function but instead creates a task to execute this. That is what the async tells the compiler – that when called directly, return a task.
  • At the end of the function code in def run_shop() there’s a call to function await asyncio.gather( *cooking_queue). There’s a few things going on here:
    • The await keywords indicates that you need wait for the work to complete but python can do something else in the meantime
    • The call to gather() actually executes all the tasks given to it as a parameter collectively as a group and then returns the results sequentially (please note that the order of the tasks being executed may be random)
    • The *customer_queue simply expands the list into a list of parameter items. So for example if customer_queue[] == [ '1', '2', '3'] then the gather( *customer_queue) would be the same as gather( '1', '2', '3').
  • When the await asyncio.gather( *customer_queue ) is called, the await keyword releases control to any activities that are pending and one of them would be to the calls to function cook_hamburger() which was added to the customer_queue list. Hence calls to cook_hamburger() would be triggered.
  • Within cook_hamburger() there is also an await asyncio.sleep(2). This simply waits for 2 seconds, however, it does not force the program to wait for the 2 seconds to complete, instead the await keyword releases python to do something else in the meantime. This is similar to step 3 in Figure 2 where the chef/waiter puts the hamburger on the grill, but then doesn’t wait for the 2 second but instead does something else (i.e. serve the next customer)
• The asyncio.run() are new keywords as part of python 3.7. In older versions of python you may see the following but it is the same as simply running asyncio.run( run_shop() ) :
  • loop = asyncio.get_event_loop()
  • loop.run_until_complete(run_shop())
  • loop.close()
• As you will notice, this is very similar to the synchronous code that covers Figure 1 above. This is the beauty of async/await

So remember, whenever there’s an await then that means python pauses at that point for that task to complete but then also releases python to do something else. That’s how the performance improvement occurs. In this example, the runtime of this is 2 seconds instead of the sequential 6 seconds!

Async Asynchronous Calling Another Async Function Code Example

Suppose you want t also call another async function once your first async function is completed – how do you go about this? Remember the rule, if you want to run something asynchronously, you have to use the await keyword, and that the function you’re calling has to be defined with async def ...

To continue with the restaurant theme, suppose that after the hamburger is cooked you ask an assistant to put the hamburger into a takeaway bag which takes 1 second. This is also another task that you need not ‘block’ and wait for it to complete. Hence, this action can be put into a function which is defined as an async. Here’s what the code can look like:

import time, datetime, time
import asyncio

customer_queue = [ "C1", "C2", "C3" ]

def get_next_customer():
    return customer_queue.pop(0)    #Get the first customer from list

async def cook_hamburger(customer):     
    start_customer_timer = timeit.default_timer()
    print( f"[{customer}]: Start cooking hamberger for customer")
    await asyncio.sleep(2)   # Sleep but release control
    end_customer_timer = timeit.default_timer()
    print( f"[{customer}]: Finish cooking hamberger for customer.  Total {end_customer_timer-start_customer_timer} seconds")
    await put_hamburger_in_takeaway_bag( customer )

async def put_hamburger_in_takeaway_bag( customer):
    start_customer_timer = timeit.default_timer()
    print( f"[{customer}]: Start packing hamberger")
    await asyncio.sleep(1)   # It takes 2 seconds to cook the hamburger
    end_customer_timer = timeit.default_timer()
    print( f"[{customer}]: Finish packing hamberger.  Total {end_customer_timer-start_customer_timer} seconds\n")

async def run_shop():
    cooking_queue = []

    while customer_queue:
        curr_customer = get_next_customer()
        cooking_queue.append( cook_hamburger(curr_customer) )   #Get each of the event loops
    await asyncio.gather( *cooking_queue )      #Run all in parallel

def main():
    print('Hamburger Shop')
    start = timeit.default_timer()
    asyncio.run( run_shop() )           #Start the event loop 
    stop = timeit.default_timer()
    print(f"** Total runtime: {stop-start} seconds ***")

if __name__ == '__main__':
    main()

The output would be:

See how once the hamburger is cooked (e.g. [C1]: Finish cooking hamburger for customer. Total 2.000924572115764 seconds), then immediately afterwards you have the [C1]: Start packing hamburger step but also gets called asynchronously.

Async Await Real World Example With Web Crawler in Python

One difficulty in learning Async / Await is that many examples provided simply provide the asyncio.sleep() as an example which is helpful to understand the concept, but not very helpful when you want to make something more useful. Let’s try a more complex example where you want to get some stock data from finance.yahoo.com and then, for that same stock, you also get the first 3 newspaper articles from news.google.com in the last 24 hours.

Now one thing you will realise is that await only works with functions that are defined as async. So you cannot call any function with await. Why? Well recall that when you call await you are expecting a function to return a task and not actually call the function, hence that function needs to be defined as async in order to tell python that it returns a task to be executed at the next available time.

Let’s see the synchronous version of the code:

import asyncio, requests, timeit
from bs4 import BeautifulSoup
from pygooglenews import GoogleNews

stock_list = [ "TSLA", "AAPL"]

def get_stock_price_data(stock):
    print(f"-- getting stock data for {stock}")
    data = {"stock":stock, "price_open":0, "price_close":0 }
    stock_page = requests.get( 'https://finance.yahoo.com/quote/' + stock, headers={'Cache-Control': 'no-cache',  "Pragma": "no-cache"})

    soup = BeautifulSoup(stock_page.text, 'html.parser')
    #<fin-streamer active="" class="Fw(b) Fz(36px) Mb(-4px) D(ib)" data-field="regularMarketPrice" data-pricehint="2" data-symbol="TSLA" data-test="qsp-price" data-trend="none" value="759.63">759.63</fin-streamer>
    data['price_close'] = soup.find('fin-streamer', attrs={"data-symbol":stock, "data-field":"regularMarketPrice"} ).text

    #<td class="Ta(end) Fw(600) Lh(14px)" data-test="OPEN-value">723.25</td>
    data['price_open'] = soup.find( attrs={"data-test":"OPEN-value"}).text

    return data

def get_recent_news(stock):
    print(f"-- getting news data for {stock}")
    gn = GoogleNews()
    search = gn.search(f"stocks {stock}", when = '24h')
    news = search['entries'][0:3]
    return news

def print_stock_update(stock, data, news):
    print(f"Stock:{ stock }")
    price_change = 0
    if int(float(data['price_open'])) != 0: price_change = round( 100 * ( float( data['price_close'])/float(data['price_open'])-1), 2)
    print(f"Open Price:{data['price_open']} Close Price:{data['price_close']} Change:{price_change}% ")
    print("Latest News:")
    for news_item in news:        
        print( f"{news_item.published}:{news_item.source.title} - {news_item.title}" )
    print("\n")

def process_stocks():
    for stock in stock_list:
        data = get_stock_price_data( stock )
        news=[]
        news = get_recent_news( stock )
        print_stock_update(stock, data, news)

if __name__ == '__main__':
    start_timer = timeit.default_timer()
    process_stocks()
    end_timer = timeit.default_timer()

    print(f"** Total runtime: {end_timer-start_timer} seconds ***")

Output as follows:

So what’s happening here. Well, you are looping through two stocks TSLA and AAPL, and for each stock the following happens sequentially:

• A call to data = get_stock_price_data( stock ) occurs in order to make a call to requests.get( 'https://finance.yahoo.com/quote/' + stock) to get the HTML page for the TSLA stock. Effectively, this page: https://finance.yahoo.com/quote/TSLA
• Next we use BeautifulSoup() in order to find the HTML snippet that contains the stock price data for the opening price and the closing price:

• After the call to yahoo is complete, then there’s a call to news = get_recent_news( stock ) which uses the module pygooglenews to get the latest google news. In fact we have used this function in our previous Twitter Bot article.
• Once this is all done, that output is printed out with the call to print_stock_update(stock, data, news)

Clearly this could be called asynchronously as we are looping each time for each stock, and then also the call to get the stock data is independent to getting the news data. However, one thing has to happen sequentially is the print_stock_update(stock, data, news) which has to wait for both the async calls to complete.

One wait to try is to simply call the website download with:

stock_page = await requests.get( 'https://finance.yahoo.com/quote/' + stock, headers={'Cache-Control': 'no-cache',  "Pragma": "no-cache"})

However, you will get the following error:

The reason is, as you may have guessed, is that the requests.get() is not created with the async def... construct and hence cannot be called asynchronously.

What you can do however is to use another ‘get’ web page module called httpx. This function is defined with async def... and can be called similar to requests. That same line would be re-written as:

import httpx
#....

async def get_stock_price_data(stock):
    print(f"-- stock data:getting stock data for {stock}")
    data = {"stock":stock, "price_open":0, "price_close":0 }

    #*** instead of requests.get('https://finance.yahoo.com/quote/' + stock)) ****
    client = httpx.AsyncClient() 
    stock_page = await client.get( 'https://finance.yahoo.com/quote/' + stock)

    soup = BeautifulSoup(stock_page.text, 'html.parser')
    #<fin-streamer active="" class="Fw(b) Fz(36px) Mb(-4px) D(ib)" data-field="regularMarketPrice" data-pricehint="2" data-symbol="TSLA" data-test="qsp-price" data-trend="none" value="759.63">759.63</fin-streamer>
    data['price_close'] = soup.find('fin-streamer', attrs={"data-symbol":stock, "data-field":"regularMarketPrice"} ).text

    #<td class="Ta(end) Fw(600) Lh(14px)" data-test="OPEN-value">723.25</td>
    data['price_open'] = soup.find( attrs={"data-test":"OPEN-value"}).text
    print(f"-- stock data:done {stock}")
    return data

Ok, that works well. However, but what about the GoogleNews() code. There is no such async version of this function, so how can this be called asynchronously? Well for this, you can actually wrap it around a new thread. A ‘thread’ is way to run a piece of code under the same CPU process but in a parallel. It warrants a whole separate article but for now you can think of it as finding a separate space to execute this independent of the current execution path. However, to execute this in a separate thread, there’s a bit more involved.

The code looks like the following:

### Original Version
def get_recent_news(stock):
    print(f"-- stock news:getting stock data for {stock}")
    gn = GoogleNews()
    search = gn.search(f"stocks {stock}", '24h') #Slow code to run asynchronously
    news = search['entries'][0:3]
    print(f"-- stock news:done {stock}")
    return news

### Asynchronous Version
async def get_recent_news(stock):
    print(f"-- stock news:getting stock data for {stock}")
    gn = GoogleNews()
    search = await asyncio.get_event_loop().run_in_executor( None, gn.search, f"stocks {stock}", '24h')
    news = search['entries'][0:3]
    print(f"-- stock news:done {stock}")
    return news

Here what’s happening is that firstly we are using the await keyword to call the gn.search() function which is now being called through this asyncio.get_event_loop().run_in_executor( .. ) function call. What’s happening here is that we are asking the asyncio module to get access to the event loop (that piece of code that continuously checks for tasks to be done) and then to run in a separate thread. The way it is called is that the parameters must be passed in separate to the function call and hence why the parameters are to be passed in after the function name itself. You will also notice that the whole function can now be defined as async def get_recent_news(stock)

How To Mix Asynchronous And Synchronous Code With Await Async in Python

Now the final problem to be solved is how do we call the two functions of get_stock_price_data( stock ) and get_recent_news(stock) to be run asynchronously, but then wait for both to finish, and THEN run the print. This is where these steps should all be grouped under one function. This is the trick to mix asynchronous and synchronous code.

In order to run a group of tasks in parallel as a group you use asyncio.gather(). However, if you want to execute a synchronous function when ALL tasks that were given to asyncio.gather() is complete, then you should wrap it in another asyncio.gather()

async def process_stock_batch(stock):
    (data, news) = await asyncio.gather( get_stock_price_data( stock ), get_recent_news(stock)  )
    print('-- print:request printing')
    print_stock_update(stock, data, news) 
    print('-- print:done')

async def process_stocks():
    run_stock_list = []
    for stock in stock_list:
        run_stock_list.append(   process_stock_batch(stock) )
    await asyncio.gather( *run_stock_list )

Before we solve it for the real world examples, lets show a simpler example. Suppose we had the following example:

import asyncio, timeit

async def get_web_data_A(index):
    await asyncio.sleep(1)
    print(f"Get Web Data-A[{index}] - sleep 1 second")
        
async def get_web_data_B(index):
    await asyncio.sleep(1)
    print(f"Get Web Data-B[{index}] - sleep 1 second")

async def process(index, start_timer):
    await asyncio.gather( get_web_data_A(index), get_web_data_B(index) )
    print(f"Calculate [{index}] - Elapsed time:[{timeit.default_timer()-start_timer}]")

async def run_all():
    start_timer = timeit.default_timer()
    for index in range(0,2):
        await process(index, start_timer)

if __name__ == '__main__':
    asyncio.run( run_all() )

This has the following output:

What is encouraging with this code, is that even though the call to get_web_data_A() and get_web_data_B() both sleep for 1 second, since they were doing that asynchronously, then the total runtime is still just a little over 1 second. This can be shown by the Calculate [0]... output. However, the problem is that the code still iterates each index sequentially, meaning, that index 0 is processed completely first, and once that’s done, then index 1 is processed. What we want instead is to run all the slow get_web_data_A() and get_web_data_B() first, and then run the code to calculate afterwards. This is where you need to first create the tasks for ALL the iterations, and then call gather() on all the tasks. See the following code:

import asyncio, timeit

async def get_web_data_A(index):
    await asyncio.sleep(1)
    print(f"Get Web Data-A[{index}] - sleep 1 second")
        
async def get_web_data_B(index):
    await asyncio.sleep(1)
    print(f"Get Web Data-B[{index}] - sleep 1 second")

async def process(index, start_timer):
    await asyncio.gather( get_web_data_A(index), get_web_data_B(index) )
    print(f"Calculate [{index}] - Elapsed time:[{timeit.default_timer()-start_timer}]")

async def run_all_2():
    start_timer = timeit.default_timer()
    task_queue = []
    for index in range(0,2):
        task_queue.append( process(index, start_timer) )
    await asyncio.gather( *task_queue )

if __name__ == '__main__':
    asyncio.run( run_all_2() )

Here, in the function async def run_all_2() when we loop, we do not call the blocking code await asyncio.gather... inside the for loop. Instead, we are adding all the tasks to call process(..) into a list called task_queue[], and then at the end of the for loop we are calling await asyncio.gather( *task_queue ) on all tasks in one go. Hence, the output is as follows:

You’ll notice that ALL the get_web_data_A() and get_web_data_B() are being called asynchronously, and then the calculate function is called on all the available data. Hence, the elapsed time for all the iterations is only 1 second, compared to the previous 2 seconds.

So what does this mean for our real world example for getting stock data from Yahoo and then calling Google News asynchronously, and then only printing the data once both are done? Well, the same principle applies. The code is as follows:

import asyncio, httpx, timeit
from bs4 import BeautifulSoup
from pygooglenews import GoogleNews

stock_list = [ "TSLA", "AAPL"]

async def get_stock_price_data(stock):
    print(f"-- stock data:getting stock data for {stock}")
    data = {"stock":stock, "price_open":0, "price_close":0 }

    client = httpx.AsyncClient()
    stock_page = await client.get( 'https://finance.yahoo.com/quote/' + stock)

    soup = BeautifulSoup(stock_page.text, 'html.parser')
    #<fin-streamer active="" class="Fw(b) Fz(36px) Mb(-4px) D(ib)" data-field="regularMarketPrice" data-pricehint="2" data-symbol="TSLA" data-test="qsp-price" data-trend="none" value="759.63">759.63</fin-streamer>
    data['price_close'] = soup.find('fin-streamer', attrs={"data-symbol":stock, "data-field":"regularMarketPrice"} ).text

    #<td class="Ta(end) Fw(600) Lh(14px)" data-test="OPEN-value">723.25</td>
    data['price_open'] = soup.find( attrs={"data-test":"OPEN-value"}).text
    print(f"-- stock data:done {stock}")
    return data

async def get_recent_news(stock):
    print(f"-- stock news:getting stock data for {stock}")
    gn = GoogleNews()
    search = await asyncio.get_event_loop().run_in_executor( None, gn.search, f"stocks {stock}", '24h')
    news = search['entries'][0:3]
    print(f"-- stock news:done {stock}")
    return news

def print_stock_update(stock, data, news):
    print('-- print:starting print')
    print(f"Stock:{ stock }")
    price_change = 0
    if int(float(data['price_open'])) != 0: price_change = round( 100 * ( float( data['price_close'])/float(data['price_open'])-1), 2)
    print(f"Open Price:{data['price_open']} Close Price:{data['price_close']} Change:{price_change}% ")
    print("Latest News:")
    for news_item in news:        
        print( f"{news_item.published}:{news_item.source.title} - {news_item.title}" )

    print("\n")

async def process_stock_batch(stock):
    (data, news) = await asyncio.gather( get_stock_price_data( stock ), get_recent_news(stock)  )
    print('-- print:request printing')
    print_stock_update(stock, data, news) 
    print('-- print:done')

async def process_stocks():
    run_stock_list = []
    for stock in stock_list:
        run_stock_list.append(   process_stock_batch(stock) )
    await asyncio.gather( *run_stock_list )

if __name__ == '__main__':
    start_timer = timeit.default_timer()
    asyncio.run( process_stocks() )
    end_timer = timeit.default_timer()

    print(f"** Total runtime: {end_timer-start_timer} seconds ***")

The key bit of code is in the async def process_stocks() which now iterates over each of the stocks, creates tasks, and then calls await asyncio.gather( *run_stock_list ) on all the stocks in one go, and then in the function process_stock_batch(stock) we have the asynchronous call to (data, news) = await asyncio.gather( get_stock_price_data( stock ), and then the synchronous call to print_stock_update(stock, data, news) once both web data is complete.

Conclusion

The await and async function is an incredibly useful feature of python which takes a bit of getting used to in order to understand the concept, but once you’ve got the hang of it, it can be incredibly useful to get an improve of the performance of your code by leveraging idle time where you are waiting for a task to complete. Remember to be sure about the sequencing and being mindful of whether you care to have a follow-up activity once that task is completed, or you can simply continue to execute.

This not easy to grasp as a beginner, but follow the example code above, and if you get stuck feel free to reach out through our email list below.

How To Use MarkItDown to Convert Documents to Markdown for LLMs

by Pubs | Jul 25, 2026 | Data Processing, Intermediate

Intermediate

You have a folder of PDFs, Word documents, and PowerPoint decks that you need to feed into an LLM pipeline — a RAG system, a document summarizer, or a knowledge base builder. The problem is that LLMs work best with plain text, and most documents are packed with binary formatting, embedded fonts, and layout metadata that the model cannot interpret. Sending a raw PDF to an LLM is like handing someone a ZIP file and asking them to read it.

Microsoft’s markitdown library solves this by converting dozens of file formats — PDF, DOCX, PPTX, XLSX, HTML, CSV, EPUB, images, and audio — into clean Markdown that any LLM can process. It is a single pip install, works from Python code or the command line, and handles the conversion pipeline so your code does not have to. The Markdown output preserves headings, tables, and code blocks in a structured format that models handle especially well.

This article covers everything you need to get documents into your AI pipelines with MarkItDown. You will learn how to install it, convert individual files and entire directories, work with the Python API, handle different file types, and build a real document-processing pipeline that prepares files for a RAG system. By the end you will have a working utility that accepts any folder of mixed-format documents and outputs a structured Markdown dataset ready for embedding.

MarkItDown: Quick Example

Before diving into the details, here is the fastest path from a file to Markdown text you can feed into an LLM. This example converts an HTML page to Markdown in four lines of Python.

# quick_markitdown.py
from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("https://example.com")
print(result.text_content[:500])

Output:

Example Domain
==============

This domain is for use in illustrative examples in documents. You may use this
domain in literature without prior coordination or asking for permission.

[More information...](https://www.iana.org/domains/reserved)

The MarkItDown() constructor creates a converter instance, and convert() accepts a file path, a URL, or a file-like object. The result is a DocumentConverterResult object — result.text_content holds the clean Markdown string ready to pass into any LLM or text processing pipeline.

The real power emerges when you need to handle PDFs, Office files, and mixed directories at scale. The sections below cover all of that, starting with installation.

What Is MarkItDown and Why Use It?

MarkItDown is an open-source Python library from Microsoft that acts as a universal document-to-Markdown converter. Think of it as a universal adapter — on one end you plug in a document in almost any format, and on the other end you get clean, structured Markdown text. The library was built specifically with LLM use cases in mind: the Markdown output is structured in a way that helps models understand document hierarchy (headings, tables, code blocks) rather than receiving a flat blob of text.

The key difference from alternatives like pypdf2 or python-docx is breadth and consistency. Those libraries require a different API for each format, handle only one format each, and return raw extracted text with no structure. MarkItDown gives you one API that handles all formats and preserves semantic structure as Markdown.

Format	MarkItDown	pypdf2 / python-docx / openpyxl
PDF	Yes	pypdf2 only
DOCX	Yes	python-docx only
PPTX	Yes	python-pptx only
XLSX	Yes	openpyxl only
HTML / URL	Yes	No
EPUB	Yes	No
CSV / JSON	Yes	No
Images (OCR)	Yes (optional)	No
Audio (transcription)	Yes (optional)	No
Unified output format	Markdown	Raw text / format-specific objects

The optional image OCR and audio transcription require extra dependencies (and an LLM API key for the best results), but the core document converters — PDF, Office, HTML — work completely offline with no API key required. That makes MarkItDown an excellent fit for any pipeline where you control the infrastructure.

Installing MarkItDown

MarkItDown is on PyPI. A basic install that handles PDFs, Office documents, and HTML covers most use cases and has minimal dependencies.

# Install the base package
pip install markitdown

# For PDF support (pdfminer.six is included in most installs)
pip install markitdown[pdf]

# For all optional extras (OCR, audio, Azure Document Intelligence)
pip install markitdown[all]

Output:

Successfully installed markitdown-0.1.1 pdfminer.six-20221105 ...

The [pdf] extra pulls in pdfminer.six for text-based PDF extraction. If you need image OCR within PDFs, you will also need an LLM plugin (covered in the advanced section). For most document-to-Markdown pipelines the base install is all you need — it handles DOCX, PPTX, XLSX, HTML, CSV, JSON, and EPUB without any extras.

Converting Files, URLs, and Streams

The MarkItDown class is the main entry point. It auto-detects the file format based on extension or MIME type, so you call the same method regardless of what you are converting.

Converting a Local File

Pass any local file path as a string. MarkItDown detects the format automatically and returns a DocumentConverterResult.

# convert_file.py
from markitdown import MarkItDown

md = MarkItDown()

# Convert a DOCX file
result = md.convert("report.docx")
print(f"Title: {result.title}")
print(f"Characters: {len(result.text_content)}")
print()
print(result.text_content[:300])

Output:

Title: Q3 Sales Report
Characters: 4821

# Q3 Sales Report

**Prepared by:** Finance Team
**Date:** September 30, 2025

## Executive Summary

Revenue grew 12% year-over-year driven by the enterprise segment...

The result.title property extracts the document title from metadata where available (DOCX, PDF). The text_content property always holds the full Markdown string. Notice that the headings from the Word document are preserved as Markdown heading levels — this is exactly the structural information that LLMs use to understand document hierarchy.

Converting a URL

Pass a URL string and MarkItDown fetches the page and converts the HTML to Markdown. This works especially well for documentation pages, Wikipedia articles, and any page with semantic HTML structure.

# convert_url.py
from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("https://docs.python.org/3/library/json.html")

print(result.text_content[:600])

Output:

json --- JSON encoder and decoder
=================================

**Source code:** [Lib/json/__init__.py](https://github.com/python/cpython/tree/3.13/Lib/json/__init__.py)

JSON (JavaScript Object Notation), specified by [RFC 7159](https://datatracker.ietf.org/doc/html/rfc7159.html)...

## json.dumps(obj, *, skipkeys=False, ...)

Serialize obj to a JSON formatted str...

Navigation menus, footers, and sidebars are typically stripped out, leaving the actual content. This makes URL conversion a fast way to pull documentation into a RAG corpus without building a custom scraper for every site.

Converting from a File-Like Object

When you receive a file as bytes (from an API response, an email attachment, or a web upload), you can convert it directly from a BytesIO object without writing it to disk first.

# convert_stream.py
import io
from markitdown import MarkItDown
import requests

md = MarkItDown()

# Download a PDF into memory and convert without saving to disk
response = requests.get("https://www.w3.org/WAI/WCAG21/wcag21.pdf")
pdf_stream = io.BytesIO(response.content)

# Must pass extension hint when using streams -- no filename to detect from
result = md.convert(pdf_stream, file_extension=".pdf")
print(f"Converted {len(result.text_content)} characters from in-memory PDF")
print(result.text_content[:400])

Output:

Converted 87432 characters from in-memory PDF

Web Content Accessibility Guidelines (WCAG) 2.1
================================================

W3C Recommendation 05 June 2018

Abstract
--------

Web Content Accessibility Guidelines (WCAG) 2.1 covers a wide range of...

The file_extension parameter is required when passing a stream because there is no filename to inspect. Always include it for streams to ensure the correct converter is selected.

Format-Specific Conversion Options

Most formats work out of the box, but a few have options worth knowing about. PowerPoint files get per-slide conversion, Excel files convert to Markdown tables, and PDFs expose page-level control.

Excel and CSV to Markdown Tables

Excel sheets are converted to Markdown tables — one table per worksheet. This is particularly useful when feeding structured data to LLMs that need to reason about tabular information.

# convert_excel.py
from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("sales_data.xlsx")
print(result.text_content)

Output (example with a two-sheet workbook):

## Sheet1

| Month | Revenue | Units |
|-------|---------|-------|
| Jan   | 84000   | 420   |
| Feb   | 91200   | 456   |
| Mar   | 78500   | 392   |

## Sheet2

| Region | Manager | Target |
|--------|---------|--------|
| East   | Alice   | 100000 |
| West   | Bob     | 95000  |

Each worksheet becomes an H2 section followed by a Markdown table. LLMs handle this format well for question-answering tasks — the model can identify column headers, filter by region, and compute simple aggregations when the data is in this structured Markdown table format rather than raw CSV text.

PowerPoint Slide Extraction

PPTX files are converted slide by slide. Each slide becomes a section with its title as a heading and the body text extracted below it. Speaker notes are included when present.

# convert_pptx.py
from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("product_roadmap.pptx")

# Show first 500 chars to see the structure
print(result.text_content[:500])

Output:

## Slide 1: Product Roadmap 2026

Vision: Ship the AI-native workflow layer by Q3.

- Q1: Core infrastructure and auth
- Q2: Integration layer + partner APIs
- Q3: Public launch
- Q4: Enterprise tier

**Notes:** Emphasize the partnership angle -- this is key differentiator vs competitors.

## Slide 2: Q1 Milestones

- Authentication service -- 95% complete
- Database migration -- in progress
...

Speaker notes appear as bold “Notes:” blocks after each slide’s content. For meeting-notes pipelines or sales-deck summarizers this is especially useful — you get the full context that the presenter intended to convey, not just the bullet points on the slide.

Batch Converting a Directory

Real pipelines rarely deal with a single file. Here is a pattern to convert every supported file in a directory, log any failures, and save all results to a single Markdown file for easy embedding.

# batch_convert.py
import os
from pathlib import Path
from markitdown import MarkItDown

SUPPORTED = {".pdf", ".docx", ".pptx", ".xlsx", ".html", ".csv", ".json", ".epub"}

def batch_convert(input_dir: str, output_path: str) -> dict:
    """Convert all supported files in input_dir to a single Markdown file."""
    md = MarkItDown()
    results = {"success": [], "failed": []}
    output_parts = []

    for fpath in sorted(Path(input_dir).rglob("*")):
        if fpath.suffix.lower() not in SUPPORTED:
            continue

        print(f"Converting: {fpath.name}")
        try:
            result = md.convert(str(fpath))
            # Separate each document with a clear section header
            section = f"\n\n---\n## Document: {fpath.name}\n\n{result.text_content}"
            output_parts.append(section)
            results["success"].append(str(fpath))
        except Exception as exc:
            print(f"  FAILED: {exc}")
            results["failed"].append({"file": str(fpath), "error": str(exc)})

    with open(output_path, "w", encoding="utf-8") as f:
        f.write("# Document Corpus\n")
        f.write(f"Converted {len(results['success'])} documents.\n")
        f.write("".join(output_parts))

    return results


if __name__ == "__main__":
    stats = batch_convert("./docs", "corpus.md")
    print(f"\nDone: {len(stats['success'])} converted, {len(stats['failed'])} failed")

Output:

Converting: annual_report.pdf
Converting: onboarding.docx
Converting: roadmap.pptx
Converting: q3_data.xlsx
  FAILED: File appears to be encrypted

Done: 3 converted, 1 failed

The try/except around each conversion is essential — encrypted PDFs, corrupted files, or unsupported subtypes will raise exceptions rather than silently producing empty output. The pattern above logs the failure and continues with the rest of the batch, which is the right behavior for any automated pipeline. The output corpus.md contains all documents separated by clear section dividers that chunk-based RAG systems can split on.

Using MarkItDown with LLM Pipelines

The most common use case is feeding converted documents into an LLM. Here is how to wire MarkItDown into an OpenAI-compatible pipeline for a simple document Q&A pattern.

# doc_qa.py
from markitdown import MarkItDown
from openai import OpenAI

def answer_from_document(file_path: str, question: str, api_key: str) -> str:
    """Convert a document to Markdown and answer a question about it."""
    md = MarkItDown()
    result = md.convert(file_path)
    doc_text = result.text_content

    # Truncate if the document exceeds a safe context window size
    max_chars = 80000  # ~20K tokens for most models
    if len(doc_text) > max_chars:
        doc_text = doc_text[:max_chars] + "\n\n[Document truncated...]"

    client = OpenAI(api_key=api_key)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": "You are a document analyst. Answer questions based only on the provided document.",
            },
            {
                "role": "user",
                "content": f"Document:\n\n{doc_text}\n\nQuestion: {question}",
            },
        ],
    )
    return response.choices[0].message.content


# Example usage
answer = answer_from_document(
    "contract.pdf",
    "What is the termination clause?",
    api_key="your-openai-api-key"
)
print(answer)

Output:

Section 12.3 of the contract states that either party may terminate this agreement
with 30 days written notice. Termination for cause requires only 5 business days
notice and must be accompanied by written documentation of the breach.

This pattern works with any OpenAI-compatible API — swap the base URL and API key for Anthropic, Gemini, or a local Ollama endpoint and the pattern is identical. The key insight is that MarkItDown’s Markdown output is substantially better context than raw extracted text because the model can use heading levels and table structure to locate specific sections of long documents.

Real-Life Example: Building a Document Corpus Preparer for RAG

This project ties together everything from this article. It accepts a folder of mixed-format documents, converts all of them to Markdown, chunks the output into LLM-ready segments, and saves a JSON file that a vector database like Chroma or Pinecone can ingest directly.

# rag_preparer.py
import json
import os
from pathlib import Path
from markitdown import MarkItDown

SUPPORTED = {".pdf", ".docx", ".pptx", ".xlsx", ".html", ".csv", ".epub"}
CHUNK_SIZE = 1500   # characters per chunk (~375 tokens)
CHUNK_OVERLAP = 150 # overlap so context is not lost at boundaries


def chunk_text(text: str, size: int = CHUNK_SIZE, overlap: int = CHUNK_OVERLAP) -> list:
    """Split text into overlapping chunks."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + size
        chunk = text[start:end]
        # Try to end on a paragraph boundary
        boundary = chunk.rfind("\n\n")
        if boundary > size // 2:
            chunk = chunk[:boundary]
        chunks.append(chunk.strip())
        start += len(chunk) - overlap
    return [c for c in chunks if len(c) > 50]  # drop tiny trailing chunks


def prepare_rag_corpus(docs_dir: str, output_path: str) -> None:
    md = MarkItDown()
    corpus = []
    chunk_id = 0

    for fpath in sorted(Path(docs_dir).rglob("*")):
        if fpath.suffix.lower() not in SUPPORTED:
            continue

        print(f"Processing: {fpath.name}")
        try:
            result = md.convert(str(fpath))
            chunks = chunk_text(result.text_content)
            for i, chunk in enumerate(chunks):
                corpus.append({
                    "id": f"chunk_{chunk_id:05d}",
                    "source_file": fpath.name,
                    "source_title": result.title or fpath.stem,
                    "chunk_index": i,
                    "total_chunks": len(chunks),
                    "text": chunk,
                })
                chunk_id += 1
            print(f"  -> {len(chunks)} chunks")
        except Exception as exc:
            print(f"  FAILED: {exc}")

    with open(output_path, "w", encoding="utf-8") as f:
        json.dump(corpus, f, indent=2, ensure_ascii=False)

    print(f"\nCorpus saved: {len(corpus)} chunks from {chunk_id} total across all docs")
    print(f"Output: {output_path}")


if __name__ == "__main__":
    prepare_rag_corpus("./knowledge_base", "rag_corpus.json")

Output:

Processing: company_handbook.pdf
  -> 47 chunks
Processing: product_specs.docx
  -> 18 chunks
Processing: roadmap_2026.pptx
  -> 9 chunks
Processing: pricing.xlsx
  -> 3 chunks

Corpus saved: 77 chunks from 77 total across all docs
Output: rag_corpus.json

The output rag_corpus.json is ready to embed. Each chunk carries its source file and title as metadata, so your vector database can filter by document when answering queries that should be scoped to a specific source. The overlapping chunks prevent context loss at boundaries — a sentence that straddles a chunk boundary will appear in both adjacent chunks, so the model always sees complete context around any retrieved passage. To extend this project, add an embedding step using sentence-transformers or the OpenAI Embeddings API and pipe the corpus directly into your vector store of choice.

Frequently Asked Questions

Can MarkItDown convert password-protected PDFs?

No — password-protected PDFs raise an exception during conversion because the underlying pdfminer.six library cannot read encrypted content without the password. You will see a PDFPasswordIncorrect or PDFEncryptionError. The fix is to pre-decrypt the PDF before conversion using a library like pikepdf: open with the password and save a decrypted copy, then convert the decrypted file. Always wrap conversion in try/except in batch pipelines so a single locked file does not abort the entire run.

Does MarkItDown extract images from PDFs?

With the base install, MarkItDown extracts text from PDFs but skips embedded images. If your PDFs are scanned documents (images of text, not actual text layers), the base converter will return little or no content. For scanned PDFs you need to add an LLM plugin — MarkItDown supports Azure Document Intelligence and OpenAI Vision as optional backends that can OCR images within documents. Pass an llm_client and llm_model to the MarkItDown constructor to enable this. The optional dependency install is pip install markitdown[all].

How should I handle very large documents?

For documents that produce more than 100,000 characters of Markdown, pass the whole text to an LLM in a single call only if your model supports a large context window (128K+). Otherwise, use the chunking pattern from the real-life example above — split the Markdown at paragraph boundaries with a 10% overlap and embed the chunks rather than the whole document. MarkItDown itself handles arbitrarily large files without issue; the bottleneck is always the downstream LLM context window, not the conversion step.

How well does MarkItDown handle complex tables?

Simple flat tables — with consistent column counts and no merged cells — convert reliably to Markdown tables. Complex tables with merged headers, multi-row cells, or heavily styled cells may produce irregular Markdown. Excel files generally convert better than Word or PDF tables because the underlying data is already structured. If you find table output unreliable for a specific document type, consider using openpyxl or python-docx for that format and MarkItDown for everything else in the same pipeline.

Can I stream the conversion output?

MarkItDown does not support streaming output — the convert() call blocks until the full document is converted and returns the complete result. For large documents this can take several seconds, especially for multi-hundred-page PDFs. If you need non-blocking behavior in an async pipeline, wrap the conversion in asyncio.run_in_executor() to run it in a thread pool and await the result without blocking your event loop.

Can I add a custom converter for a format MarkItDown does not support?

Yes — MarkItDown has a plugin system. You can register a custom DocumentConverter subclass for any file extension. Implement the convert() method that accepts a file path and returns a DocumentConverterResult, then register it via md.register_converter(MyConverter()). This is useful for proprietary formats specific to your industry — medical records in HL7, CAD files, or internal XML schemas — where you have the parsing logic but need to plug it into the same batch pipeline that handles standard formats.

Conclusion

MarkItDown removes the most painful part of building LLM document pipelines: writing and maintaining a different parser for every file format. A single MarkItDown().convert() call handles PDFs, DOCX, PPTX, XLSX, HTML, CSV, EPUB, and more, returning clean Markdown that LLMs can reason over rather than raw binary or poorly-structured text. The real-life example above gives you a complete RAG corpus preparer that you can extend — add an embedding step, plug in a vector store, or swap in a different chunking strategy depending on your document types.

The next step is to test it on your actual document set. Run the batch converter, inspect the output Markdown for any formatting issues specific to your files, and tune the chunk size to match your embedding model’s token limit. For OCR-heavy or audio transcription pipelines, explore the optional llm_client parameter to unlock MarkItDown’s full feature set.

Official documentation and source: github.com/microsoft/markitdown. PyPI package: pypi.org/project/markitdown/.

Related Articles

• How To Use Narwhals for DataFrame-Agnostic Python Code
• How To Use Python Box for Dot-Notation Dict Access
• How To Use PyArrow for Parquet File Processing in Python
• How To Use Python Dask for Parallel Data Processing
• How To Use Python more-itertools for Advanced Iteration
• How To Use Python DuckDB for Analytical SQL

« Older Entries

Next Entries »

Related Articles

Further Reading: For more details, see the Python asyncio documentation.

Frequently Asked Questions

What is async/await in Python?

async def defines a coroutine function and await pauses execution until an asynchronous operation completes. This enables concurrent I/O operations without threading, using the asyncio event loop.

When should I use async/await instead of threading?

Use async/await for I/O-bound tasks like network requests and database queries with many concurrent connections. Use threading for CPU-bound tasks or libraries that do not support async.

How do I run multiple async tasks concurrently?

Use asyncio.gather(task1(), task2()) to run multiple coroutines concurrently. Use asyncio.create_task() to schedule without immediately waiting.

What does ‘coroutine was never awaited’ mean?

You called an async function without await. Async functions return coroutine objects that must be awaited. Add await before the call or use asyncio.run() from synchronous code.

Can I mix synchronous and asynchronous code?

Yes. Use asyncio.run() to call async from sync. Use loop.run_in_executor() to run blocking functions inside async code without blocking the event loop.

Related Articles

• How To Use Python snoop for Function Tracing and Debugging
• How To Use Python responses for Mocking HTTP Requests in Tests
• How To Use Python freezegun for Mocking Time in Tests
• How To Use Python natsort for Natural Sort Order
• How To Use Python pendulum for Better Date and Time Handling
• How To Use Python structlog for Structured Logging
• Plugin Architecture For Your Code Using pyplugs in Python3