Intermediate
You have a folder full of PDFs — technical manuals, scanned invoices, research papers with charts and diagrams — and you need your AI application to actually understand them. Not just the text layer, but the tables, the figures, the visual layout that gives the numbers meaning. Traditional RAG pipelines let you down here: they strip PDFs to plain text, lose every diagram, mangle every table, and then wonder why the AI confidently returns wrong answers. If you have hit this wall while building a document Q&A system, Morphik was built to solve exactly that problem.
Morphik is an open-source multimodal retrieval engine with a Python SDK that lets you ingest PDFs, images, and text documents into a unified knowledge store and then query them with natural language. Under the hood it uses ColPali — a technique that embeds each page of a document as an image, preserving visual context — so charts and schematics are searchable just like prose. The free tier at dev.morphik.ai gets you started with no infrastructure to manage. You install the SDK, grab your connection URI, and you are ingesting documents in three lines of Python.
In this article we will cover how to install the Morphik Python SDK, how to connect to the hosted service, how to ingest text and PDF files, how to run semantic search with retrieve_chunks(), how to use RAG queries with query(), how to organise documents into folders, and how to use the async client for production workloads. By the end you will have a working multimodal document search system you can drop into any Python project.
Morphik in Python: Quick Example
Here is the minimum viable Morphik workflow — ingest a piece of text and query it back with a natural language question — so you can see the shape of the API before we go deeper:
# quick_morphik.py
from morphik import Morphik
# Connect to Morphik (replace with your URI from dev.morphik.ai)
db = Morphik("morphik://owner_id:your_token@api.morphik.ai")
# Ingest a text document with metadata
doc = db.ingest_text(
content="Python 3.12 introduced the experimental JIT compiler. "
"Enable it with PYTHON_JIT=1 before running your script. "
"Numeric loops and tight iteration show the biggest gains.",
metadata={"topic": "python", "version": "3.12"}
)
print("Ingested doc ID:", doc.external_id)
# Query with RAG -- Morphik retrieves relevant chunks and generates an answer
response = db.query(
query="How do I enable the Python JIT compiler?",
filters={"topic": "python"}
)
print(response.completion)
Output:
Ingested doc ID: doc_a3f9c1b2e7d04512
To enable the Python JIT compiler, set the environment variable PYTHON_JIT=1
before running your script. This feature was introduced in Python 3.12 and
provides the most noticeable speedups for numeric loops and tight iteration.
The ingest_text() call stores the content in Morphik’s vector store with the metadata you supply. The query() call does retrieval and generation in one step — it finds the most relevant chunks, passes them to an LLM, and returns the synthesized answer in response.completion. The filters argument narrows the search to documents whose metadata matches, so your queries stay scoped to the right corpus.
The sections below show how to ingest PDFs, work with folders, run raw chunk retrieval, and use the async client for concurrent workloads.
What Is Morphik and How Does It Work?
Morphik Core is a multimodal RAG engine — a system that stores, indexes, and retrieves documents of mixed content types (text, images, PDFs with visual elements) so that an LLM can answer questions over them accurately. The key differentiator is how it indexes PDFs: instead of running OCR and discarding the visual structure, Morphik embeds each page as an image using a technique called ColPali. This means a flowchart, a table with shaded header rows, or an engineering schematic is represented in the index as faithfully as a paragraph of prose.
Think of traditional RAG as a librarian who can only read books if someone first types out all the text by hand, throwing away any photos or tables. Morphik is more like a librarian who photographs each page and indexes the photograph — so when you ask “what does the diagram on page 14 show?”, the system actually has that diagram in its index.
| Feature | Traditional RAG | Morphik |
|---|---|---|
| PDF ingestion | OCR to plain text | Page-level image embeddings (ColPali) |
| Diagrams and charts | Lost or garbled | Indexed and searchable |
| Tables | Mangled to strings | Preserved visually |
| Metadata filtering | Varies by framework | Built-in on every call |
| Multi-tenancy | DIY | Built-in (folders + scoping) |
| MCP support | No | Yes (Claude, Cursor) |
| Self-hostable | Depends on stack | Yes (Docker or direct install) |
You interact with Morphik through the Python SDK (pip install morphik) or the REST API. The SDK wraps the API into a clean synchronous or async client. Documents you ingest are stored server-side; your queries hit that stored index. Let us install everything and get connected.
Installing Morphik and Getting a Connection URI
Install the Morphik Python SDK with pip. It has no mandatory dependency on pandas, numpy, or any heavy ML library — the heavy lifting happens on the server:
# terminal
pip install morphik
Next, sign up for a free account at dev.morphik.ai/signup. After signing in, navigate to your dashboard and copy your connection URI — it looks like this:
morphik://owner_id:your_token@api.morphik.ai
This single URI contains your owner ID, authentication token, and the API endpoint. Pass it to the Morphik constructor and you are connected. If you are self-hosting Morphik on your own infrastructure, use a plain HTTP(S) base URL instead:
# connect_morphik.py
from morphik import Morphik
# Hosted service (replace with your real URI)
db = Morphik("morphik://owner_id:your_token@api.morphik.ai")
# Self-hosted (adjust host and port as needed)
# db = Morphik("http://localhost:8000")
# Verify the connection by listing documents
docs = db.list_documents()
print(f"Connected. Documents in store: {len(docs.documents)}")
Output:
Connected. Documents in store: 0
The list_documents() call returns a ListDocsResponse object — access the list of documents through .documents. An empty list is expected on a fresh account. Keep your URI out of source code: store it in an environment variable and read it with os.environ.get("MORPHIK_URI").
Ingesting Text and Files
Morphik supports three main ingestion methods: plain text via ingest_text(), files (PDFs, images, Office docs) via ingest_file(), and entire directories via ingest_directory(). All three accept a metadata dict that you can use to filter results later.
Ingesting Plain Text
Use ingest_text() when you already have text content in memory — from a database field, an API response, or a scraped web page. Pass the text and any metadata key-value pairs you want to filter on later:
# ingest_text.py
import os
from morphik import Morphik
db = Morphik(os.environ["MORPHIK_URI"])
doc = db.ingest_text(
content="""
FastAPI is a modern Python web framework built on Starlette and Pydantic.
It generates OpenAPI docs automatically, validates request/response data
via type hints, and supports both sync and async handlers natively.
Benchmark: FastAPI handles ~50,000 requests/sec on a single core.
""",
metadata={"category": "frameworks", "language": "python", "year": 2024}
)
print("Document ID:", doc.external_id)
print("Metadata stored:", doc.metadata)
Output:
Document ID: doc_b7c2a14f93e05821
Metadata stored: {'category': 'frameworks', 'language': 'python', 'year': 2024}
Ingesting PDF and Image Files
For files on disk — PDFs, PNGs, JPEGs, Word documents — use ingest_file(). This is where Morphik’s multimodal indexing kicks in: PDFs are processed page-by-page, preserving charts and diagrams in the index:
# ingest_pdf.py
import os
from morphik import Morphik
db = Morphik(os.environ["MORPHIK_URI"])
# Ingest a technical PDF -- diagrams, tables, and text all indexed
doc = db.ingest_file(
file="reports/q2_technical_spec.pdf",
metadata={"type": "spec", "department": "engineering", "quarter": "Q2"}
)
print(f"Ingested: {doc.external_id}")
print(f"Filename: {doc.filename}")
# Ingest an image -- useful for scanned documents or standalone charts
img_doc = db.ingest_file(
file="charts/architecture_diagram.png",
metadata={"type": "diagram", "project": "backend-v2"}
)
print(f"Ingested image: {img_doc.external_id}")
Output:
Ingested: doc_c9d4e26a0b1f7843
Filename: q2_technical_spec.pdf
Ingested image: doc_f1a3b85c2d094e67
Morphik handles the embedding and indexing asynchronously on the server — you get a document ID back immediately and the indexing completes in the background. For large PDFs, allow a few seconds before querying. The ingest_directory() method works the same way but accepts a folder path and recursively ingests all supported files inside it.
Retrieving Chunks and Querying with RAG
Morphik gives you two query modes: retrieve_chunks() for raw semantic search that returns matching text/page fragments, and query() for full RAG that returns an AI-generated answer grounded in those fragments. Use chunks when you want to build your own generation step; use query() when you want an answer directly.
Semantic Search with retrieve_chunks()
The retrieve_chunks() method runs a vector similarity search over your indexed documents and returns the top-k matching chunks with their scores and source metadata. This is useful when you want to inspect what the retrieval step is finding before passing it to a model:
# retrieve_chunks.py
import os
from morphik import Morphik
db = Morphik(os.environ["MORPHIK_URI"])
# Retrieve the 3 most relevant chunks for a query
chunks = db.retrieve_chunks(
query="How does FastAPI handle request validation?",
filters={"category": "frameworks"},
k=3
)
for i, chunk in enumerate(chunks, 1):
print(f"\n--- Chunk {i} ---")
print(f"Score: {chunk.score:.4f}")
print(f"Source: {chunk.document_id}")
print(f"Text: {chunk.content[:200]}")
Output:
--- Chunk 1 ---
Score: 0.9421
Source: doc_b7c2a14f93e05821
Text: FastAPI is a modern Python web framework built on Starlette and Pydantic.
It generates OpenAPI docs automatically, validates request/response data
via type hints, and supports both sync and async handlers natively.
--- Chunk 2 ---
Score: 0.7803
Source: doc_b7c2a14f93e05821
Text: Benchmark: FastAPI handles ~50,000 requests/sec on a single core.
The score field is a cosine similarity value between 0 and 1 — higher means more relevant. The filters argument accepts any metadata key-value pairs you set during ingestion, so you can scope a query to a specific project, document type, or date range without fetching everything and filtering in Python.
RAG Queries with query()
The query() method combines retrieval and generation into one call. Morphik finds the most relevant chunks and passes them to an LLM with your question, returning a synthesized answer in response.completion. This is the method to use when building a chatbot or Q&A interface over your documents:
# rag_query.py
import os
from morphik import Morphik
db = Morphik(os.environ["MORPHIK_URI"])
response = db.query(
query="What is the request throughput of FastAPI and how does it handle data validation?",
filters={"category": "frameworks"},
k=5 # how many chunks to use as context
)
print("Answer:")
print(response.completion)
print("\nSources used:", [s.document_id for s in response.sources])
Output:
Answer:
FastAPI handles approximately 50,000 requests per second on a single core.
It validates request and response data using Python type hints via Pydantic,
automatically generating OpenAPI documentation in the process. Both sync
and async handlers are supported natively through the Starlette foundation.
Sources used: ['doc_b7c2a14f93e05821']
The response.sources list tells you which documents contributed to the answer — essential for building citations into your UI or auditing hallucinations. The k parameter controls how many retrieved chunks get sent to the LLM as context; higher values give the model more information but cost more tokens per request.
Organising Documents with Folders
When you are storing documents from multiple projects or teams, you want to keep them separate so queries from one project do not bleed into another. Morphik handles this with folders — named scopes that act like directories in a filesystem. You create a folder, ingest documents into it, and then pass the folder to your queries:
# folders.py
import os
from morphik import Morphik
db = Morphik(os.environ["MORPHIK_URI"])
# Create a folder for a project
folder = db.create_folder(
name="backend-v2-docs",
description="Architecture specs and API references for backend v2"
)
print(f"Folder created: {folder.name}")
# Ingest a file directly into the folder
doc = folder.ingest_file(
file="specs/api_reference.pdf",
metadata={"type": "reference"}
)
print(f"Ingested into folder: {doc.external_id}")
# Query scoped to this folder only
response = folder.query("What authentication method does the API use?")
print(response.completion)
# Retrieve chunks from this folder only
chunks = folder.retrieve_chunks("rate limiting configuration")
print(f"Found {len(chunks)} relevant chunks")
Output:
Folder created: backend-v2-docs
Ingested into folder: doc_e5f2c3a9b0d18764
The API uses JWT bearer tokens for authentication, issued at /auth/token
with a 24-hour expiry. Refresh tokens are valid for 30 days.
Found 3 relevant chunks
Folders also support nested paths for deeper hierarchies. Pass full_path="/projects/alpha/specs" to create_folder() and Morphik creates the parent folders automatically. Use folder_depth=-1 on list or retrieve calls to include all descendant folders in the scope.
Using the Async Client
For production applications that handle concurrent requests — a FastAPI endpoint that fields multiple users’ document queries at once, for example — the synchronous client will block your event loop. Switch to AsyncMorphik to use await throughout:
# async_morphik.py
import asyncio
import os
from morphik.async_ import AsyncMorphik
async def search_documents(query_text: str) -> str:
async with AsyncMorphik(os.environ["MORPHIK_URI"]) as db:
response = await db.query(
query=query_text,
filters={"category": "frameworks"},
)
return response.completion
async def main():
# Run two queries concurrently
results = await asyncio.gather(
search_documents("How does FastAPI validate requests?"),
search_documents("What is FastAPI built on top of?"),
)
for i, result in enumerate(results, 1):
print(f"\nQuery {i} answer:")
print(result[:200])
asyncio.run(main())
Output:
Query 1 answer:
FastAPI validates requests using Pydantic models and Python type hints,
automatically generating OpenAPI documentation from the same definitions.
Query 2 answer:
FastAPI is built on Starlette (the ASGI framework) and Pydantic (the
data validation library).
The async with AsyncMorphik(...) as db pattern is important — the context manager ensures the underlying HTTP session is properly closed when your function exits. Both queries above run concurrently using asyncio.gather(), so the total wall-clock time is roughly equal to the slower of the two, not their sum.
Real-Life Example: Building a PDF Knowledge Base CLI
Here is a practical command-line tool that lets you ingest a folder of PDF reports and then interactively query them. This pattern is useful for internal knowledge bases, legal document search, or engineering spec lookup:
# pdf_knowledge_base.py
import os
import sys
from pathlib import Path
from morphik import Morphik
def ingest_reports(db: Morphik, reports_dir: str) -> int:
"""Ingest all PDFs in a directory and return the count."""
folder = db.create_folder(name="reports", description="Ingested PDF reports")
pdf_files = list(Path(reports_dir).glob("*.pdf"))
if not pdf_files:
print(f"No PDFs found in {reports_dir}")
return 0
for pdf_path in pdf_files:
doc = folder.ingest_file(
file=str(pdf_path),
metadata={
"filename": pdf_path.name,
"type": "report"
}
)
print(f"Ingested: {pdf_path.name} -> {doc.external_id}")
return len(pdf_files)
def interactive_query(db: Morphik) -> None:
"""Run an interactive Q&A session over ingested documents."""
folder = db.create_folder(name="reports") # Retrieve existing folder
print("\nPDF Knowledge Base ready. Type 'quit' to exit.\n")
while True:
question = input("Ask a question: ").strip()
if question.lower() in ("quit", "exit", "q"):
print("Goodbye.")
break
if not question:
continue
response = folder.query(query=question, k=5)
print(f"\nAnswer: {response.completion}")
if response.sources:
source_names = [s.metadata.get("filename", s.document_id)
for s in response.sources]
print(f"Sources: {', '.join(set(source_names))}")
print()
def main():
uri = os.environ.get("MORPHIK_URI")
if not uri:
print("Error: set the MORPHIK_URI environment variable first.")
sys.exit(1)
db = Morphik(uri)
if len(sys.argv) > 1 and sys.argv[1] == "ingest":
reports_dir = sys.argv[2] if len(sys.argv) > 2 else "./reports"
count = ingest_reports(db, reports_dir)
print(f"\nIngested {count} PDF(s). Run without arguments to query.")
else:
interactive_query(db)
if __name__ == "__main__":
main()
Sample session:
# First run: ingest the PDFs
$ python pdf_knowledge_base.py ingest ./reports
Ingested: q2_technical_spec.pdf -> doc_c9d4e26a0b1f7843
Ingested: api_reference.pdf -> doc_e5f2c3a9b0d18764
Ingested: architecture_guide.pdf -> doc_a1b2c3d4e5f60789
Ingested 3 PDF(s). Run without arguments to query.
# Second run: interactive query
$ python pdf_knowledge_base.py
PDF Knowledge Base ready. Type 'quit' to exit.
Ask a question: What authentication method does the API use?
Answer: The API uses JWT bearer tokens issued at /auth/token with a 24-hour
expiry. Refresh tokens are valid for 30 days and must be rotated on each use.
Sources: api_reference.pdf
Ask a question: quit
Goodbye.
The ingest_reports() function uses a folder to scope all reports together. The interactive_query() function references the same folder by name, so queries only search the ingested reports — not any other documents in your Morphik account. You can extend this by adding metadata filters (e.g., {"quarter": "Q2"}), showing chunk scores in the output, or replacing the CLI loop with a FastAPI endpoint for a web-based interface.
Frequently Asked Questions
What file types does Morphik support for ingestion?
Morphik supports PDFs (including scanned and visually rich ones), images (PNG, JPEG, and other common formats), plain text, and Office documents. PDFs are the primary use case where Morphik provides the biggest advantage over traditional RAG, because it indexes each page as an image via ColPali rather than stripping the content to plain text. For images — standalone charts, scanned invoices, whiteboard photos — Morphik embeds them directly without any OCR step.
Do I need to run my own vector database or embedding model?
No. The hosted Morphik service at dev.morphik.ai handles all of that server-side. You supply documents and queries; Morphik handles embedding, indexing, storage, and retrieval. If you self-host Morphik Core, you run it via Docker and it manages its own internal vector store. Either way, you do not configure a separate Pinecone, Weaviate, or Chroma instance — Morphik is a self-contained system.
How is morphik.query() different from retrieve_chunks()?
retrieve_chunks() runs the retrieval step only — it returns the raw text or page chunks that are most similar to your query, with similarity scores. query() does retrieval plus generation: it fetches relevant chunks and sends them to an LLM with your question, returning a synthesized answer. Use retrieve_chunks() when you want to build your own generation pipeline or inspect what the system is finding. Use query() when you want a direct answer and are happy with Morphik choosing the LLM.
How do I filter queries to specific documents or projects?
There are two complementary mechanisms. First, use the filters argument to match on metadata you set during ingestion — for example, filters={"department": "engineering", "year": 2024} restricts retrieval to documents with those exact metadata values. Second, use Folder objects to create hard namespace boundaries: documents ingested into a folder are only retrieved when you query through that folder object. Combining both gives you fine-grained control over multi-tenant and multi-project knowledge bases.
Is Morphik free to use?
Morphik Core is source-available under the Business Source License 1.1. The hosted service at dev.morphik.ai has a free tier for personal and indie use. Commercial production deployments that generate more than US $2,000 per month in gross revenue require a paid commercial key. Self-hosted deployments under that revenue threshold are free. Each code version automatically re-licenses to Apache 2.0 four years after its first release, so the project is on a clear path to becoming fully open source.
Can I use Morphik with Claude or other AI assistants via MCP?
Yes. Morphik ships with built-in Model Context Protocol (MCP) support, which means you can connect it to Claude Desktop, Cursor, and other MCP-compatible tools and use your Morphik knowledge base as a context source directly inside those tools. You configure the MCP server endpoint in your client’s settings and then your AI assistant can retrieve from Morphik-indexed documents without any additional code. Full setup instructions are in the Morphik MCP docs.
Conclusion
Morphik closes the gap between “my AI can answer questions about text” and “my AI can answer questions about my actual documents” — the ones with tables, charts, diagrams, and scanned images that traditional RAG pipelines mangle beyond recognition. In this article we covered how to install the SDK and connect using a URI, how to ingest text with ingest_text() and files with ingest_file(), how to run semantic search with retrieve_chunks(), how to use full RAG with query(), how to scope documents to projects with folders, and how to switch to AsyncMorphik for concurrent workloads.
A good next step is to extend the PDF knowledge base CLI from the real-life example: add a web frontend with FastAPI, wire in metadata filtering by date or author, or hook the retrieve_chunks() output into your own custom LLM call so you control the generation step. For teams managing large document corpora, explore Morphik’s nested folders and the folder_depth=-1 parameter to query across an entire project tree in one call.
The official documentation lives at dev.morphik.ai/docs and covers the full REST API, self-hosting setup, integrations with Google Suite and Slack, and the knowledge graph features for visualising relationships between documents. The community Discord is active if you run into ingestion issues with unusual PDF layouts.