How It Works

The file is uploaded to durable storage. A source record is created in the database with status queued, and an ingestion job is placed on the processing queue.

A background worker picks up the job and downloads the file. The appropriate parser is selected based on the file's MIME type:

The extracted text is split into smaller passages (chunks) optimized for embedding and retrieval. The chunker uses token-aware splitting with configurable parameters:

Overlap ensures that information at chunk boundaries is not lost. Each chunk retains a reference to its source document and position.

Each chunk is converted into a 1024-dimensional vector using an embedding model. Embeddings are generated in batches for efficiency.

Chunks and their embedding vectors are stored in PostgreSQL with the pgvector extension. Each chunk is associated with its source document, tenant, and position metadata. The source status updates to ready.

When you (or an agent) runs a query, the query text is embedded using the same model. pgvector performs a cosine similarity search across all indexed chunks for the tenant, returning the top-k most relevant passages with their similarity scores and source metadata.

-- Simplified query (actual uses pgvector operators)
SELECT chunk_text, source_id, similarity_score
FROM chunks
ORDER BY embedding <=> query_embedding
LIMIT 5;