Custom digital newspaper archive software for a private collection: newspaper issues in PDF become an archive where any article can be found by keyword, by meaning or by asking a question, with the source always cited.
The problem
The client had a collection of newspaper issues as PDF files. Finding an article meant opening files one by one, and many pages were scans with no text. The archive also had to stay private: only people invited by the administrator could get in.
What I built: digital newspaper archive software
I built a private web archive where every issue is stored as newspaper title, date and PDF, and every page becomes searchable.
- Text extraction with OCR fallback: a page keeps its own text layer when it has one. Pages with fewer than 40 characters are rendered at 220 DPI and read with Tesseract. Old scans become searchable too.
- Italian full-text search: search uses Italian stemming and ignores accents, so “citta” finds “città”. Exact phrases rank first, with highlighted snippets.
- Search by meaning: page text is split into overlapping blocks and turned into 768-dimension vectors with a multilingual model. Readers find articles even when they do not know the exact words.
- Assistant with citations: a reader asks a question in plain language. The system retrieves up to 8 passages, by meaning and by keyword, and Claude answers only from those passages, citing title, date and page. It is switched on from the admin panel with the client’s own Anthropic key.
- Bulk upload from a folder: the administrator selects a folder of PDFs and the date is read from each file name. Duplicates are skipped by title and date or by SHA-256 fingerprint, so an upload can be repeated safely.
- Invite-only access: two roles, administrator and reader. The panel manages users, titles, the AI key and backups.
How it performs
- Memory cap of 8 GB on the embedding service, with a daily restart. Before the cap it had grown to about 15 GB and blocked the queue for a week.
- Nightly backup at 03:30: a compressed database dump, keeping the last 14, plus an incremental mirror of the PDFs.
- Stuck jobs recovered automatically: a job left running for more than 20 minutes goes back in the queue, up to 6 attempts.
Built with
Python 3.11 and FastAPI for the web app. PostgreSQL 16 with pgvector, unaccent and pg_trgm for data, text search and vectors. PyMuPDF and Tesseract for text and OCR. Sentence-transformers with the multilingual-e5-base model for embeddings, run as a separate local service. The Anthropic API for the assistant. Everything runs as systemd services behind an Apache reverse proxy.
Why it was done this way
I kept the job queue inside PostgreSQL instead of adding Redis. Workers claim jobs with FOR UPDATE SKIP LOCKED, so several workers run in parallel and never process the same issue twice. The server also hosts other sites, so one less service to run and secure was a real gain.
