Document Similarity Service

Project roadmap

1. 01

   Design the service

   ~2h

   Write DESIGN.md: documents (id, title, text and metadata), how text becomes vectors (TF-IDF with scikit-learn, optionally sentence embeddings for comparison), how the index is stored (the vectorizer and matrix saved with joblib, plus a mapping from rows to ids), how adding and deleting documents works (rebuilding in the background when many changed), the API, and how quality will be measured. Choose a dataset with known duplicates (for example the Quora question pairs). Done when: DESIGN.md covers vectors, the index, updates and evaluation.

2. 02

   Import and clean documents

   ~1.5h

   Write an import command that loads documents from CSV with pandas, validates the required columns, removes empty and duplicate ids, cleans the text (lowercase, remove HTML and extra whitespace), and stores them in SQLite. Report rows that were skipped and why. Done when: a CSV of documents is imported, and bad rows are reported.

3. 03

   Build the similarity index

   ~2.5h

   Fit a TfidfVectorizer (with sensible settings such as sublinear term frequency and removing very common and very rare words), build the matrix, and write a search function that vectorizes a query and returns the top K documents by cosine similarity (with linear_kernel on normalized vectors), excluding the document itself when searching by id. Save and load the index atomically (write to a temporary file, then rename). Done when: searching for a document's text returns it and closely related documents at the top.

4. 04

   Build the FastAPI service

   ~2h

   Build a FastAPI app with POST /documents, DELETE /documents/{id}, POST /similar (text, k and an optional minimum score) and GET /documents/{id}/similar, with Pydantic models for every request and response and limits on text length and k. New documents are searchable right away (vectorized with the current vocabulary), and a background task rebuilds the index when many documents changed. Done when: every endpoint works from the /docs page.

5. 05

   Handle concurrency and recovery

   ~1.5h

   Make searches keep using the old index while a rebuild runs, then swap it in atomically. On startup, load the saved index, or rebuild it from the database if it's missing or out of date (compare a stored document count and checksum). Log search times. Done when: searches keep working during a rebuild, and a deleted index file is rebuilt at startup.

6. 06

   Evaluate quality

   ~2h

   Using the labeled duplicate pairs, measure how often the true duplicate is in the top 1 and top 5 results, and precision and recall at different score thresholds, reported with pandas. Compare TF-IDF settings (and embeddings if you added them), and choose a default threshold for calling something a duplicate. Record the results in README.md. Done when: README.md shows the measured quality and the chosen threshold.

7. 07

   Test it

   ~2h

   Write pytest tests: text cleaning, the search ranking on a tiny hand-made corpus, excluding the document itself, saving and loading the index, the API with FastAPI's TestClient (including validation errors), and a rebuild while searching. Done when: pytest passes.

Resources

• Docsscikit-learn: TfidfVectorizer
• Docsscikit-learn: cosine similarity and kernels
• DocsFastAPI tutorial
• Docspandas user guide
• DocsSentence Transformers documentation