Document Search Service
Build a document search service in C# that indexes text documents (articles, manuals, support answers) and returns the best matches for a query, ranked by relevance, with the matching words highlighted in a short snippet. You'll build the search engine yourself: text is split into normalized words, an inverted index stored in PostgreSQL maps each word to the documents and positions where it appears, and results are ranked with BM25, the scoring behind most search engines, with support for phrases in quotes. Documents can be added, updated and deleted while searches keep running, and the API returns JSON with System.Text.Json. You'll learn how search engines work, tokenization, inverted indexes, relevance ranking, concurrent updates, and measuring search quality.
- Estimate
- ~14h
- Steps
- 7
- Completed by
- 0
- Proposed by
- codeseed.app
System.Text.Json · PostgreSQL
Project roadmap
- 01
Design the search service
~2hWrite DESIGN.md: documents (id, title, body and a version), text analysis (lowercase, remove accents and punctuation, stop words and simple stemming), the PostgreSQL tables (documents, terms, and postings with term, document, field, frequency and positions), BM25 scoring with a title boost, phrase queries using positions, and how updates replace a document's postings in one transaction. Done when: DESIGN.md explains indexing, querying and updates with an example.
- 02
Analyze text
~1.5hWrite an Analyzer that turns text into tokens with positions: Unicode normalization and accent removal, lowercasing, splitting on non-letters, removing stop words, and a simple stemmer (such as Porter, implemented or from a library). The same analyzer is used for documents and queries. Done when: unit tests show Running shoes! and running shoe produce the same terms.
- 03
Index documents in PostgreSQL
~2.5hCreate the tables with Npgsql and migrations, with indexes on postings by term. Indexing a document deletes its old postings and inserts the new ones in bulk (with binary COPY for speed) inside one transaction, and keeps the statistics BM25 needs (document count, and average field lengths). Expose PUT /documents/{id} and DELETE /documents/{id}. Done when: indexing, updating and deleting keep the index consistent.
- 04
Search and rank
~3hImplement GET /search?q=&limit=&offset=: analyze the query, load the postings for its terms, compute BM25 scores with the title boost, support "quoted phrases" by checking consecutive positions, and return the top results. Build a snippet around the best-matching part of the body with the matches wrapped in <mark> and everything else HTML-escaped. Done when: searches return relevant results with correct snippets, and phrase queries only match exact phrases.
- 05
Handle concurrency and limits
~1.5hMake sure searches never see a half-updated document (transactions do this), limit query length, number of terms and page size, set a timeout on searches, and cache frequent searches briefly, invalidating the cache when documents change. Log each search with its duration and result count. Done when: searching while documents are being updated always gives consistent results.
- 06
Measure search quality
~1.5hIndex a real set of documents (for example a public dataset of articles) and write 20 test queries with the documents you'd expect at the top. Measure precision at 5 and mean reciprocal rank, try different title boosts and stemming on or off, and record the results in README.md. Done when: README.md compares the settings with measured numbers.
- 07
Test it
~2hWrite xUnit tests: the analyzer, BM25 scores on a tiny corpus calculated by hand, phrase queries, snippets and escaping, updates removing old terms, and API integration tests with PostgreSQL in Testcontainers. Done when: dotnet test passes.
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack