Background Import Platform

Project roadmap

1. 01

   Design the import system

   ~2h

   Write DESIGN.md: an import has a file, a column mapping, a status (uploaded, mapped, processing, completed, failed or cancelled), counters (rows processed, imported and rejected) and the last completed chunk. Each chunk is a job processing a range of rows; rejected rows are stored with their row number and errors. Imported records are matched by a natural key (such as SKU or email) and upserted. Describe what happens when a job fails or the server restarts. Done when: DESIGN.md covers states, chunks, errors and recovery.

2. 02

   Upload and map columns

   ~2h

   Build an upload page accepting CSV files up to a size limit, stored with a random name. Read the header and the first 10 rows to show a preview, and let the user map file columns to fields (suggesting matches by name). Check that required fields are mapped before starting. Done when: a large CSV can be uploaded, previewed and mapped.

3. 03

   Process in chunks

   ~3h

   When the import starts, count the rows and dispatch a batch of jobs (Bus::batch), one per chunk of 1,000 rows. Each job reads only its rows (streaming the file with SplFileObject or League CSV), validates each row with Laravel's Validator, upserts valid rows by the natural key in one query (Model::upsert) inside a transaction, stores rejected rows with their errors, and updates the counters. Done when: a 200,000-row file imports completely in the background.

4. 04

   Make imports resumable

   ~2.5h

   Record each finished chunk in a table, and make jobs skip chunks already finished, so retrying or re-dispatching a batch never processes rows twice. Configure job retries with backoff and a timeout. Add a command that finds imports stuck in processing (for example after a server restart) and dispatches their unfinished chunks. Allow cancelling an import. Done when: killing the queue worker mid-import and restarting it finishes the import with correct counts and no duplicates.

5. 05

   Show progress and results

   ~2h

   Build an import page that shows the status, a progress bar and the counters, updating every few seconds (by polling, or with broadcasting). When finished, show a summary with the most common errors and a button to download the rejected rows as a CSV with an extra errors column, ready to fix and re-upload. Done when: progress updates live, and the rejected rows download works.

6. 06

   Add limits and cleanup

   ~1h

   Limit how many imports each user can run at once and the queue's concurrency, stop an import when the rejection rate is very high (for example the wrong file), and delete uploaded files and rejected rows after a set number of days with a scheduled command. Done when: limits apply, and old files are cleaned up.

7. 07

   Test it

   ~2.5h

   Write feature tests with the queue running synchronously and small test files: column mapping, validation errors recorded with row numbers, upserts not duplicating on a second import, resuming after a simulated failure in the middle of a chunk, cancelling, and the rejected rows download. Done when: php artisan test passes.

Resources

• DocsLaravel: queues and job batching
• DocsLaravel: validation
• DocsLaravel: Eloquent upserts
• DocsLeague CSV
• DocsLaravel: task scheduling