DataShelf

Challenge

Large CSV imports block web requests and leave users staring at a spinner with no feedback. Product catalog files — especially SanMar-style apparel data with tens of thousands of rows — need asynchronous processing with visible progress and reliable upserts into a relational schema.

Solution Architecture

• Authenticated upload flow — drag-and-drop CSV validation (up to ~100 MB), file stored on disk with file_uploads tracking
• Queue-based processing — ProcessCsvUpload job on database queue; statuses flow uploaded → processing → completed
• League CSV streaming — row-by-row parse with updateOrCreate upserts on unique_key into products table
• Batched progress persistence — database updated every 100 rows so the UI can poll without hammering the server
• Live dashboard polling — frontend hits /api/file-statuses every 2 seconds with animated progress bars and status badges
• Docker deployment — PHP-FPM, Nginx, MySQL 8, and phpMyAdmin via docker-compose.yml

flowchart LR
  User[User] --> Upload[CSV Upload]
  Upload --> Disk[File Storage]
  Upload --> Record[file_uploads Record]
  Record --> Queue[ProcessCsvUpload Job]
  Queue --> Parse[League CSV Parse]
  Parse --> Products[(products Table)]
  Dashboard[Dashboard] -->|poll 2s| API[/api/file-statuses]
  API --> Record

Results

• Handles 36,000+ row product catalogs with visible progress and failed-row counts
• Keeps HTTP requests non-blocking via database-backed queue workers
• Shipped polished landing page, sidebar dashboard, and concurrent dev workflow (composer dev runs server + queue + Vite)
• Evolved from an early resume-upload prototype into a focused data pipeline branded as DataShelf

Timeline

| Phase | Focus | |-------|-------| | Prototype | Auth scaffolding, basic file upload | | Core pipeline | Queue jobs, League CSV parsing, product upserts | | Live UX | Progress polling, status badges, error modals | | Production | Docker Compose stack, landing page polish, log cleanup tooling |