E-COMMERCE SCRAPER - PYTHON + FIRECRAWL

ARCHITECTURE
Firecrawl handles JavaScript-heavy sites (Selenium-equivalent). Output: clean JSON for each product. Local validation pipeline catches bad data before storage.

PRODUCT SCHEMA
fields: sku, name, price (int cents), currency, in_stock (bool), rating (0-5), reviews_count (int), category, image_url, last_updated (timestamp)
Validation: price > 0, rating 0-5, reviews_count >= 0, URL is valid

PYTHON SCRIPT

import asyncio
from firecrawl import Firecrawl
import json

fc = Firecrawl(api_key='your_key')

async def scrape_product(url):
    data = await fc.scrape(url)
    product = {
        'sku': data.get('sku'),
        'name': data.get('name'),
        'price': int(float(data.get('price', 0)) * 100),
        'rating': float(data.get('rating', 0)),
        'in_stock': data.get('in_stock', False)
    }
    validate_product(product)
    return product

DATA FLOW
1. Load product URL list from CSV
2. Batch scrape 50 URLs concurrently
3. Validate schema (reject malformed)
4. Write to products.jsonl (append-only)
5. Upload to database or cloud storage

RUN TIME
500 products: ~8 minutes (rate limit: 1 req/sec). Output: 25MB JSONL file. Schema validation catches 2-5% bad records (retry with fallback parsing).