Validated Web Scraping and Extraction Pipelines
Build scrapers that validate records, track crawl coverage, and expose missing or corrupted data before export.
Before & after
Same prompt, same agent — the only difference is the skill.
Web scraping for e-commerce data is definitely doable, but it depends on a few factors. First, which site are we scraping? Does it have a public API that would be easier than web scraping? If we do scrape, we need to check robots.txt and the terms of service to make sure we're allowed. Also, is the site static HTML or does it use JavaScript to load product data dynamically? That changes our approach — static HTML can be scraped with BeautifulSoup, but JavaScript-rendered pages need something like Selenium or Puppeteer. How much data are we talking about — a few hundred products or millions? That affects our architecture. Do you need it as JSON, CSV, or in a database? And how often do you need to refresh the data? I can put together a scraping solution, but let me know these details first so I can recommend the best approach.
E-COMMERCE SCRAPER - PYTHON + FIRECRAWL
ARCHITECTURE
Firecrawl handles JavaScript-heavy sites (Selenium-equivalent). Output: clean JSON for each product. Local validation pipeline catches bad data before storage.
PRODUCT SCHEMA
fields: sku, name, price (int cents), currency, in_stock (bool), rating (0-5), reviews_count (int), category, image_url, last_updated (timestamp)
Validation: price > 0, rating 0-5, reviews_count >= 0, URL is valid
PYTHON SCRIPT
import asyncio
from firecrawl import Firecrawl
import json
fc = Firecrawl(api_key='your_key')
async def scrape_product(url):
data = await fc.scrape(url)
product = {
'sku': data.get('sku'),
'name': data.get('name'),
'price': int(float(data.get('price', 0)) * 100),
'rating': float(data.get('rating', 0)),
'in_stock': data.get('in_stock', False)
}
validate_product(product)
return product
DATA FLOW
1. Load product URL list from CSV
2. Batch scrape 50 URLs concurrently
3. Validate schema (reject malformed)
4. Write to products.jsonl (append-only)
5. Upload to database or cloud storage
RUN TIME
500 products: ~8 minutes (rate limit: 1 req/sec). Output: 25MB JSONL file. Schema validation catches 2-5% bad records (retry with fallback parsing).About this skill
Your scraper finishes successfully, but half the prices are blank or the crawl stopped after page one. This skill builds extraction pipelines that distinguish valid data, genuine empty results, and silent failures, with evidence for every rejected record.
What it does
- Chooses among direct APIs, local Python, browser rendering, and Firecrawl based on the actual data source and access requirements.
- Defines field types, required values, business constraints, and missing-value behavior before writing extraction logic.
- Handles pagination, bounded retries, rate limits, duplicate records, and resumable checkpoints.
- Preserves source URLs, raw values, extraction locations, and document page references for debugging.
- Separates rejected records from accepted output and reports gaps in crawl coverage.
- Tests malformed responses, missing fields, repeated cursors, and realistic layout changes.
What you get
A right-sized implementation or repair with an explicit extraction contract, validation rules, failure handling, and a verification plan. When execution is available and authorized, results include observed acceptance counts and unresolved coverage gaps.
Best for
Developers collecting website, API, or document data who need dependable structured output, especially when an existing scraper quietly returns incomplete or misleading results.
SKILL.md preview
---
name: validated-web-scraping-and-extraction-pipelines
description: >-
Build and repair web scraping, crawling, document extraction, and API parsing pipelines using source profiling, explicit schemas, field evidence, and crawl reconciliation. Choose local Python or Firecrawl according to demonstrated requirements. Use when the user says "build a scraper", "my scraper keeps returning empty rows", "extract these PDFs into structured data", "crawl all the product pages", "validate this API import", "should I use Firecrawl or Python?", or "my crawler stops after page one".
---
# Validated Web Scraping and Extraction Pipelines
… (sign up to view the full skill)More development skills
View all Development skills →Performance Bottleneck Profiling: Node, Python, Go
Classifies a slowdown, gives exact profiler commands, reads the output, and proves the fix with same-conditions p95/p99 numbers.
Feature Scaffolding with Working Tests
Turn a short feature spec into code, runnable tests, and integration wiring that match your existing project.
Feature Flag Rollouts and Kill Switches
Design staged flag rollouts, verify rollback paths, and audit flag debt with explicit defaults, gates, and retirement plans.
Microcontroller Firmware and RTOS Engineering
Build and debug MCU firmware with bounded ISR/RTOS timing, safe DMA ownership, and measured power budgets.