Web Scraping and Browser Automation Scripts
Get Playwright scripts that scrape, fill forms, and capture screenshots, find the data source first, and stay working after redesigns.
Before & after
Same prompt, same agent — the only difference is the skill.
Web scraping can be useful for collecting data from websites. You'll need to think about how to extract the information you want. Different sites have different structures so you might need to adjust your approach for each one. You could use a browser automation tool to navigate and click through pages. Parsing HTML can help you extract specific elements. Consider the performance and speed of your scraper. You might want to respect rate limits to avoid overloading the servers. Error handling is important too.
BROWSER AUTOMATION WORKFLOW
SETUP:
Target: competitor-saas.example.com/pricing
Extract: Plan names, prices, feature count, feature list text, CTA button text
Output format: JSON per competitor (rows with plan_name, price_usd, features_json, cta_text)
STEP 1: Navigate and scan
Load pricing page → wait for pricing-table selector to render (timeout: 10s)
Capture current URL (verify no redirect)
STEP 2: Click through tabs (if pricing is tabbed)
If 'Monthly/Annual toggle' exists → click Annual → wait 1s for rerender
Extract both versions (frequency data)
STEP 3: Parse plan cards
Select all .pricing-plan elements
For each plan card:
- Extract text: plan-name, plan-price, feature-list (bullets or checkmarks)
- Extract CTA button: button text, destination URL
STEP 4: Screenshot differentiators
Capture full pricing section as PNG (size comparison, visual hierarchy)
STEP 5: Save and alert
Write JSON: competitors/pricing_2026-07-31.json
Log: '3 plans extracted, feature parity analysis ready'
OUTPUT EXAMPLE:
{
"competitor": "AppName",
"pricing_page_url": "...",
"plans": [
{"name": "Starter", "price_monthly": 29, "features": ["API access", "10k calls/mo"]},
{"name": "Pro", "price_monthly": 99, "features": ["Unlimited API", "Priority support"]}
],
"scraped_at": "2026-07-31T14:32:00Z"
}
NOTE: Respect robots.txt; 2sec delay between page loadsAbout this skill
Your scraper worked on Tuesday and returned empty rows on Thursday. Your form bot double-submitted after a retry. Your script passes locally and times out in headless CI. This skill writes browser automation that checks where the data actually lives before reaching for a browser, waits on real signals instead of sleeps, and fails loudly instead of quietly returning garbage.
What it does
- Finds the cheapest data source first: embedded JSON, the page's own XHR endpoint, or static HTML, and drives a full browser only when needed
- Writes Playwright locators scoped to stable anchors (role, label, test id, row containers) that survive cosmetic redesigns
- Handles the hard parts: login sessions reused via saved state, pagination, infinite scroll, iframes, popups, downloads, cookie banners
- Makes form automation safe with dry-run mode, a pre-submit screenshot, success and error detection, and idempotency so retries never double-submit
- Captures stable screenshots with frozen animations, masked dynamic regions, loaded fonts, and triggered lazy images
- Validates every extracted row against a schema and alerts on selector drift, with polite rate limits and clear limits on what should not be automated
What you get
A runnable script in your stack (Python or Node), with the data schema, wait strategy, failure handling, and scheduling notes. Pasted scripts get a diagnosis and a targeted fix.
Best for
Developers, ops people, and analysts automating recurring web chores: competitor price tracking, report downloads behind a login, bulk form entry, and scheduled page captures.
SKILL.md preview
---
name: web-scraping-browser-automation-scripts
… (sign up to view the full skill)More development skills
View all Development skills →Performance Bottleneck Profiling: Node, Python, Go
Classifies a slowdown, gives exact profiler commands, reads the output, and proves the fix with same-conditions p95/p99 numbers.
Feature Scaffolding with Working Tests
Turn a short feature spec into code, runnable tests, and integration wiring that match your existing project.
Feature Flag Rollouts and Kill Switches
Design staged flag rollouts, verify rollback paths, and audit flag debt with explicit defaults, gates, and retirement plans.
Microcontroller Firmware and RTOS Engineering
Build and debug MCU firmware with bounded ISR/RTOS timing, safe DMA ownership, and measured power budgets.