All
Articles 172,414Blog Posts 162,922Tech Tutorials 45,954Research Papers 33,304News 21,784
⚡ AI Lessons
Dev.to · Devil Scrapes
1d ago
A SUCCEEDED run with 3 rows, and every row was fabricated
Quick answer A scraper run can report SUCCEEDED with rows in the dataset and still have...
Dev.to · Devil Scrapes
6d ago
Our Scraper Said Every EU Tender Was Titled "F"
Quick answer TED — the EU's official public-procurement journal — publishes every tender...
Dev.to · Devil Scrapes
6d ago
An API With No Pagination Is a Billing Problem
Quick answer The CPSC's recall API has no pagination. None — no page, no offset, no page...
Dev.to · Devil Scrapes
6d ago
The French Registry Says 10,000 Results. It Means the Cap.
Quick answer France's official company registry is open, keyless and free — and it will...
Dev.to · Devil Scrapes
1w ago
5 states, 2 working filters: scraping US childcare license registries
Five states, one query language, and an "active licenses only" checkbox that only actually filters...
Dev.to · Devil Scrapes
1w ago
Same API standard, four incompatible schemas: scraping state cosmetology license registries
"Just query the Socrata API" is true and also useless advice. Socrata SODA is a real open standard —...
Dev.to · Devil Scrapes
1w ago
Google Trends API: the 200 OK that means you got soft-blocked
Google Trends has no public API. What it has is the same internal JSON endpoints the...
Dev.to · Devil Scrapes
1w ago
We charged the customer, then validated their input
We caught this one an hour before publishing, and it is the kind of bug that would have looked like...
Dev.to · Devil Scrapes
1w ago
Your eBay scraper is billing customers for adverts
Scraping eBay search results looks like a solved problem until you count the rows and find you have...
Dev.to · Devil Scrapes
1w ago
Zoopla dropped __NEXT_DATA__ and our scraper returned zero rows
Our first cloud run against Zoopla cleared Cloudflare cleanly and returned zero rows. That...
Dev.to · Devil Scrapes
1w ago
A regex ate the room count and turned 53 m into 353
Our test suite caught a bug that no amount of staring at the regex would have found: a French...
Dev.to · Devil Scrapes
2w ago
The API where 200 OK means your request was invalid
The World Bank's v2 API is free, keyless, and serves 20,000+ development indicators for every country...
Dev.to · Devil Scrapes
2w ago
A 404 from SEC's XBRL API is usually the correct answer
SEC's XBRL API is the best-kept secret in financial data. No key, no scraping of HTML filings, no...
Dev.to · Devil Scrapes
2w ago
GBIF has a hard wall at exactly 100,001 records
GBIF — the Global Biodiversity Information Facility — indexes over three billion species occurrence...
Dev.to · Devil Scrapes
2w ago
Crossref's funder filter wants a DOI, not a name — and 3 more traps
Crossref's works endpoint is about as friendly as a public API gets: no key, no rate-limit paperwork,...
Dev.to · Devil Scrapes
2w ago
The JSON Schema bug that only fires on a customer's first real query
There is a category of bug that passes every test you write, ships to production, and then fails on a...
Dev.to · Devil Scrapes
2w ago
The day our scraper fleet shipped nothing — and the afternoon it shipped three
We spent an entire working day last week shipping nothing. Nine scrapers were queued, every piece of...
Dev.to · Devil Scrapes
📣 Digital Marketing & Growth
⚡ AI Lesson
2w ago
Redfin caps at 350, Apple caps at ~50, Glassdoor caps at 3 — none of them are rate limits
Short version: three review/listing platforms, three different reasons you stop getting more data....
Dev.to · Devil Scrapes
2w ago
Scraping 7 EU car marketplaces: normalize the units, never the language
Seven European car marketplaces. Seven national languages. Seven different ideas of what a "listing"...
Dev.to · Devil Scrapes
📊 Data Analytics & Business Intelligence
⚡ AI Lesson
3w ago
Levels.fyi Compensation Scraper: total comp is not one number
Quick answer Levels.fyi's own compensation reports separate base, stock, and bonus per...
Dev.to · Devil Scrapes
📊 Data Analytics & Business Intelligence
⚡ AI Lesson
2mo ago
Charting Twitch Chat Velocity: Hype Curve Analysis for Esports VODs
Quick answer: To measure Twitch chat hype or velocity, pull the full VOD chat using the...
Dev.to · Devil Scrapes
📐 ML Fundamentals
⚡ AI Lesson
2mo ago
Training a Twitch chat toxicity classifier on real VOD data at scale
Quick answer: Twitch has no public API for VOD chat replay. To build a Twitch toxicity classifier...
Dev.to · Devil Scrapes
📊 Data Analytics & Business Intelligence
⚡ AI Lesson
3mo ago
Discogs Price Scraper: pull every marketplace listing as clean JSON
Quick answer: The Discogs marketplace is the world's largest secondary market for vinyl, CDs, and...
Dev.to · Devil Scrapes
📣 Digital Marketing & Growth
⚡ AI Lesson
3mo ago
Bing Search API Replacement: scrape SERP results for $1.05/1K
Quick answer: Microsoft retired the Bing Search API on August 11, 2025. There is no longer an...
DeepCamp AI