Web scraping is using a program to read web pages and pull out specific information, the way a person would by visiting and copying. Done responsibly, it collects only public data, follows the site's terms and robots.txt, requests pages slowly, and uses an official API whenever one exists.
Refreshing a clerk's website to see whether anything has changed on a case is dull work, and it is easy to forget on a busy day. We handed it to a small program. Our Docket Scraper checks public court records on a schedule and reports what changed, so nobody has to refresh anything.
That is web scraping in its most useful and least controversial form: public information, checked politely, saving a person a chore. Scraping has a mixed reputation because it is also used carelessly. This post covers what it is, when it makes sense, and the rules we hold ourselves to.
What is web scraping?
Web scraping is the practice of using software to load web pages and extract particular pieces of information from them, such as a date, a price, or a status. The program does what a person with a browser would do, but on a schedule and without getting bored.
A scraper has three parts. It fetches a page. It parses the page to find the data it wants. It stores or compares the result. Our docket tool adds a fourth step: it compares today's result with the last one and reports only the difference, because "nothing changed" is not worth an alert.
Scraping is different from using an API. An API is a door the site owner built for programs, with documented rules. Scraping reads the pages built for people. When a site offers an API for the data you need, use it. We explain the difference in what is an API integration.
What do businesses use web scraping for?
Businesses use web scraping to monitor public information that changes and has no feed or API: court dockets, public notices, government listings, their own published content, and similar pages. The common thread is a person who currently checks a website by hand on a regular basis.
Uses we have built or would build:
- Monitoring public records. Watching a docket or a public register for changes.
- Migrating your own content. Our WP Ripper tool crawls a site, extracts the real content, downloads the images, and imports everything into WordPress as native blocks. When a client is moving their own site and nobody has a clean export, crawling the public pages is the practical route.
- Auditing your own site. Crawling every page to find broken links, missing titles, or thin content.
Notice how many of those involve your own site or public records. Those are the easy cases. Collecting someone else's content to republish it is a different matter, and not one we take on.
How do you scrape a website responsibly?
Scrape a website responsibly by collecting only public data, reading and following the site's terms of use and robots.txt file, keeping your request rate low, identifying your program honestly, and stopping when asked. If the site offers an API or a data download, use that instead.
Our checklist, in order:
- Look for an API or export first. It is more reliable and it is what the owner wants you to use.
- Read the terms of use. If they prohibit automated access, that is your answer.
- Check robots.txt. It states which paths the site asks programs to stay out of. Respect it.
- Take public data only. Nothing behind a login you were not given for this purpose. No personal data you have no reason to hold.
- Go slowly. One request, a pause, the next request. A scraper should be less of a burden on the server than a person clicking around.
- Run it no more often than you need. If the data changes daily, checking every minute is rude and pointless.
- Keep only what you need. Store the fields you use, not whole pages.
The rules that apply depend on the site, the data, and where you operate, so have your own attorney review any scraping project that goes beyond plainly public information.
How does a scheduled scraper run without a server?
A scheduled scraper can run without a server by using a scheduler that someone else operates. The job starts at a set time, runs for a few seconds or minutes, saves its result, and shuts down. Nothing stays switched on between runs.
Our Docket Scraper is a Node program that runs on GitHub Actions, a CI service, on a schedule. It uses a free CI runner and there is no server to maintain. A schedule in that system is a few lines of configuration, for example:
on:
schedule:
- cron: '0 14 * * 1-5'That example means "at 14:00 UTC, Monday through Friday". The job runs, compares what it found with what it saw last time, and reports the changes. We compare this approach with the alternatives in what is a cron job.
Some pages only show their content after JavaScript runs in a browser. For those, a plain fetch returns an empty shell. The fix is a headless browser, a real browser engine driven by code. WP Ripper's command-line crawler works this way, using Playwright. It is slower and heavier, so we use it only when a simple request will not do.
What goes wrong with web scrapers?
Web scrapers break when the page they read changes. A redesign moves the data, the parser finds nothing, and a badly built scraper reports "no changes" forever. The fix is to treat an empty or malformed result as an error and alert a person.
| Problem | What it looks like | How we handle it |
|---|---|---|
| Page layout changed | Parser returns nothing | Fail loudly, do not report "no change" |
| Site is down | Request times out | Retry later, alert if it persists |
| Too many requests | Site starts refusing | Slow down, lower the schedule |
| Content needs JavaScript | Empty page from a plain fetch | Use a headless browser |
A scraper is a small program with a maintenance cost. Budget a little time each year for the day the source site changes.
Frequently asked questions
Is web scraping legal?
It depends on what you collect, how, and from where. Public data gathered politely is the low-risk end. Ignoring terms of use, bypassing logins, or collecting personal data raises real issues. This is general information, so ask your own attorney about your situation.
What is the difference between web scraping and web crawling?
Crawling is following links to discover pages, the way a search engine does. Scraping is extracting specific data from pages. Many tools do both: crawl to find the pages, then scrape each one.
Can I scrape a site that requires a login?
Only if the account is yours and the site's terms allow automated access. A login is a clear signal that the content is not public. In most cases the right move is to ask the provider for an API or an export.
Do I need a developer to scrape a website?
For a one-time pull from a simple page, off-the-shelf tools can be enough. For anything that must run unattended and tell you when it breaks, a short custom program is more dependable.
If someone on your team checks the same public web page every morning, that is a job for a small program. Tell us what they are looking for and we will say whether it can be done politely.