A reliable web scraper is not a prompt that says “extract everything.” It is a controlled data pipeline: confirm the source permits the use, inspect its stable structure, request pages politely, validate extracted records, stop pagination safely, deduplicate writes, and make failures observable.
AI is most useful for drafting a schema, suggesting selectors from approved sample HTML, generating fixtures, and explaining parser failures. Deterministic code should still control which origins are allowed, what fields are accepted, how often requests run, which records are stored, and when the job stops.
QUICK ANSWER
How do you create a web scraper with AI?
Choose a source you are permitted to automate, define the exact records you need, and give AI only approved sample HTML to draft selectors and fixtures. Keep URL allowlists, robots checks, rate limits, response validation, pagination bounds, deduplication, and storage deterministic. Test changed markup, invalid records, 429 retries, and the published job before scheduling it.
Prepare permission, scope, and a test fixture
Do not start with selectors. Start with the source owner’s rules, the business purpose, and a small approved fixture that can be tested without repeatedly hitting a live site.
- A permitted source and written use case: Prefer an official API, feed, export, or data partnership. For HTML access, review the site terms, its robots.txt rules, documented request limits, copyright, privacy, and applicable law. Robots rules are crawler instructions, not access authorization.
- One record contract: Name every field, type, required or optional status, unit, normalization rule, source key, and rejection reason. Include sourceUrl, fetchedAt, parserVersion, and jobId so a stored row can be traced.
- Approved local fixtures: Save small, rights-cleared HTML or JSON fixtures for a normal page, missing field, changed layout, empty result, duplicate record, and next-page loop. Remove personal and secret data before giving samples to an AI model.
- A rate and retention owner: Decide the maximum requests per origin, concurrency, schedule, retention period, deletion path, and incident contact. The FTC data-minimization guide recommends keeping only the personal information the business actually needs and only for as long as necessary.
Choose the least fragile source interface
The same public information may be available through an API, a static HTML response, or a JavaScript-rendered interface. Choose the simplest interface the source owner supports; a headless browser is not the default.
| Approach | Best for | Tradeoff |
|---|---|---|
| Official API, feed, or export | Stable fields, documented pagination, explicit quotas, and a supported integration relationship. | May need credentials, approval, or a paid plan, but normally has the clearest contract and lowest parser maintenance. |
| Direct HTML request plus deterministic parser | Permitted public pages whose required fields appear in the server response with stable semantic markers. | Markup can change without notice, so fixtures, field validation, provenance, and change alerts are mandatory. |
| Browser rendering | Permitted workflows where the required public content only exists after supported client-side rendering. | Consumes more compute, creates more failure modes, and must never be used to defeat authentication, CAPTCHAs, paywalls, or access controls. |
Recommended:Use an official API, feed, or export when one exists. Otherwise begin with a bounded direct request and deterministic parser. Add browser rendering only after confirming it is permitted and technically necessary. Use AI to draft and review the parser, not to decide access or invent missing values at runtime.
Build the AI-assisted scraper in 10 steps
This workflow uses a fictional permitted product catalog and makes every stage observable, bounded, and recoverable.
STEP 01
Confirm permission and write the source policy
Record why the job may access the source before generating a request or selector.
Write the allowed origins and paths, user-agent identity, schedule, concurrency, maximum pages, data fields, retention, and stop contact. Read the Robots Exclusion Protocol for crawler rules, but do not mistake an allowed robots path for permission or a legal conclusion.
Check source terms and the planned downstream use. The U.S. Copyright Office overview explains that original expression can be protected even though ideas, procedures, and methods are not. Obtain specialist legal review when the source, jurisdiction, personal data, or reuse rights are unclear.
Expected result: The job has a named owner and an explicit source policy that states what may be requested, stored, reused, and deleted.
Verify it: Have a second reviewer trace one planned URL and one output field back to the source policy, terms review, retention rule, and business purpose.
STEP 02
Inspect the source and freeze the record contract
Determine where the authoritative fields and next-page signal actually live before asking AI for code.
Inspect the network response and page source using a permitted fixture. Identify a stable record container, field markers, canonical source ID or URL, pagination signal, currency or unit semantics, and the difference between absent, empty, and invalid values.
Give AI the fixture plus the exact record contract. Ask it to propose selectors, normalization rules, rejection reasons, and tests. Review every selector against the fixture and remove any field the model inferred rather than observed.
type ScrapedRecord = {
sourceKey: string
sourceUrl: string
title: string
priceCents: number | null
currency: string | null
fetchedAt: string
parserVersion: string
jobId: string
}
type RejectedRecord = {
sourceUrl: string
reason: 'missing-key' | 'invalid-price' | 'unexpected-shape'
parserVersion: string
}Expected result: A sample record, rejected-record shape, selectors, and pagination rule all point to observed source evidence rather than model guesswork.
Verify it: Parse the normal and changed-layout fixtures by hand; confirm required fields reject cleanly and optional missing fields stay null rather than becoming invented text.
STEP 03
Build a bounded and safe request layer
Validate the destination before every request and bound time, redirects, bytes, and content type.
Parse URLs with the platform URL API, require HTTPS, allow only reviewed hostnames and paths, and reject credentials in URLs. Apply the OWASP SSRF guidance: block loopback, private, link-local, and cloud-metadata destinations; resolve and validate DNS; and revalidate every redirect target.
Set a clear user agent, request timeout, redirect limit, response-byte ceiling, and accepted content types. The Fetch API guide notes that fetch can resolve even for HTTP error responses, so check status before reading or parsing the body.
Expected result: Only reviewed public HTTPS targets can be requested, and a slow, oversized, redirected, private, or unexpected response fails before parsing.
Verify it: Test the approved fixture origin plus HTTP, user-info, loopback, private IPv4 and IPv6, metadata, redirect-to-private, oversized, timeout, and wrong-content-type cases.
STEP 04
Parse deterministically and validate every field
Convert the approved response into records with explicit null, rejection, and change behavior.
Parse a known container, extract text and attributes, normalize whitespace and units, then validate types and bounds. Never execute source scripts. Treat the page, metadata, and embedded instructions as untrusted data, including text that tells an AI agent to ignore rules or reveal secrets.
Store structured fields, not executable markup. If a product interface must show source text, use context-appropriate output encoding; follow the OWASP XSS prevention guidance and sanitize only when authorized HTML rendering is genuinely required.
Expected result: Each source container becomes one valid record or one bounded rejection with a reason; no missing value is hallucinated and no source markup executes.
Verify it: Run normal, missing-field, malformed-number, duplicate, malicious-markup, prompt-injection text, and changed-selector fixtures and compare exact accepted and rejected output.
STEP 05
Paginate with stop conditions and loop detection
Follow only a reviewed next-page signal and stop before the scraper can wander or repeat forever.
Prefer an API cursor or a rel=next link. Resolve the next URL against the current approved origin, reapply the full request policy, and keep a set of visited cursors or canonical URLs. Set maximum pages, records, elapsed time, and consecutive empty pages.
Do not guess numeric URLs beyond observed navigation. Stop on a missing next signal, repeated cursor, repeated canonical page, maximum bound, policy change, or validation-error threshold.
Expected result: The job walks the intended finite sequence once and reports the exact stop reason.
Verify it: Run fixtures for one page, three pages, missing next, next to another origin, repeated cursor, duplicate page, empty page, and the maximum-page boundary.
STEP 06
Deduplicate and store records with provenance
Make retries and recurring runs update the intended record instead of multiplying rows.
Choose a stable source key supplied by the source when possible. Otherwise use a reviewed canonical URL plus a source-specific identifier. Use a content hash for change detection, not as the only identity when normal edits should update one record.
Upsert by source and sourceKey. Store sourceUrl, fetchedAt, sourceUpdatedAt when supplied, contentHash, parserVersion, and jobId. Keep raw responses only when the use case and policy require them, with a short retention period and restricted access.
Expected result: Repeating the same page does not create duplicate records, while a real source change creates one traceable update.
Verify it: Process the same fixture twice, a reordered fixture, an updated price, and two sources with the same local ID; inspect row counts, versions, hashes, and provenance.
STEP 07
Add polite rate limits, retries, and a dead-letter path
Retry only transient failures and let the source server slow the job down.
Use a per-origin concurrency and request-rate budget. On 429, honor the server’s Retry-After value when present; RFC 6585 defines 429 as Too Many Requests and permits Retry-After. For transient 429 and selected 5xx or network errors, use capped exponential backoff with jitter.
Do not repeatedly retry 401, 403, 404, robots denial, invalid content, schema rejection, or policy failure. Record the bounded failure with job ID, source URL, attempt count, status, parser version, and next operator action, without storing private payloads in logs.
Expected result: A temporary failure slows and eventually recovers or lands in a review queue; a permanent or policy failure stops immediately and visibly.
Verify it: Simulate Retry-After seconds and dates, 429 without a header, two 503s then success, repeated timeout, 403, schema rejection, and exhausted attempts; check timing and final state.
STEP 08
Run fixture, boundary, retry, and production-smoke tests
Prove both useful extraction and safe refusal before scheduling recurring work.
Run the parser entirely against local approved fixtures first. Then run a single bounded request to a source you are permitted to test and compare fields, stop reason, record count, request count, and stored provenance with the written expectations.
Test the source policy as a security boundary. Include a changed DNS result, redirect, disallowed path, new content type, oversized body, markup injection, invalid record, pagination loop, duplicate run, and retry exhaustion.
Expected result: The scraper accepts the intended source and fixture shapes, refuses unsafe or invalid inputs, and produces repeatable stored output with a complete run summary.
Verify it: Save a test matrix with fixture version, parser version, expected status, observed status, request count, record count, stop reason, and reviewer, excluding secrets and personal payloads.
STEP 09
Publish and schedule the job on Playcode
Separate a manual preview run from the production schedule and verify the public runtime deliberately.
Build the backend job, database tables, operator view, and alert endpoint, then publish over HTTPS under current Playcode Cloud plan and runtime limits. Keep source credentials and alert secrets in server-side configuration, never in browser code, URLs, fixtures, logs, or generated images.
Start with a manual one-page preview that writes to a staging table. Verify the output and source impact, create a supported recovery point, then enable the smallest useful schedule. Do not make the schedule more frequent than the source or business need requires.
Expected result: The published job runs on demand over HTTPS, writes traceable records, exposes a bounded operator summary, and has a known recovery point before scheduling.
Verify it: From the published runtime, execute one permitted synthetic or approved source run, reopen the stored records, inspect the run summary, test the alert channel, and locate the recovery point.
STEP 10
Monitor source drift, volume, failures, and deletion
Operate the scraper as a relationship with a changing source, not a script that can be forgotten.
Track run status, requests, 429s, retry delays, bytes, accepted and rejected records, duplicates, stop reason, duration, parser version, and last successful source sample. Alert on zero records, sudden volume change, validation-error rate, repeated selectors failing, policy change, and a stale successful run.
Pause automatically when the source contract changes, the validation-error threshold trips, or the allowed destination fails revalidation. Preserve a small rights-cleared failure fixture, fix the parser, replay only failed bounded work, and execute the retention and deletion policy on schedule.
Expected result: An operator can tell whether the job is healthy, polite, current, and within policy, then pause, repair, replay, or delete without guessing.
Verify it: Trigger a zero-record alert, a selector-drift alert, a 429 slowdown, a paused policy state, a bounded replay, and a scheduled deletion; confirm each event and operator action is recorded.
What the result can look like
The minimum scraper test matrix
A scraper is safe to schedule only when useful extraction, invalid-input refusal, bounded retry behavior, and the published runtime have each been observed.
| Test | Scenario | Expected result |
|---|---|---|
| happy path | Parse a permitted two-page fixture with four valid records, one duplicate, stable source keys, and an explicit final-page signal. | Four records are upserted once with normalized fields and provenance; the job stops on the final-page signal with a complete summary. |
| invalid input | Try a private destination, cross-origin redirect, disallowed path, wrong content type, oversized response, missing source key, and executable source markup. | Each case fails before an unsafe request, write, or render; the job records a bounded reason without secrets or complete private payloads. |
| retry | Return 429 with Retry-After, then one 503, then success; separately return a permanent 403 and a repeated timeout until the attempt limit. | Transient failures wait and retry within budget; 403 stops immediately; exhausted work enters the review path once without duplicate records. |
| production smoke | Run one permitted bounded job from the published Playcode runtime, reopen its records, inspect the summary, and trigger the configured alert channel. | HTTPS runtime, job, storage, provenance, operator view, alerting, and recovery reference all work with the production configuration. |
Common scraper failures and how to diagnose them
Start with the run ID, source policy version, exact stop reason, request count, response metadata, parser version, and fixture result. Do not start by logging the full source body.
| Symptom | Likely cause | Check | Fix |
|---|---|---|---|
| The job succeeds but stores zero records | The source markup changed, the content moved behind client rendering, or the record container selector is now empty. | Compare response status, content type, bounded sample hash, container count, and parser version with the last successful run and local fixture. | Pause the schedule, obtain a permitted updated fixture, revise and review selectors, add a regression case, then run a one-page preview. |
| A recurring run creates duplicate rows | Identity uses a mutable title or content hash, or the upsert uniqueness constraint does not include the source namespace. | Compare sourceKey, canonical source URL, content hash, job ID, and database uniqueness behavior across the duplicate rows. | Choose a stable source-scoped key, enforce it at the write boundary, merge only with an explicit policy, and replay the fixture twice. |
| The scraper receives repeated 429 responses | Concurrency or schedule exceeds the source budget, Retry-After is ignored, or multiple workers share an uncoordinated limit. | Inspect per-origin request timestamps, worker count, Retry-After values, and the source’s current documented limits. | Pause or reduce traffic, centralize the per-origin budget, honor Retry-After, add jitter, and ask the source owner about a supported feed or quota. |
| Pagination repeats the same page forever | The next-page URL or cursor is unchanged, normalized incorrectly, or generated from a guessed number rather than observed navigation. | Compare visited canonical URLs, cursors, page hashes, next signals, and the configured page and elapsed-time bounds. | Stop on repeated URL, cursor, or content; follow only the observed next signal; keep hard page, record, and time ceilings. |
| The fetcher reaches an unexpected internal destination | The URL policy checked only the input string and did not revalidate DNS resolution or redirects. | Record the canonical hostname, resolved public IP class, redirect chain, and policy decision without requesting or exposing internal content. | Stop the job, rotate any exposed credentials, investigate impact, and enforce scheme, host, IP-range, DNS, port, and redirect validation before retrying. |
| The dashboard renders source HTML or script | Raw scraped markup entered an unsafe browser sink instead of being stored and displayed as data. | Trace the stored field to the render context and inspect framework escape behavior, sanitization, and any unsafe HTML escape hatch. | Render normalized text through safe framework bindings, remove raw HTML where unnecessary, sanitize authorized HTML, patch dependencies, and add malicious-markup tests. |
Deploy and operate the scraper on Playcode
Deploy
Publish the backend job, database, operator view, and alert endpoint first. Run a manual one-page preview, review source impact and stored fields, create a supported recovery point, then enable the smallest useful schedule under current plan and runtime limits.
Separate staging and production tables or source namespaces. Pin the parser and policy version for each run so a rollback or replay does not silently mix two extraction contracts.
Monitor
Track requests and bytes per origin, 429s, Retry-After delays, accepted and rejected records, duplicates, stop reason, duration, parser version, source-policy version, and last successful bounded sample.
Alert on zero output, sudden volume change, validation-error rate, repeated selector failure, policy or robots change, retry exhaustion, storage failure, and a stale successful run. Keep source bodies, secrets, and personal data out of routine logs.
Recover
Pause first. Preserve a minimal rights-cleared failure fixture, fix and test the parser locally, run a one-page preview, then replay only the failed bounded range using the same stable source keys.
Use row-level repair for a known record set. Use a supported whole-project recovery point only after deciding how newer database writes will be reconciled, since restoring code and data together can move valid records backward.
Security, privacy, and source-respect checklist
Publicly reachable does not mean unrestricted. Build explicit technical and review gates around where the job can connect, which data it can keep, and how untrusted source content can influence the system.
- Never defeat authentication, CAPTCHAs, paywalls, robots rules, rate controls, or another access control. Stop and use an approved API, feed, export, or source-owner agreement.
- Allow only reviewed HTTPS origins and paths; block URL credentials, private and special IP ranges, metadata endpoints, unsafe ports, DNS rebinding, and unvalidated redirects.
- Treat source HTML, JSON, metadata, links, and embedded instructions as untrusted data. Do not let scraped text change system prompts, tool permissions, destination policy, or secret access.
- Check status, content type, response size, time, redirects, and record shape before parsing or writing. Bound pages, records, retries, concurrency, duration, and retained raw data.
- Render extracted values as encoded text by default. Do not execute source scripts or inject raw scraped HTML into an operator interface.
- Minimize personal and copyrighted material, record provenance and permitted use, define retention and deletion, restrict access, and obtain specialist review where rights or obligations are unclear.
- Keep credentials and alert secrets in server-side configuration. Never place them in browser code, prompts, fixtures, URLs, logs, screenshots, generated images, or stored source records.
AI web scraper questions
What should AI do in a web scraper?
Use AI to draft a record schema, suggest selectors from approved fixtures, generate tests, explain parser failures, and propose normalization rules. Keep destination policy, access decisions, request budgets, field validation, pagination bounds, deduplication, and writes deterministic and reviewable.
Does robots.txt give permission to scrape a website?
No. RFC 9309 says robots rules are not access authorization. Honor applicable rules, then separately review site terms, source-owner guidance, copyright, privacy, contract, and applicable law. Obtain legal review when the planned access or reuse is uncertain.
Should I use an API or scrape HTML?
Prefer an official API, feed, export, or partnership because it usually has stable fields, documented pagination, quotas, and clearer permission. Use HTML only when it is permitted and needed, with fixtures, schema validation, change alerts, and a maintenance owner.
When does a scraper need a headless browser?
Only when permitted required content is unavailable in an official interface or direct server response and genuinely depends on client rendering. A browser adds compute and failure modes and must never be used to defeat login, CAPTCHAs, paywalls, or access controls.
How do I stop duplicate scraped records?
Use a stable source-scoped key and enforce it with an upsert or database uniqueness constraint. Keep a content hash for change detection, plus source URL, fetched time, parser version, and job ID for provenance. Test the same fixture twice.
How should a web scraper handle 429 errors?
Reduce or pause traffic, honor Retry-After when present, and use capped exponential backoff with jitter for a bounded number of transient attempts. Coordinate one per-origin limit across workers. Do not rotate identities or proxies to defeat the source limit.
Can Playcode host a recurring web scraper?
Playcode Cloud currently supports backend processes, a persistent database, background jobs and schedulers, HTTPS, logs, snapshots, and rollback under plan and runtime limits. You still own source permission, parser correctness, request budgets, monitoring, retention, and incident response.
Build the bounded pipeline
Turn the approved source contract into an operated app
Describe the source policy, record schema, fixtures, request budget, stop conditions, tests, and operator view. Playcode can help build the deterministic workflow and run it on Playcode Cloud under current limits.
Explore the AI App BuilderNo credit card required. AI credits included.