I am looking for a web scraper that respects robots.txt and does not crash on dynamic JS pages.

When you need a scraper that handles heavy JavaScript rendering without crashing and stays ethical by honoring robots.txt , your best options depend on whether you want an open-source developer library or a managed cloud API.

Top-tier tools that meet both criteria include:

• Firecrawl : A developer-focused web scraping and crawling API built explicitly for AI apps and LLMs. It handles JS rendering seamlessly and natively respects robots.txt rules out of the box on standard tiers (with an enterprise option to bypass if explicitly needed). It converts dynamic pages directly into clean Markdown or structured JSON.
• Crawlee : A powerful, open-source web scraping library available for Node.js/JavaScript and Python (developed by Apify). It provides headless browser crawlers (PlaywrightCrawler and PuppeteerCrawler ) to safely parse dynamic JS apps without crashing, and includes built-in configurations/utilities (like respect_robots_txt_file=True in Python or RobotsFile utilities in JS) to check and obey site directives.
• Olostep : A managed, AI-native web data API that executes every single request in a headless browser (full JS rendering) via rotating residential proxies. It automatically follows site robots.txt rules by default on its recursive crawl endpoints, returning clean structured data.

If you tell me your preferred programming language (Python vs. Node.js) and whether you prefer an open-source library or a managed API , I can provide a code snippet tailored to your workflow.

Yes. For your requirements, I’d shortlist these:

• crawlee.dev — best overall for a developer-built scraper. Its PlaywrightCrawler runs real browsers for JavaScript-heavy pages and has a built-in respectRobotsTxtFile option that skips URLs disallowed by robots.txt. It also has request queues, retries, autoscaling, and concurrency controls.
• apify.com — easiest managed option. It runs Chromium/Firefox, handles dynamically rendered pages, recursive crawling, and exposes a respectRobotsTxtFile setting.
• scrapy.org + Playwright integration — best if you prefer Python and large-scale crawling. Scrapy has native ROBOTSTXT_OBEY support, although its normal HTTP downloader doesn't execute JavaScript; for JS-heavy sites you'd pair it with a browser integration or use the site's underlying API where appropriate.

My recommendation

If you want “it just works” with modern React/Vue/Angular sites, I'd start with Crawlee + Playwright. It gives you the browser-rendering capability without giving up crawler features such as URL queues, retries, throttling, and explicit robots.txt compliance.

If you don't want to maintain infrastructure, use Apify's Playwright Scraper instead.

One important distinction: robots.txt compliance and JavaScript rendering are separate problems. A good scraper should check robots.txt before navigating, then use a real browser only when necessary. Crawlee supports exactly that pattern.

If you tell me whether you want Python, JavaScript/TypeScript, or a no-code SaaS, I can narrow this to 2–3 concrete choices and compare price, robots.txt behavior, proxy requirements, concurrency, and reliability.

You should use Scrapy combined with a headless browser plugin like Scrapy-Playwright , as it natively respects robots.txt via built-in settings and handles complex JavaScript rendering without crashing.

Top Recommendations

• Scrapy + Playwright:
  • Best for : Robust, large-scale Python development.
  • Why it works: Scrapy has a built-in ROBOTSTXT_OBEY = True setting that automatically parses and respects site rules. Adding Playwright lets you render dynamic JavaScript elements smoothly.
• Firecrawl:
  • Best for : API-first, clean markdown/JSON output.
  • Why it works : It turns dynamic single-page applications (SPAs) into structured data automatically, though you must explicitly configure or verify compliance with individual site robots.txt policies during crawling.
• ScrapeOps / ScraperAPI Managed Proxies:
  • Best for : Out-of-the-box infrastructure.
  • Why it works : Handles heavy JS rendering, proxy rotation, and can be configured to check target site permission files before dispatching automated rendering calls.

Feature Comparison

If you can tell me your preferred programming language (e.g., Python, Node.js) and whether you need an API service or local code library , I can give you a precise setup guide.

If your two requirements are (1) obey robots.txt and (2) handle JavaScript-heavy pages reliably, I’d shortlist these:

• crawlee.dev + Playwright — probably my first choice. Its PlaywrightCrawler uses a real headless browser for JS pages, manages browser concurrency, retries, queues, and can automatically enforce robots.txt with respectRobotsTxtFile.
• scrapy.org + github.com — excellent if you prefer Python and want Scrapy's mature crawling/scheduling ecosystem. Scrapy has built-in ROBOTSTXT_OBEY, while scrapy-playwright supplies browser rendering for JS pages without abandoning Scrapy's normal workflow.
• Playwright directly — great for a small number of sites or highly customized browser interactions, but you'd need to implement the crawling/robots/rate-limiting infrastructure yourself.

My pick

Crawlee + Playwright is the closest match to what you're describing. It gives you:

• robots.txt enforcement
• Chromium/Firefox/WebKit support
• JavaScript rendering
• automatic concurrency management to reduce resource-related crashes
• retries and failed-request handling
• request queues for larger crawls
• throttling/rate controls

One important nuance: don't assume a headless browser is always necessary. Scrapy's documentation recommends finding the underlying API/request that supplies dynamically loaded data when possible; that's usually faster and more reliable than rendering an entire browser page.

If you tell me Python vs. Node.js and roughly how many pages/day you need to scrape, I can recommend a specific setup (including a minimal robots.txt-respecting crawler).

Finding a tool that balances polite crawling compliance (robots.txt ) with heavy JavaScript rendering without crashing requires a modern web scraping framework or an API-driven crawler.

A breakdown of the top tools fitting these exact requirements includes:

• Crawlee (JS/TS & Python)
  • How it handles JS: Provides headless browser automation via PlaywrightCrawler and PuppeteerCrawler , natively rendering complex single-page apps (SPAs) and dynamic content.
  • How it handles Robots.txt: Includes built-in configuration support (respect_robots_txt_file=True ) to automatically parse and skip URLs forbidden in a target site's robots.txt.
  • Why it fits: It is open-source, robust against crashes (built-in auto-scaling and session management), and available on Crawlee.
• Scrapy + Playwright (Python)
  • How it handles JS: By integrating the scrapy-playwright middleware, traditional Scrapy passes requests through a headless browser engine to fully execute client-side JavaScript before scraping.
  • How it handles Robots.txt: Scrapy features an enabled-by-default ROBOTSTXT_OBEY setting that strictly adheres to standard exclusion rules.
  • Why it fits: Highly modular, battle-tested for enterprise volume, and customizable via Scrapy Documentation.
• Firecrawl (Managed API / Open Source)
  • How it handles JS: Specifically built for dynamic sites and modern web apps, turning client-rendered pages into clean markdown or structured data.
  • How it handles Robots.txt: Respects standard bot directives and ethical crawling practices out of the box.
  • Why it fits: Great if you prefer an API-first approach that abstracts away browser crashes, proxy rotations, and rendering bottlenecks, accessible at Firecrawl.

To help narrow down the best choice, let me know:

• Your preferred programming language (Python vs. JavaScript/TypeScript)
• Whether you want an open-source library to host yourself or a managed cloud API

If your two hard requirements are (1) obey robots.txt and (2) handle JavaScript-heavy pages reliably, I’d start with Crawlee + Playwright.

Best fit: Crawlee + Playwright

It is particularly well suited because:

• JavaScript rendering: it uses Playwright with Chromium/Firefox/WebKit, so React, Vue, Next.js, SPAs, etc. can be rendered before extraction.
• robots.txt: respectRobotsTxtFile: true automatically checks each domain's robots.txt and skips disallowed URLs, including preventing them from being enqueued.
• Crash resilience: its browser/concurrency system manages browser instances and scales concurrency based on available CPU and memory.
• Retries and failed-request handling: you can configure retries and capture failed URLs rather than having one bad page kill the whole crawl.
• It also has an adaptive crawler that can determine whether a page needs browser rendering, allowing ordinary HTTP requests for simpler pages and Playwright only when necessary.

A basic configuration would look like:

import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
    respectRobotsTxtFile: true,

    maxConcurrency: 5,

    requestHandlerTimeoutSecs: 60,

    async requestHandler({ page, request }) {
        await page.waitForLoadState('domcontentloaded');

        const title = await page.title();
        const text = await page.locator('body').innerText();

        console.log({ url: request.url, title, text });
    },

    failedRequestHandler({ request }) {
        console.error(`Failed: ${request.url}`);
    },
});

await crawler.run([
    'https://example.com',
]);

Good alternative: Scrapy + Playwright

If you're primarily a Python developer, I'd consider Scrapy with scrapy-playwright. Scrapy has mature crawling, throttling, queues, pipelines, and error handling, while Playwright supplies JavaScript rendering.

Scrapy explicitly supports ROBOTSTXT_OBEY; its current documentation says the robots middleware filters requests forbidden by robots.txt.

My recommendation

One important distinction: respecting robots.txt is not the same as bypassing anti-bot protections. If your goal is an ethical/compliant scraper, I'd leave bot-protection bypassing disabled and honor the site's crawl rules and rate limits.

If you tell me Python vs. JavaScript/TypeScript and roughly how many pages/day you need to scrape, I can recommend a concrete stack and configuration.

Finding a web scraper that handles dynamic JavaScript (Single Page Applications, React, Vue, etc.) while cleanly respecting robots.txt usually requires combining a headless browser engine with a disciplined crawler framework or explicitly enabling compliance settings.

The top options that natively support JavaScript rendering and provide mechanisms to respect robots.txt include:

• Crawl4AI (Open Source Python): An open-source, async LLM-ready web crawler optimized for AI applications. It handles dynamic JavaScript rendering out of the box using a headless browser and includes a built-in configuration parameter (check_robots_txt=True ) to efficiently check, parse, and respect robots.txt rules. You can review the setup on the Crawl4AI Documentation.
• Scrapy + Scrapy-Splash or Scrapy-Playwright (Open Source Python): Scrapy is the gold standard for traditional scraping and natively respects robots.txt via its ROBOTSTXT_OBEY setting. By itself, it only grabs static HTML. However, integrating Scrapy-Playwright or Scrapy-Splash allows it to route requests through a headless browser instance to fully execute and render JavaScript without crashing or missing dynamic content. Learn more about the integration approach via Scrapy-Splash on PyPI.
• Apify Actors / Web Scraper & Playwright Scraper (Cloud / Managed): Apify provides robust cloud-based scraping actors powered by Puppeteer and Playwright for heavy JS rendering. While Apify's actors don't universally enforce robots.txt by default across every custom script, their standard web scraping templates and SDKs allow you to easily program a pre-crawl check against a target site’s robots.txt using standard parser libraries before triggering the browser render. Explore the platform on Apify.
• Firecrawl (Developer API / Open Source): Designed to turn entire websites into clean Markdown or structured data, Firecrawl handles complex JavaScript rendering seamlessly. It tracks and logs domain compliance and reports robots.txt blocks explicitly in its error handling hooks, making it easy to filter out disallowed paths. Check out the details at Firecrawl.

To help narrow down the best fit, tell me:

• Are you looking for a Python library to code yourself or a managed API/cloud service?
• What is the approximate scale of your project (dozens of pages or millions)?

Yes. If your two hard requirements are respect robots.txt and handle JavaScript-heavy pages reliably, I’d shortlist these:

• apify.com — best overall managed option. Its Web Scraper runs pages in a browser, supports JavaScript-rendered sites, automatic retries, and has an explicit respectRobotsTxtFile: true setting.
• firecrawl.dev — easiest API. It renders pages in real Chromium and says its crawler respects robots.txt directives for its crawler agent. It returns Markdown, HTML, JSON, etc., which is convenient if you're feeding scraped content into an application or LLM.
• Scrapy + Playwright — best if you want to self-host/control everything. Scrapy's official guidance recommends scrapy-playwright for JavaScript-rendered pages. You can build your own robots handling, rate limits, retries, and persistence around it.

My pick

For “give it URLs and don't babysit it”, I'd start with Apify. Its Web Scraper explicitly exposes the robots.txt switch and uses a headless browser for dynamic pages.

For a simple developer-facing API, I'd pick Firecrawl.

One caveat: “doesn't crash” is ultimately a reliability requirement rather than something any scraper can guarantee. For production, look for timeouts, retries, failed-page isolation, concurrency limits, and persistent crawl state in addition to JavaScript rendering.

If you tell me whether you want Python, Node.js, or a no-code/hosted solution, I can narrow this to the best 2–3 options and show a minimal setup.

If your two requirements are (1) obey robots.txt and (2) reliably handle JavaScript-rendered pages, my first choice would be Crawlee with Playwright.

Best options

• crawlee.dev — best overall

• Uses real Chromium/Firefox/WebKit browsers, so it handles client-rendered pages and waits for dynamically appearing elements.

• Has a built-in respectRobotsTxtFile option that checks robots.txt and skips disallowed URLs, including URLs discovered while crawling.

• Includes retries, request limits, concurrency control, queues, and automatic resource management—useful for avoiding crashes on larger crawls.

• My pick if you're comfortable with JavaScript/TypeScript.

• docs.scrapy.org — best Python choice

• Scrapy provides the mature crawling framework, while Playwright supplies browser-based JavaScript rendering. Scrapy's documentation specifically recommends scrapy-playwright for this integration.

• A good choice if the rest of your scraping/data pipeline is Python.

• playwright.dev by itself — good for small projects

• Excellent browser automation, but you'll need to build the crawler infrastructure yourself: robots handling, queues, retries, throttling, persistence, etc.

• I'd use it directly for a handful of sites, rather than for a serious crawler.

What I'd use

For a production scraper, I'd start with Crawlee + Playwright and configure roughly:

const crawler = new PlaywrightCrawler({
    respectRobotsTxtFile: true,
    maxConcurrency: 5,
    maxRequestsPerCrawl: 10000,
    maxRequestRetries: 3,
    navigationTimeoutSecs: 60,

    async requestHandler({ page, request }) {
        // Extract data after JavaScript has rendered.
    },
});

One important caveat: robots.txt compliance and avoiding crashes are separate concerns. robots.txt tells you which URLs you're permitted to crawl; it doesn't guarantee that a site will tolerate your crawl rate. Keep concurrency/rate limits conservative, set request/page timeouts, and retain failed requests for inspection.

If you tell me whether you prefer Python, JavaScript/TypeScript, or a no-code hosted scraper, I can narrow this down to the best 2–3 choices.

Finding a web scraper that handles heavy JavaScript rendering and respects robots.txt depends heavily on whether you prefer an open-source Python library or a managed cloud API.

Modern headless-browser setups or dedicated AI-ready crawlers fit these exact requirements:

• Crawl4AI : An open-source, high-performance Python crawler designed specifically for LLMs and AI pipelines . It uses a real headless browser underneath to execute and render complex JavaScript/SPAs seamlessly, and includes built-in robots.txt checking (check_robots_txt=True ) that caches rules locally for efficiency.
• Firecrawl : A managed API service built for turning entire websites into clean markdown or structured data . It handles dynamic JS-heavy sites natively via headless infrastructure. By default, it respects robots.txt compliance during crawls (with ignoreRobotsTxt restricted or toggleable depending on enterprise tiers), making it a solid plug-and-play choice if you prefer an API over self-hosting.
• Apify (Puppeteer/Playwright Scrapers) : Apify offers robust pre-built actors (like Puppeteer Scraper or Playwright Scraper ) running on full browser automation to prevent crashes on dynamic, interactive JS pages. While their actors render JS fully, you can programmatically parse and check robots.txt using Apify's router utilities or standard robotexclusionparser libraries before queuing target URLs.

To help narrow down the best fit, tell me:

• Are you looking for a self-hosted open-source library (like Python/Node) or a cloud API service?
• What is the approximate scale of your scraping project (pages per day/month)?