Firecrawl

Why deploy firecrawl on Render?

Firecrawl is an open-source web scraping and crawling API that converts websites into clean, LLM-ready data. It solves the problem of extracting structured content from web pages at scale, handling JavaScript rendering, anti-bot measures, and data cleaning so developers don't have to build custom scrapers.

This template deploys Firecrawl's complete five-service stack—API server, Playwright worker, Postgres queue, RabbitMQ, and Redis—already wired together over Render's private network with auto-generated secure credentials for the database and admin UI. Instead of manually configuring connection strings, setting up persistent storage for stateful services, and debugging inter-service communication, you get the full scraping backend running in one click. The template also auto-redeploys when upstream publishes new images, so you stay current without monitoring releases yourself.

Architecture

What you can build

After deploying, you'll have a fully operational Firecrawl API running at a public URL that can scrape, crawl, and extract content from any website—with Playwright handling JavaScript-rendered pages. The setup includes a persistent Postgres-backed job queue, so you can kick off long-running crawls and poll for results without worrying about lost state. Point your existing Firecrawl SDK code or MCP client at your new endpoint and it works immediately, no API key required.

Key features

• Full stack pre-wired: Deploys API, Playwright worker, Postgres with pg_cron queue, RabbitMQ, and Redis connected over private network with no manual configuration.
• Auto-generated secrets: POSTGRES_PASSWORD and BULL_AUTH_KEY are generated on first deploy, avoiding insecure default credentials.
• Persistent stateful storage: Postgres data and RabbitMQ state survive redeploys via attached SSD disks rather than ephemeral filesystems.
• Chromium Playwright worker: Separate private service runs headless Chromium for JavaScript-rendered page scraping, isolated from the main API.
• Auto-deploys upstream images: Pinned to ghcr.io/firecrawl/*:latest so Render redeploys automatically when upstream publishes new image digests.

Use cases

• Compliance officer scrapes vendor sites without data leaving company infrastructure
• ML engineer crawls documentation to build RAG knowledge base for chatbot
• Startup developer replaces paid Firecrawl API during local testing
• DevOps team routes scraper through proxy for stable egress IP

What's included

Prerequisites

• OpenAI API Key: Enables AI features like JSON extraction, the extract endpoint, and structured agent output for Firecrawl.
• Proxy Server URL: Outbound HTTP proxy URL to route scrapes through a proxy for stable egress IP.
• Proxy Username: Basic authentication username for the outbound HTTP proxy.
• Proxy Password: Basic authentication password for the outbound HTTP proxy.
• SearXNG Endpoint: Self-hosted SearXNG URL to replace Google as the search backend for the search endpoint.

Next steps

1. Open https://.onrender.com/v0/health/liveness in your browser — You should see {"status":"ok"} confirming the API and all backend services are connected
2. Test a scrape by running curl -X POST 'https://.onrender.com/v2/scrape' -H 'Content-Type: application/json' -d '{"url": "https://firecrawl.dev"}' — You should receive a JSON response containing the scraped markdown content within 30 seconds
3. Open the queue admin UI at https://.onrender.com/admin/<BULL_AUTH_KEY>/queues (find BULL_AUTH_KEY in your firecrawl-api environment variables) — You should see the Bull dashboard listing active queues with your test scrape job completed

Resources

• Firecrawl official site