Firecrawl allows you to turn entire websites into LLM-ready markdown
Firecrawl is an API service that takes a URL, crawls it, and converts it into clean markdown. We crawl all accessible subpages and give you clean markdown for each. No sitemap required.
​How to use it?
We provide an easy to use API with our hosted version. You can find the playground and documentation here. You can also self host the backend if you’d like.
Check out the following resources to get started:
- API: Documentation
- SDKs: Python, Node, Go, Rust
- LLM Frameworks: Langchain (python), Langchain (js), Llama Index, Crew.ai, Composio, PraisonAI, Superinterface, Vectorize
- Low-code Frameworks: Dify, Langflow, Flowise AI, Cargo, Pipedream
- Others: Zapier, Pabbly Connect
- Want an SDK or Integration? Let us know by opening an issue.
Self-host: To self-host refer to guide here.
​API Key
To use the API, you need to sign up on Firecrawl and get an API key.
​Features
- Scrape: scrapes a URL and get its content in LLM-ready format (markdown, structured data via LLM Extract, screenshot, html)
- Crawl: scrapes all the URLs of a web page and return content in LLM-ready format
- Map: input a website and get all the website urls - extremely fast
​Powerful Capabilities
- LLM-ready formats: markdown, structured data, screenshot, HTML, links, metadata
- The hard stuff: proxies, anti-bot mechanisms, dynamic content (js-rendered), output parsing, orchestration
- Customizability: exclude tags, crawl behind auth walls with custom headers, max crawl depth, etc…
- Media parsing: pdfs, docx, images.
- Reliability first: designed to get the data you need - no matter how hard it is.
- Actions: click, scroll, input, wait and more before extracting data
You can find all of Firecrawl’s capabilities and how to use them in our documentation
​Crawling
Used to crawl a URL and all accessible subpages. This submits a crawl job and returns a job ID to check the status of the crawl.