We are launching Crawlee for Python, an open-source library for web scraping and browser automation. Quickly scrape data, store it, and avoid getting blocked with auto-generated human-like fingerprints, headless browsers, and smart proxy rotation.
Crawlee is a robust web scraping and browser automation library for Python. It enables developers to build reliable crawlers quickly and efficiently.
Crawlee is developed by experienced web scraping professionals who use it daily for large-scale data extraction projects.
pipx run crawlee create my-crawler
pip install 'crawlee[playwright]'
playwright install
import asyncio
from crawlee.playwright_crawler import PlaywrightCrawler, PlaywrightCrawlingContext
async def main():
crawler = PlaywrightCrawler(
max_requests_per_crawl=5,
headless=False,
browser_type='firefox',
)
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
await context.enqueue_links()
data = {
'url': context.request.url,
'title': await context.page.title(),
'content': (await context.page.content())[:100],
}
await context.push_data(data)
await crawler.run(['https://crawlee.dev'])
await crawler.export_data('results.json')
if __name__ == '__main__':
asyncio.run(main())
Seamlessly extract data from any website without coding.
Easy web scraping for anyone.
Create APIs where they don't exist. With kimono you can quickly…
Saldor extracts the best web data for LLMs.
Transform websites into customizable APIs instantly.
Painless data extraction and web automation
Crawl, parse, and scale web data seamlessly
The simplest API for web scraping without getting blocked.
Browser automations powered by AI.
Data scraping made simple
Match with like-minded professionals for 1:1 conversations
Go from Slack Chaos to Clarity in Minutes
Personalize 1000s of landing pages in under 30 mins
The first LLM for document parsing with accuracy and speed
AI Assistants for SaaS professionals
AI-powered phone call app with live translation
Delightful AI-powered interactive demos—now loginless
AI Motion Graphics Copilot
Pop confetti to get rid of stress & anxiety, 100% AI-free
Smooth payments for SaaS