What Was Tested
Python library mode — the documented path for content extraction.
Environment:
- macOS M-series
- Python 3.13
- repoverifiertest user (clean environment)
- pip3 install trafilatura — no virtual environment needed
Install
pip3 install trafilatura
Clean install. No errors. No dependency conflicts on Python 3.13.
What Worked
Plain text extraction:
import trafilatura
downloaded = trafilatura.fetch_url('https://en.wikipedia.org/wiki/Python_(programming_language)')
text = trafilatura.extract(downloaded)
Markdown output:
text = trafilatura.extract(downloaded, output_format='markdown')
Real article extraction:
downloaded = trafilatura.fetch_url('https://paulgraham.com/writes.html')
text = trafilatura.extract(downloaded, output_format='markdown')
# LENGTH: 3121
Honest Findings
Returns None on paywalled pages: TechCrunch article pages returned None. This is correct behavior — trafilatura does not bypass paywalls or render JavaScript.
Homepage extraction is minimal by design: TechCrunch homepage returned 517 characters — only content trafilatura was confident was "main content." Navigation teasers and promotional blocks were correctly excluded. This is the intended behavior for an LLM pipeline — you scrape article URLs, not homepages.
Works best on: open, text-based pages — blogs, documentation, Wikipedia, research articles, product pages.
Does not work on: paywalled content, JavaScript-rendered SPAs, pages requiring authentication.
What Was Not Tested
- CLI mode (
trafilatura -u "https://...") - Crawling mode (sitemap + batch processing)
- Metadata extraction (
trafilatura[metadata]) - XML/CSV output formats
Verdict: SOLID
The core claim — extract main text from web pages into LLM-ready format — works exactly as documented. Three-line install, three-line usage. No API key, no server, no Docker. Runs natively on macOS with Python 3.13.