What Was Tested
Python library mode — the documented path for content extraction.
Environment:
- macOS M-series
- Python 3.13
- repoverifiertest user (clean environment)
- pip3 install trafilatura — no virtual environment needed
Install
pip3 install trafilatura
Clean install. No errors. No dependency conflicts on Python 3.13.
What Worked
Plain text extraction:
import trafilatura
downloaded = trafilatura.fetch_url('https://en.wikipedia.org/wiki/Python_(programming_language)')
text = trafilatura.extract(downloaded)
Returned clean article content. Navigation, sidebars, and boilerplate correctly stripped.
Markdown output:
text = trafilatura.extract(downloaded, output_format='markdown')
Returned structured markdown with headers, bold text, and tables preserved —
directly usable as LLM context.
Real article extraction:
downloaded = trafilatura.fetch_url('https://paulgraham.com/writes.html')
text = trafilatura.extract(downloaded, output_format='markdown')
# LENGTH: 3121
Full article extracted cleanly. No noise.
Honest Findings
Returns None on paywalled pages: TechCrunch article pages returned None.
This is correct behavior — trafilatura does not bypass paywalls or render JavaScript.
Homepage extraction is minimal by design: TechCrunch homepage returned 517
characters — only content trafilatura was confident was "main content." Navigation
teasers and promotional blocks were correctly excluded. This is the intended behavior
for an LLM pipeline — you scrape article URLs, not homepages.
Works best on: open, text-based pages — blogs, documentation, Wikipedia,
research articles, product pages.
Does not work on: paywalled content, JavaScript-rendered SPAs, pages requiring
authentication.
What Was Not Tested
- CLI mode (
trafilatura -u "https://...")
- Crawling mode (sitemap + batch processing)
- Metadata extraction (
trafilatura[metadata])
- XML/CSV output formats
Verdict: SOLID
The core claim — extract main text from web pages into LLM-ready format —
works exactly as documented. Three-line install, three-line usage.
No API key, no server, no Docker. Runs natively on macOS with Python 3.13.