adbar / trafilatura

What Was Tested

Python library mode — the documented path for content extraction.

Environment:

• macOS M-series
• Python 3.13
• repoverifiertest user (clean environment)
• pip3 install trafilatura — no virtual environment needed

Install

pip3 install trafilatura

Clean install. No errors. No dependency conflicts on Python 3.13.

What Worked

Plain text extraction:

import trafilatura
downloaded = trafilatura.fetch_url('https://en.wikipedia.org/wiki/Python_(programming_language)')
text = trafilatura.extract(downloaded)

Returned clean article content. Navigation, sidebars, and boilerplate correctly stripped.

Markdown output:

text = trafilatura.extract(downloaded, output_format='markdown')

Returned structured markdown with headers, bold text, and tables preserved — directly usable as LLM context.

Real article extraction:

downloaded = trafilatura.fetch_url('https://paulgraham.com/writes.html')
text = trafilatura.extract(downloaded, output_format='markdown')
# LENGTH: 3121

Full article extracted cleanly. No noise.

Honest Findings

Returns None on paywalled pages: TechCrunch article pages returned None. This is correct behavior — trafilatura does not bypass paywalls or render JavaScript.

Homepage extraction is minimal by design: TechCrunch homepage returned 517 characters — only content trafilatura was confident was "main content." Navigation teasers and promotional blocks were correctly excluded. This is the intended behavior for an LLM pipeline — you scrape article URLs, not homepages.

Works best on: open, text-based pages — blogs, documentation, Wikipedia, research articles, product pages.

Does not work on: paywalled content, JavaScript-rendered SPAs, pages requiring authentication.

What Was Not Tested

• CLI mode (trafilatura -u "https://...")
• Crawling mode (sitemap + batch processing)
• Metadata extraction (trafilatura[metadata])
• XML/CSV output formats

Verdict: SOLID

The core claim — extract main text from web pages into LLM-ready format — works exactly as documented. Three-line install, three-line usage. No API key, no server, no Docker. Runs natively on macOS with Python 3.13.