conorbronsdon / avoid-ai-writing

The claim

> "Skill that audits and rewrites content to remove AI writing patterns. > Use it with your favorite agents including Claude Code, OpenClaw, Codex, > and Hermes."

The README backs this with a demo claiming "35+ AI tells" caught in one paragraph, and a bundled deterministic scoring engine separate from the prompt-only skill.

What it is

A dual-artifact repo: SKILL.md (the agent-facing prompt, 69 documented pattern categories) plus detector/ (a standalone Node scoring engine, zero dependencies, 54 type categories — intentionally a different count from the prompt catalog, mapped in detector/CATEGORIES.md). Also ships an MCP server wrapper, Cursor rule port, and a style-guide compatibility layer for external linters like Vale.

Environment

• Cloned fresh from main; detector/patterns.js v3.33.2
• Node v20.19.6 (repo floor: >=18, satisfied)

What we verified directly

1. Test suite: passes clean, and it's real regression coverage, not padding. npm test runs 8 suites — pattern fixtures, categories anti-drift contract, preservation validator, corpus helpers, style checker, quote normalizer — all green. Many test names cite specific GitHub issue numbers (#62, #67, #69, #73, #77, #107, #109, #123), confirming these were written against real bugs, not synthesized for show.

2. Preservation validator: all 27 cases pass. detector/validate.js correctly errors on edited code blocks, altered file paths, dropped URLs, touched frontmatter, and changed table cells, while passing ordinary prose rewrites and cosmetic fixes (heading case, emoji removal) as warnings rather than hard errors. This is the mechanism that would catch a rewrite corrupting structured content — verified working, not just documented.

3. Self-published limitations: unusually candid. PROOF.md runs the detector against the project's own docs and reports the result including unflattering findings — a real detector bug (em-dash carve-out gaps, now fixed as #67) and, more importantly, a false-positive study against 875 human / 779 machine paragraphs showing paragraph-level ROC-AUC of 0.501 (chance) and 0.623 at document level, with the explicit conclusion that "the composite score cannot reliably separate machine text from human text." One inaccuracy worth noting: the README's About text says PROOF.md documents "two defects"; the file actually documents four findings, one of which is a real detector bug and the rest are honest writing-quality critiques — a minor undersell, not an oversell.

4. Install path: works, but the version floor is a soft warning, not a hard gate. The documented npx [email protected] add ... command requires Node >=22.20.0; on Node 20.19.6 it printed an EBADENGINE warning and then completed installation successfully anyway to both .agents/skills/ and .claude/skills/. Worth documenting precisely rather than assuming the warning meant failure.

5. Detector recall against fresh text: mixed, and this is the finding that matters most. We built a synthetic AI-sounding paragraph covering several claimed pattern categories at once, then isolated each one against both our own phrasing and the README's own literal example text:

| Pattern | Our phrasing | Their own exact demo phrasing | |---|---|---| | Chatbot closer ("feel free to reach out...") | Missed (paraphrased) | Fired correctly, severity: critical | | Chatbot opener ("Certainly!") | Missed | Also missed, even above the word-count floor, in a real sentence | | Emoji-bulleted list with bold labels | Missed | Also missed — tested with the README's own demo text verbatim, scored 0 / Clean | | Vague attribution, significance inflation, tier-1 vocabulary, em dash, "In order to" | Caught correctly | — |

The closer-pattern gap is explainable and fair: the detector appears to match specific known phrasings rather than the semantic category, so a worded-differently chatbot sign-off slips through. That's a legitimate, disclosable brittleness, not a defect.

The emoji-bullet miss is a different class of finding: it reproduces against the project's own flagship README example, word for word. The demo that's meant to prove the tool's range doesn't fully hold up against the shipped code.

Caveats

• The detector is a pattern-density diagnostic, not an AI-authorship classifier — the project says this itself, clearly, in PROOF.md. A review or any downstream use that treats the score as "percent chance this was written by AI" would misrepresent what the maintainer explicitly disclaims.
• Vocabulary-tier and structural rules (dashes, filler phrases, attribution, inflation) tested reliably in our sample. Phrase-anchored rules (chatbot artifacts) are narrower than the README's framing suggests, and at least one — the emoji-bulleted list pattern — did not fire on the project's own canonical example.
• Everything that runs as code (tests, preservation validator, self-scan, install) is machine-verified and reproducible. The one gap is between the README's illustrative prose and the actual detector's current behavior on that same illustrative text — a documentation/code sync issue, not a broken engine.

Verdict: SOLID (with caveats)

Real code, a genuinely deep regression suite tied to real issues, a working preservation guard, and unusually honest self-published limits — that's a stronger evidentiary floor than most repos in this category. It's not a clean SOLID because the flagship demo doesn't fully reproduce against the shipped detector, and one advertised pattern (chatbot openers) missed its own signature phrase in direct testing. Genuinely useful tool, oversold in one specific, documented, testable way.