← All Posts

I built a chatbot that refuses to recommend repos it hasn't personally tested

Balaji Loganathan · September 13, 2026

The problem

I kept losing hours to AI/dev-tooling repos that looked great on GitHub — good README, decent star count — and then didn't do the thing the README said, or broke on setup. So I started manually testing repos before recommending them anywhere, and eventually turned that into a small site: repoverifier.dev.

The rule is simple: clone the repo on a clean macOS user with no prior config, follow the README exactly as written, test the actual core claim (not just "does npm install succeed"), and publish what happened — including the parts where the README was wrong. Every review lands on one of five verdicts: HIDDEN GEM, SOLID, PARTIAL, OVERHYPED, DANGEROUS. No sponsorships, no pay-to-submit — I pick what gets reviewed.

The part worth writing up: constraining the chatbot to what's verified

The homepage is a chat interface ("What are you trying to build?") that recommends repos based on what you're trying to do. The obvious failure mode for that kind of feature is a chatbot that "helpfully" recommends something outside the reviewed set — which would silently reintroduce the exact hype problem the site exists to fix.

So retrieval is fenced:

  • Every review gets embedded (text-embedding-3-small) into a rv_embeddings table on Postgres/pgvector at publish time.
  • A chat query gets embedded the same way, then matched via cosine similarity (embedding <=> query_embedding) — no open-web RAG, no general knowledge fallback beyond a small static KB used only when the embeddings provider is unreachable.
  • Generation runs on Groq's llama-3.3-70b-versatile — cheap enough to run this as a free feature — but the model only ever sees retrieved review content in its context. If nothing in the corpus matches, it says so instead of guessing.
The tradeoff: the assistant can't discuss a repo I haven't reviewed yet, which is a real limitation as the corpus grows past ~23 reviews. I'd rather have "I haven't tested that one" than a plausible-sounding hallucination about a tool's setup steps.

What I'd want feedback on

  • Does a 5-verdict taxonomy (HIDDEN GEM / SOLID / PARTIAL / OVERHYPED / DANGEROUS) read clearly at a glance, or is that one bucket too many?
  • Is retrieval-fenced-to-verified-content a pattern worth generalizing for other "trust" products, or is it just right-sized for a small review corpus?
Site: https://repoverifier.dev — no signup wall, reviews are public.