Methodology
What gets measured, how a grade is arrived at, what this method cannot see, and how to argue with it.
What a grade describes
A grade applies to one plugin at one commit, and it describes the artifact, not the people who wrote it. It is an opinion formed from published code using the rubric below, which is reproducible: the corpus is pinned, every finding cites a file and a line, and a script in the repository rebuilds the tree so anyone can check the work.
A grade is not a safety guarantee, not a statement about anyone’s intent or competence, and not a prediction that a plugin is good or useful. It goes stale — a plugin whose source floats can change the day after it is scored.
The five dimensions
| Dimension | What it looks at |
|---|---|
| Execution surface | Hooks and what they run, and when. MCP server launch commands. Anything that executes on load or use. Events that fire without an explicit user action are weighted above ones that don’t. |
| Instruction surface | Skill, command, and agent prose: hidden or zero-width text, embedded shell, instructions to withhold information from the user, fetch-and-execute lines. |
| Source integrity | Is the source pinned to a commit, or does it float? Is code fetched at runtime, and if so is that fetch pinned or integrity-checked? |
| Provenance | Account and repository age, contributors, release cadence, license. A modifier only — see the caps. |
| Hygiene | Does the documentation match what ships? Context cost. Files present but never referenced. |
How the letter is decided
Not by adding up points. A weighted average lets good provenance launder a bad hook, so grades are set by caps on the worst thing found:
- A high-confidence critical finding — fetch-and-execute in a hook, credential read plus network egress, unpinned remote code in an MCP launch — caps at F.
- Any high finding caps at C.
- A floating source reference caps at B, because the code can change after you install it.
- Provenance can lift a grade within the band left by the caps, never past one.
One consequence worth stating plainly: the floating-source cap fires for essentially any plugin that ships inside its own marketplace repository, because the manifest format has no pinning field for that layout. B is a structural ceiling for a large part of the ecosystem, not an accusation.
Verification
Findings are produced with LLM assistance and human review. Every finding is then given to a second, independent reviewer whose only instruction is to knock it down: confirm the snippet exists as quoted, confirm the code path is reachable, and check whether the “risk” is really just the plugin doing what it openly says it does. Findings that fail that pass are dropped. In the first audit this moved every disputed grade up, never down.
A reviewer that crashes is not a reviewer that disagreed. Synthesis refuses to run while any finding lacks a verdict, and no grade may improve without citing the specific text that refuted the finding. That rule exists because the pipeline once did the opposite, which is written up in the first audit.
How plugins are chosen
Directory plugins are selected by install count and by category coverage. Long-tail samples are drawn reproducibly: GitHub code search for .claude-plugin/marketplace.json, unique repositories from the first result pages, sorted by repository name, and the entries at fixed percentiles taken. The query, date, counts, and resulting picks are published with each audit so the draw can be rerun.
What this method cannot see
- Code fetched at runtime. When a plugin runs
npx,uvx, or an install script at launch, the fetched payload is outside the reviewed tree. The grade assesses the fetch mechanism, not what comes back. - Remote MCP servers.Their behavior lives on someone else’s infrastructure and can change at any time.
- Anything only visible at runtime. Nothing is executed. Network behavior, telemetry payloads, and permission prompts are inferred from source.
- The future. Floating sources mean findings are true of a snapshot. They may be stale in either direction by the time you read them.
- Absence. Verification removes false positives; it does nothing about false negatives. A short findings list is not proof that there is nothing to find.
Token estimates
Context costs are estimates, not tokenizer-exact, and are calibrated against claude plugin details, which reports the real figures for a plugin you have installed. Treat totals as within roughly twenty percent. Always-on cost (the index entry carried in every session) and on-invoke cost (the body, loaded when a skill fires) are reported separately, because summing them overstates what a session actually costs by an order of magnitude.
Disputes and corrections
Every project named in an audit is contacted before publication and offered a reply.
- A factual error — a wrong line, a misquoted snippet, a misread code path — is corrected the same day, with a dated entry on the corrections page.
- A disputed interpretationgets the maintainer’s reply published verbatim next to the finding. The grade does not move without new evidence, and the argument does not continue in public threads.
- A finding shown to be wrong is retracted. The retraction stays visible and any grade change is noted.
Anything reported as an unfixed weakness in someone’s project is disclosed privately first, with at least fourteen days before the details are published.