Talk to anyone maintaining a widely used open source project about security reports and you will hear the same thing. The reports have got worse, and there are more of them.
Some of that is volume. A popular library attracts attention, and attention includes people running automated tools against it and forwarding whatever comes out. Some of it is incentive: a CVE is a line on a CV, and a public disclosure is a portfolio piece whether or not anything was actually exploitable.
And some of it is new. Language models made it trivial to produce a report that reads like security research. Confident structure, plausible technical vocabulary, a severity rating, a proof of concept that does not work when you run it. Maintainers of several high profile projects have been vocal about the volume of this they now receive, and the time it costs them to establish that a well written report describes nothing at all.
I work on an AI security tool. I think this is worth saying plainly: my part of the industry has made a maintainer's inbox meaningfully worse, and any tool that wants to be useful here has to start by not adding to that.
Three ways a report wastes a maintainer's time
It reports a possibility. The report says an endpoint appears to lack an authorisation check, or a function may be vulnerable to injection under certain conditions. Appears and may are doing all the work. Somebody now has to read the code, work out whether the theoretical path is reachable, and establish whether the control two layers up already handles it. That investigation is the actual cost, and it happens whether or not the finding turns out to be real.
It arrives with a clock attached. Coordinated disclosure exists for good reasons and I am not arguing against it. But a ninety day window applied uniformly assumes a maintainer with paid time to spend on it. Applied to someone maintaining a library in the evenings, a disclosure deadline is not a coordination mechanism. It is a threat with a date on it.
It transfers the work. A report that says here is a problem, good luck, has moved a task from the person who found it to the person who did not. On a commercial engagement that is the deal, because somebody paid for the finding. On an unpaid project it is somebody else's homework, handed over with a severity rating to make it urgent.
None of these require bad intent. Most of the people sending these reports think they are helping. The structure produces the outcome regardless.
The economics are the wrong way round
Enterprise software gets a scheduled penetration test, a security budget, and a team whose job includes reading the report.
The library three levels deep in that enterprise's dependency tree, without which their product does not build, gets whatever its maintainer can manage after work. It carries comparable risk to the systems that depend on it and a fraction of the resources.
That asymmetry is well documented and I have nothing new to add to the diagnosis. What is worth noting is the second order effect: because the resources are not there, the marginal cost of a low quality security report is much higher for a maintainer than for an enterprise security team. The enterprise absorbs a false positive as an hour of somebody's paid Tuesday. The maintainer absorbs it as an evening they were going to spend on the feature they actually wanted to build.
Which means the same report, sent to two recipients, does proportionally more damage to the one with less capacity to handle it.
What a good report looks like
I would rather propose a standard than complain about the absence of one. Four properties, and I think all four are necessary.
Confirmed, not suspected. The report should describe something that was done, not something that might be possible. Here is the request we sent, here is the response, here is the data we should not have been able to reach. If it could not be demonstrated, it is a code review comment and should be labelled as one.
Reproducible without trust. The maintainer should be able to verify the finding independently. That means the actual requests, in full, in a form that can be replayed. A finding you cannot reproduce is a finding you have to take on faith, and nobody should be asked to take security claims on faith.
Private until the maintainer decides otherwise. No clock, no advisory drafted in advance, no disclosure date set by the reporter. The person who has to fix it decides when it becomes public. If that takes six months because they have a job and a family, it takes six months.
Accompanied by the fix. Not a suggestion of what to do. The actual change, as a pull request, scoped to the issue, ready to review and merge or close. This is the property most often missing and the one that shifts a report from a cost to a contribution.
That fourth point is the one I would defend hardest. The asymmetry in open source security is not primarily an asymmetry of information. Maintainers usually understand their code better than the person reporting on it. It is an asymmetry of time. A report adds to the time deficit. A patch reduces it.
What I am not claiming
Automated testing does not replace an experienced researcher looking hard at a specific target. There is a category of finding, particularly anything involving subtle protocol behaviour or a chain of individually harmless issues, where a person spending a week beats any current tool. I expect that to remain true for a while.
I am also not claiming that everything an automated system produces meets the standard I just described. Confirming exploitation reduces false positives substantially. It does not eliminate them, and a maintainer reviewing output from any tool, including ours, should treat the first few findings with scepticism until the tool has earned otherwise.
And I am aware of the obvious objection to a company publishing standards that its own product happens to meet. Fair. The standard is not novel. Every good researcher I have worked with already operates this way, and the reason it needs stating is that automation made it cheap to operate the other way.
Where Deus Rex sits
We are running a private beta that gives open source projects a full autonomous penetration test at no cost. Web, iOS and Android. Findings confirmed by exploitation, delivered with the requests so you can reproduce them yourself, private until you decide otherwise, with fixes raised as pull requests you review and merge or close on your own terms.
There is no card, no seat count, and no upsell waiting at the end. What we ask in return is feedback, because a tool that has only been pointed at commercial targets has a narrow view of the world, and open source projects are structured differently from the enterprise applications we have spent twelve years testing. If the tool earns it, a few honest words we can quote. If it does not, we would rather hear that.
Spots are limited because every submission is reviewed by hand rather than queued, and both platforms of mobile support mean the assessments take real time to run.
If you maintain something people depend on and you have ever thought somebody should look at this properly, that is the offer.
deusrex.ai/open-source
Written by
Tim De Wachter
CTO Deus Rex
