LucaCappelletti94 commented on PR #2527: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2527#issuecomment-5765048520
> > > I think the gate should also be about false positives (not just true negatives) > > > > > > What do you mean by false positives? > > I mean the chance of the fuzz test failing on a PR that did not cause whatever problem the fuzzer found. > > I think if we run non deterministic tests on PRs we are going to inject some noise into the process -- I would recommend against it if at all possible (at least until we have some confidence that the fuzz test is mostly successful -- e.g. run it for a week or more every day and not find anything new) I am not sure how that would occur, suppose the PR compiles to `f` and the main is compiled to `g`. For a given fuzzer input `x`, if `f(s)` crashes and `g(x)` does not, there is no non-determinism. The only non-deterministic aspect is that a failing input `x` might not be found in the limited 10 minutes window. The scenario you describe where `f(x)` crashes and `g(x)` does not but it should can only happen if I configured ClusterFuzzLite poorly, if I understood it correctly. To be clear, I **perfectly agree** that if the CI starts to randomly give out bugs to PRs that have nothing to do with that bug it would be something extremely problematic and to be absolutely avoided. If it happens to be the case, I would suggest to only keep the periodic fuzz job. That being said, again if I understood correctly its documentation, that should not be the case as it would make ClusterFuzzLite principal use case be nigh useless. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
