On Wed, Oct 14, 2020 at 6:18 AM Dik Takken <dik.tak...@gmail.com> wrote: > > However, then we should also make sure that the example > code actually works, and on which PHP versions. As soon as examples can > be run right from the documentation pages, the examples will be run far > more frequently than they are now, simply because it is so much easier > to do. Broken examples will cause more disappointing experiences. >
Agreed 100%. I do think it should be reasonable to write a one-time doc scanner to look for <programlisting role="php" /> and attempt to run those in a container. It's not going to be a guarantee that the code does what we *expect*, but we can probably turn it into a report of "things that definitely don't parse" + "things that raise E_ERROR" + "things that warn/notice, should be looked at" + "things that we hope probably work maybe" and from there just have a prominent "report a bug about this example" button which prefills a lot of the bugs.php.net form data and/or maybe even avoid duplication by attaching "me too" to reports that have already been filed for a given code snippet hash. Bear in mind translations sometimes have unique bugs (variable names and example content get translated too), so scanning the other languages would be a good followup task. TL;DR - Yes, this is work to do up front, but it's pretty clear that we can turn this from a massive pile of wtf into a tractable problem. > I don't know if example code is currently tested automatically. If not, > adding a sandbox for running them may also offer an opportunity for > automatic testing. > It's automatically run in the brain of whoever commits the example to the docs repository. :( Usually this doesn't lead to any *majorly* broken code, but a typo here or there is sadly not uncommon. Once in a while we get a report via bugs.php.net and it's a quick fix, but bugs exist. > - A general sandboxed php implementation hosted > by the owners of php.net (requires that it be secured > and may lead to additional hosting costs), > https://3v4l.org/ (not open source https://3v4l.org/about - > but similar to what I was thinking of) > The simple implementation here is a "try this code" button that pops 3v4l.org open in a new tab with the code prefilled and running. That means a conversation with the owner of 3v4l.org obviously, but potentially doable, and probably not hard. > or https://psalm.dev/ (open source) are projects in that area > (Matthew Brown is one of the authors of Psalm) > I don't like the idea of executing that on www.php.net for a few reasons, but someone else mentioned the possibility of donated cpu time from somewhere that's worth a conversation. > - A WebAssembly solution, e.g. https://phan.github.io/demo/ > (forked from https://oraoto.github.io/pib/) (I'm one of the maintainers of Phan) > Honestly, I like this solution best. There are drawbacks, and we'd probably need to suppress it in certain reference chapters (e.g. mysql, since there's no DB to talk to), but damn if it don't feel skippy trying it out just now. (On my admittedly well connected, beefy dev workstation) -Sara
