I apologize if there was any tone to my post, the only place I intended there to be any was when I pointed out the --no-check-certificates flag. Please understand Perl 6 is the second or third language I have found that does not secure its ecosystem. The number of non-language projects which suffer from this problem is in the double digits.
Despite the fact I know what Perl 6 is and have found its mailing list, I am not a smart man. I have merely watched myself break a great number of things and watched other people fix them. On Thu, Jul 27, 2017 at 3:51 AM, Patrick R. Michaud <pmich...@pobox.com> wrote: > The details of how to use the star GitHub repository are in > tools/star/release_guide.pod . > > You're entirely welcome to create a bundling that has a better build system > than what Rakudo Star uses -- indeed, Rakudo Star has always been intended to > be just one of many possible bundlings of Rakudo and useful modules. Even if > you just create a repo to demonstrate a better way of organizing modules, > that'd be helpful. > > The earliest versions of the Rakudo Star build system started out by trying > to use Git submodules to manage packages, but it quickly proved to be > unwieldy and almost impossible to understand and maintain. Perhaps the > submodule ecosystem has changed since then, though. Can you give an example of how submodules were insufficient? I will take whatever you have to add at face value, I'm just interested in some concrete example because I have seen plenty of things done for very diffuse and nebulous reasons. Most issues I have seen that arise with submodules come from people trying to treat the submodule directory in a way that is different than other objects tracked by Git. If you treat it like a source file you're tracking most problems should disappear, at least in theory. There's still some unfortunate submodule command names. On Thu, Jul 27, 2017 at 4:22 AM, Steve Mynott <steve.myn...@gmail.com> wrote: > On 27 July 2017 at 09:13, R0b0t1 <r03...@gmail.com> wrote: >> Are there any releases signed by the developers? The official releases >> located at https://rakudo.perl6.org/downloads/star/ do not seem to >> have signatures available. > > Can you be more specific about the technologies you suggest and the > exact problems you are trying to solve? > > Maybe you mean something like tarballs signed with GPG? Or SHA256 > sums? In both cases there is still an issue with distributing > signatures and the checksum files. > > I could upload a checksum file but since this would be in the same > directory as the tarball and both could be manipulated by a potential > attacker there seems little point. > > Ultimately you are able to (mostly) read the files in the source > tarballs and audit for backdoors yourself (if this is your threat > model). > The solution is to either give a GPG signature of the release tarball, or, because signing and verifying the signatures of large files can take a long time, summing the file in multiple ways and then signing the list of digests. Example: http://distfiles.gentoo.org/releases/amd64/autobuilds/current-stage3-amd64/, see release.DIGESTS.asc. Of course there is still the problem of communicating the release keys to someone in the first place, but if the release key is on a public keyserver and its ID is referenced on the project site somewhere that typically works well enough. >> As a stopgap measure I attempted to clone >> https://github.com/rakudo/star and run Configure.pl. I felt it strange >> that it warned me against using the code in the repository as cloned >> from the repository (why does the repository exist?) but continued > > The repository exists to create the distributed source tarball itself. > You need to read the documentation starting with release-guide.pod and > understand it. > > This is really a prerequisite for any useful criticism of the build process. > > Yes it does look strange and needs improvement but there are reasons > for it and most of it is written in easily understandable perl 5. > >> with --force, only to find out that some of the folders in the >> directory structure need to be initialized. A mailing list posting was >> found detailing how to continue >> (http://www.nntp.perl.org/group/perl.perl6.compiler/2013/05/msg8915.html) >> and the steps were followed. Unfortunately, the nqp package download >> failed with error 404, which made me see the following command was >> part of the preparation script: >> >>>wget --no-check-certificate >>>https://rakudo.perl6.org/downloads/nqp/nqp-2017.07-9-gc0abee7.tar.gz -O >>>src/nqp-2017.07-9-gc0abee7.tar.gz >> >> I have no words. Can anyone justify this choice? > > Odd since there seemed to be enough words in your email! Really I > don't have to justify anything to you but spreading knowledge of how > this works is generally useful. > > The reason the released NQP package wasn't shipped was because there > was a bug in the released one. This is documented in > "lastmin-fixes.txt". This process isn't ideal as this file admits. > This is only the second time the release tarballs weren't used. > > The NQP tarball you want is in the actual Rakudo Star tarball not the > git repo for the build process. In hindsight I probably could have > pointed it straight at a github versioned zip (although this would > have added unzip as a dependency!) or just not bothered and shipped > with a broken -ll-exception. > What I want is to verify the code I run before I run it. From my position the easiest way to do this was to try to grab code from Git. The star repository was the only one that looked like it had everything in one place and a way to use those things. >> Are there any signed releases, or do I have to do the equivalent of >> curl|sudo? > > Extracting a tarball and manually running scripts isn't the same as > running curl|sudo since you have the chance to read what the scripts > do before running them. > > There is one subsystem where this isn't the case and its left as an > exercise for the reader :-) > NQP? I was told that has bytecode in it. If possible I would request that this is changed in the future. >> What is being done should actually be done with Git submodules, or the >> more recent feature, subtrees. There is no reason for the releases to >> be prepared in this way. What has been created is a very inflexible >> package management system. If what you want is a package management >> system, then I would suggest adapting Gentoo's portage to the >> project's needs, as KDE seems to have done with their Craft build >> system. However what the project needs is version control that will >> handle dependencies. > > The Rakudo Star system does use git submodules for the module dependencies. > > Maybe submodules in R* should be used for the Rakudo, NQP and MoarVM > dependencies -- normally based on a tagged release but allowing a > version a few commits later for bug fixes (which also should be tagged > as .1). This isn't a bad idea. > > The problem with this is that the monthly Rakudo (not Star) tarballs > still wouldn't get these fixes (and it's quite a lot of work for > Rakudo upstream to produce potentially three extra tarballs for each > patch) so most of the third party packaging (which doesn't tend to use > Star) wouldn't get the fixes anyway. Still any extra tagging of bug > fixes as .1 would help. I still think we need a -STABLE type branch as > well. > > The whole rakudo (and star) build process doesn't fit in well with any > third party systems (gentoo, cmake whatever). Having Perl 5 as a > dependency for Perl 6 seems to be more reasonable than using anything > else. > Why are those things unsuitable? I have used them enough to know that they can not solve every problem, but it has eventually become clear to me that it is almost always easiest to try to add whatever functionality is necessary to an already existing build system than trying to manage it myself. I have seen a great number of edge cases, but even then it may be best to push a solution to, say, CMake, than maintaining a less comprehensive build system specifically for a project. I understand Configure.pl and friends are already written but if it comes up in the future it is something to consider. > It's open source and if you feel strongly enough about this to > actually do the work you can fork the repo and make changes in your > own branch. We would actively encourage contributions and it's easier > to evaluate actual code changes than emailed suggestions. > > But be aware talk is cheap and code costs time. For the most part my issue is the lack of signed releases. What I would like to do is start experimenting with Perl 6, but it looks like I will need to create a virtual machine to do this. Not just for security reasons, but because I am having a lot of problems using even the release tarball to build R*. I will do some more research and probably won't end up bothering the list with the issues. Since the issue is signed releases, I can't fix that my self. In the interest of making release signing easier I proposed some changes to how the release is generated because I was confused with what was happening. If those changes aren't necessary then that is fine, however I am worried that releases are obtained by downloading something else that is not itself verified to generate the release. Please understand that even should I have the time to contribute, I still have to read and understand the project. My message is more a question of why the build system is the way it is. On Thu, Jul 27, 2017 at 12:48 PM, <zof...@zoffix.com> wrote: > > Quoting R0b0t1 <r03...@gmail.com>: > >> Are there any releases signed by the developers? The official releases >> located at https://rakudo.perl6.org/downloads/star/ do not seem to >> have signatures available. > > > There's Rakudo the compiler and Rakudo Star distribution > which is the compiler + docs + some modules. > > Yes, the compiler releases *are* signed. You can get the signatures off > GitHub: https://github.com/rakudo/rakudo/tags or the compiler directory > on the downloads page: https://rakudo.perl6.org/downloads/rakudo/ > > Rakudo Star release guide does advise to sign and it's unfortunate it's not > being > done with the explanation of it being too much effort. It's just two > commands to run, > which can be automated, like it was done for the compiler releases. > >>> wget --no-check-certificate >> >> >> I have no words. Can anyone justify this choice? > > > No, that's awful and should be removed. > >> […] should actually be done […] I would suggest […] the project needs >> […] There are various articles […] they are slightly more complicated to >> use. > > > Keep in mind a large portion of this project is maintained by volunteers, > using their > spare time and whatever knowledge they possess. Jumping in with a mountain > of criticisms > to tear our volunteers a new asshole isn't very helpful, I think, as the > replies sent > to you so far indicate. > If everything was going well would I have much reason to comment? Most of my reply hopefully indicated confusion. As far as I know there is a much simpler way to accomplish what is currently being done, and it will be easier to maintain. The fix would be the removal of a bunch of code and using some new Git commands (which admittedly might end up being added to a script somewhere). > There was talk of making a leaner distribution: Rakudo compiler + zef module > installer. > Perhaps, you could start on that effort and do it the Right Way and once > that's working well, > Rakudo Star can be similarly improved. > I think this is a very worthy endeavour but I will likely not have time to contribute in the immediate future. Regrettably I have not been able to turn my experience with open source software into money and as such my time is very limited. I would hope the module installer verifies modules, as this is another common problem with languages and their distributions. The initial download may be verifiable but then (depending on how the installation is set up) the first update of the package manager associated with the language will download unverified code over HTTP and then run it as root. My apologies if I didn't address part of someone's message. This email is already very long and hopefully most of it is irrelevant if I can get Perl 6 working in a VM. R0b0t1.
