Hi All,
First, I want to apologize to everyone using the FreeDOS Archive on GitLab (
https://gitlab.com/FreeDOS <https://gitlab.com/FreeDOS> ) that may have
received a bunch of messages regarding failed and successful CI/CD pipelines
over the last few days. This was the result of some work I was doing, your
permission level for those projects and of course your notification settings.
A few days back, there was a small issue with one of the projects. Whenever
that project was updated and a new release was created, the developer wanted to
have a FreeDOS compatible and directly installable package. Unfortunately,
there was some problems with doing that.
When GitLab creates a release, it takes everything in the project and zips it
up. That part of the process is fine. Projects there are kept in a fully ready
to “Zip and Ship” directory structure with all necessary files included. But,
it would also include the project informational files stored in the root
directory which would collide with other projects. But the major problem was,
when performing this function, it would put everything in the zip archive under
a “(project)-(comit-hash)” directory. Both problems making the zip archive
completely incompatible with all package managers under FreeDOS.
But, I knew this should be a solvable problem. I have seen many projects on
GitLab/GitHub that included additional files with their releases. Often, those
files are releases for other platforms, binaries only and other such related
items. So, I started digging into the CI/CD (continuous integration and
deployment) support for GitLab.
I’ve never used the CI/CD tools for Git before. Ugh, you think it would be
simple. But, no. However, after spending a few days reading numerous pages of
inaccurate, incomplete and obsolete information, I got a prototype CI/CD
pipeline up and running.
The CI/CD pipeline is not perfect. But, it does some cool stuff. There are a
couple things left to do on it. But, I have not figured those out yet.
Possibly, there may be someone on the mailing list who is an expert with Git
CI/CD runners and pipelines who can advise or help with the remaining stuff.
At present, the CI/CD pipeline does a bunch of stuff whenever a commit is
pushed or merged into the main/master branch. This is not an all inclusive
list. Just some of the major features already present.
Validates directory structure and standard package format.
Validates included required LSM metadata and prevent extra package LSM files.
Validates files do not exist in places that will cause installation collisions.
Checks that sources are present.
Prevents erroneous files in system directory locations like HELP and NLS.
Determines if a new release is warranted by checking the version in the
projects LSM file.
When a new release is desired, the CI/CD will also do these.
Adjust timestamps to the correct date/time (assuming commit tools, like
fdvcs.sh, are used to keep the timestamp database up to date).
If a LSM-Support field is not present in the LSM file, it will scan the project
and determine if LFN files are present.
Depending on the location of those files (excluding SOURCES), LFN support will
be determined as not needed, optional or required.
No LFNs - not needed
LFNs found in standard system directories (like BIN, HELP, etc). - required
LFNs found in other program directories - optional
If the LFN scan was positive, an LFN-Support field will be added to the
released package LSM metadata. The created field will either be “required” or
“compressed”
If an LFN-Support field is “compressed”, it will compress LFNs under the
directories into one or more LFNFILES.ZIP archives. To prevent this, a project
can just preset the LFN-Support field in the LSM file as “required”.
It will pre-compress the project SOURCES to save a few bytes, increase instal
speed (when sources are installed) and for a couple other reasons discussed on
this list earlier. (this optional and can be disabled).
Update the LSM field “Modified-date” to “${today}.0”
Add the LSM field “Packager” with the pipeline script and it’s version.
Add the LSM field “Git-Site” with a URL to the project on GitLab.
Add the LSM field “Git-Commit” with the short hash that triggered the creation
of the release.
Possibly add the LSM field “LFN-Support” when not present and the LFN scan
finds them.
Possibly add the LSM field “Timestamps” when the timestamp update detects that
the database is has not up-to-date and one or more files were not stamped.
Compress the whole thing up into a FreeDOS package manager compatible zip
archive.
Once that is all done successfully, the CI/CD pipeline finishes up the process
by:
Generating a new repository tag based on the LSM version.
Creates a Release for that Tag.
Includes a link in the Release to download the FreeDOS package.
In addition to all of that, the pipeline supports the ability to compile the
project. This was something I added and was testing yesterday with a known easy
to compile project (V8Power Tools). It only requires NASM and UPX to compile
and already included both DOS and Linux cross-platform compiler scripts. It's
latest release includes a package compile by the CI/CD. (
https://gitlab.com/FreeDOS/util/v8power/-/releases
<https://gitlab.com/FreeDOS/util/v8power/-/releases> )
In theory, with the support to compile a project in the pipeline, if each
project included a script to build from source that took care of all the
requirements to build a project, the entire OS could be built from scratch. It
would also make it much easier for anyone to compile a project from source.
Maybe someday.
So, the CI/CD pipeline does a lot. However, there are a couple things that
could be done and some others I have not figured out yet.
First, you may have received a bunch of vague messages like “Successful or
Failed Pipeline” “ or “created using release-cli” from GitLab yesterday. Sorry
about that. That should settle down now. As for the messages themselves, it is
possible to add some additional information. But, I’m not sure what extra
information would be useful there that can be added. We will have to see.
As for what there is left to do that I have not yet figured out, it really just
boils down to a couple of things.
Ensure the FreeDOS package has a permanent link. At present, as far as I can
tell, the download link will remain good at least until another release is
created. At which point, the previous package may get deleted and the link
become invalid. I think this would just require turning the build artifact into
a release asset and using the appropriate link.
Clicking the link to download the FreeDOS package sends you to a different
page. On which, you can then download it. It would be better to just have the
link download the package. This may solve itself when the artifact is turned
into an asset with a permanent link.
When the script determines a release is not required by checking the package
LSM metadata, have the pipeline stop without trying to generate a release
anyway and causing a “Failed Pipeline” message. The script portion sees this is
the case. But, the YML seems to either ignore that result or not run at all.
So, IDK. Possibly, always do both when it is triggered by the master/main
branch. Then only always just perform validation on other branches (like
unstable, devel, etc).
For now, I’ve only added the pipeline to a couple projects.
Going forward, I will add it to existing projects when they are updated. This
will prevent being bombarded by hundreds of GitLab messages from adding the
pipeline to everything now. Then getting even more messages if/when anytime I
update or improve the current CI/CD iteration.
It is a work in progress.
:-)
Jerome_______________________________________________
Freedos-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/freedos-devel
