Hi Nicolas,
before I start arguing, I appreciate all your hard work on improving the
Go guidelines, spending so much time explaining reasoning behind your
decision. Also appreciating the effort on making the life easier for
packagers, not just in the Go land, but throughout the distribution as
well. There is a lot of great ideas in your Go proposal I would like to
see implemented and widely used. The more transparent and easy-to-use
spec files, the better for all maintainers. Let's keep moving forward,
improving the infrastructure for other folks and pushing new ideas and
improvements into practical solutions.
On 12/17/2017 08:11 AM, nicolas.mail...@laposte.net wrote:
Hi,
I am proposing for inclusion a set of rpm technical files aimed at automating
the packaging of forge-hosted projects.
- Packaging draft:https://fedoraproject.org/wiki/More_Go_packaging
-https://pagure.io/packaging-committee/issue/734
- go-srpm-macros RFE with the technical
files:https://bugzilla.redhat.com/show_bug.cgi?id=1526721
This proposal is integrated with and depends on
thehttps://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation
draft
It builds on the hard work of the Go SIG and reuses the rpm automation
ofhttps://fedoraproject.org/wiki/PackagingDrafts/Go when it exists, and
produces compatible packages.
Can you describe what you mean by "compatible packages"? I mentioned a
list of things that you did not answer fully. Most important to me:
- your macros do not generate build-time dependencies, which I see as
one of the drawbacks. Do you plan to ignore them?
- support for more subpackages. You describe how to specify which files
go to which packages
(https://fedoraproject.org/wiki/More_Go_packaging#Separate_code_packages)
but you don't say how list of provided packages and a list of
dependencies are generated for the subpackages.
- reproducible evaluation of macros. Either for running automatic
analysis over spec files or for debugging purposes. I would like the
macros to be built in top of binaries I can run separately. With Jakub
(jca...@redhat.com) we were discussing many times we could provide a
minimal rpm built on top of gofedlib that would extract all the
necessary pieces, including the naming convention so everyone can import
provided library guided by the same rules as the guidelines enforce. I
would like to see us synchronizing on this topic at some point.
What it does:
- drastically shorter spec files, up to 90% in some cases, often removing
hundreds of lines per spec.
+1 here, the shorter the spec file the better as long as it is still
clear and transparent
By better I mean:
- easier to read, easier to understand, easier to adopt
- less places to change and look if a spec file change is needed
By clear I mean:
- it is obvious what the spec file is actually declaring to do
- it is easy to customize various parts (e.g. list of tests, list of
dependencies)
- simple, packager-friendly spec syntax
+1 as long as each macro provides a single functionality. E.g.
- generate a list of provided packages
- generate a list of tests
- generate a list of dependencies
+1 to the %goname, %gourl, %gosource, %goinstall, %__goprovides,
%_gorequires and other usefull macros as long as they are simple and
transparent
As long as the macros are optional and user can use any of them if it is
suitable (suitable meant per a packager judgment).
- automated package naming derived from the native identifier (import path). No
more packages names without any relation with current upstream naming.
Can you provide a list of package names that diverge from this convention?
For historical reasons there are some packages that does not fall into
the naming schema. Either because a go project got migrated to a
different repo or because it make/made sense to combine some projects
together and ship them in a single rpm. I don't mention Kubernetes,
Docker and other popular projects as you already mention exception for them.
- working Go autoprovides. No forgotten provides anymore.
Sometimes it make sense to skip some provided package. E.g. some
packages provide Windows specific code
which has no use on Linux. So the claim is not completely valid.
- working Go autorequires. No forgotten requires anymore.
Depending on how you generate the list you can non-intentionally require
more than is needed.
Which causes installation of unneeded trees of rpms.
E.g. for building purposes there is no need to install dependencies of tests
So the claim is not completely valid.
(Valid for both requires/provides): Sometimes you only need to
install/provide a subset of a Go project.
There are not many projects that imports every package another project
provides.
So to save time packaging dependencies of unimported packages (and their
following maintenance),
it is beneficial to generate only partial list of required/provided
packages.
Plus, in case CGO, there is no 1:1 mapping of C libraries to their
corresponding rpms.
- strict automated directory ownership (used by autorequires and autoprovides).
- centralized computation of source URLs (via Forge-hosted projects packaging
automation). No more packages lacking guidelines. No more broken guidelines no
one notices.
In other words to use %gourl macro?
- easy switch between commits, tags and releases (via Forge-hosted projects
packaging automation). No more packages stuck on commits when upstream starts
releasing.
The issue is not about upstream releasing, it's about projects that
actually import snapshots of other projects instead of particular
releases. Given by the nature of the Go land.
- guidelines-compliant automated snapshot naming, including snapshot timestamps
(via Forge-hosted projects packaging automation). No more packages stuck in
2014.
The issue is not that a package is stuck and not updated for more than 6
months. The issue is API backwards incompatibility that does not allow
to easily update to a newer version. And a lack of man power to perform
an update.
- guidelines-compliant bootstraping.
- systematic use of the Go switches defined by the Go maintainer. Easy to do
changes followed by a mass rebuild.
- flexibility, do the right thing transparently by default, leave room for
special cases and overrides.
- no bundling (a.k.a. vendoring) due to the pain of packaging one more Go
dependency.
Can you more elaborate on that? Cause this is unavoidable no matter how
great any Go guidelines get.
Choice of bundeling is not about how many more Go dependencies you need
to package.
It's due to API backwards incompatibility reasons.
- centralized Go macros that can be audited and enhanced over time.
+1 as long as they are kept on a reasonable granularity level
- aggressive leverage of upstream unit tests to detect quickly broken code.
Will not work in general due to API backwards incompatibility issues.
We need to move the tests out of the spec files if we want to seriously
test Go projects.
Saying that, we can run subset of tests to at least have some kind of
smoke tests.
no reliance on external utilities to compute code requirements. No
more dependencies that do not match the shipped Go code.
Please, rephrase or remove the point as for the CGO it is useful to have
external utility that can list all imported C header files and try to
find a set of packages that provide the header files (and corresponding
libraries).
Please consult packaging draft for full information.
The proposal has been tested in Rawhide and EL7 over a set of ~ 140 Go
packages. This set is a mix of current Fedora packages, bumped to a more recent
version, rewrites of Fedora packages, and completely new packages.
I hope posting the second part of the automation will answer some questions
people had on
thehttps://fedoraproject.org/wiki/Forge-hosted_projects_packaging_automation
draft
Regards,
I am surprised you are able to do that without any API issues. Have you
tried to de-bundle and build Kubernetes, Docker, Etcd, Prometheus and
other big projects from the same set of Go packages? At some point you
have/had to switch to compat packages or bundling.
=========================================
From the guidelines:
https://fedoraproject.org/wiki/More_Go_packaging#Limitations
Totally agree with the release discipline in many Go projects.
gives up on many conventions of current Fedora Go packaging, as they
were an obstacle to the target packaging efficiency.
(Still part of the Limitations): Can you make a list of conventions you
consider as obstacles? I would like to comment on each of them.
https://fedoraproject.org/wiki/More_Go_packaging#Excluding_testing_in_specific_directories
%gochecks . transport/http transport/grpc transport option internal
integration-tests/storage examples
The excluding mechanism is not intuitive, my first understand would be
to test all the listed directories. Maybe use %gocheck -v the same way
is grep does?
https://fedoraproject.org/wiki/More_Go_packaging#Handling_API_changes
API changes may require the creation of Go code compatibility packages
I would not recommended doing that unless you have an army of packagers.
It not just about creating one compat package. It is not surprising API
incompatible change goes hand to hand with updating of dependencies. So
instead of creating one compat package you will end up creating a tree
of compat packages. I have already mentioned that at
https://pagure.io/packaging-committee/issue/382#comment-147649. Plus,
multiply the number of compat packages to take care with the active
Fedora branches + epel7. On paper it looks like a very good solution but
in reality it is not. As long as we don't know how much the Go projects
suffer the API incompatibilities, I don't want to spawn thousand compat
rpms and maintain them.
a packager, that identifies an API change in his Go package, MUST
notify gol...@fedoraproject.org at least a week before pushing his
changes to any release (and those releases SHOULD include
fedora-devel). This grace period can be waived by FPC for security
reasons.
From the point of view of updating popular projects like Etcd,
Kubernetes, Prometheus, etc. this is highly impractical. Usually, there
are more dependencies to update. If I will have to wait one week to
update etcd, than another week to push its update to stable branch, then
I will not really be effective in updating.
a packager, that identifies an unannounced API change in another Go
package, MUST notify gol...@fedoraproject.org at once.
those notices SHOULD be copied to the maintainers of packages affected
by the change, when they are known.
This is going to be a regular thing unless a tooling detecting this
changes is available.
a packager, that identifies that the code he packages, is broken by
API changes in another project, SHOULD notify politely the upstream of
the broken project of the API change, if he can not ascertain it is
already aware of the change.
Some upstream Go projects are responsive, some are not. A lot of Go
projects are forked, abandoned or inactive. Or the upstream is a single
person that does not have time to deal with distribution specific
guidelines. But I appreciate the effort to spread the good practices.
However, sometimes the API change is intentional cause there is no other
way.
=========================================
Usual workflow of updating Go packages:
1. choose a commit to update to
2. update a list of provided packages
3. update a list of build/run-time dependencies (if needed create spec
files for new Go projects and open review requests)
4. update a list of tests (skip some tests that fail on specific
architectures + open upstream issues, resp. PR with a fix)
5. optionally, update %install/%build section
6. perform scratch-build (may require to override some rpms)
7. perform push and raw build
8. create package update in Bodhi
9. push updates into stable if there is not enough karmas
## Some issues one can encounter with when performing a Go package update
- update is backward incompatible (it may remove ability of some package
Go projects to get built)
- 5 (arbitrary number here) or more new Go projects need to be packaged
into distribution
## What we need
- automatic updates of Go packages (each time there is a new release or
on demand to a specific commit)
- spread maintenance among a group of packagers
- run unit-tests and integration tests (if available) in CI/CD (e.g.
after each update, on periodic basis)
- automatic every routine job we do so we can concentrate on "real" and
"interesting" problems in the Go land
The updated (and fairly improved/extended) guidelines cover points 2),
3), 4), 5) to some extend.
Comparing the new way with the current way (listing only some +/- that
popped in my head during interpretation):
+ provided packages, runtime dependencies are automatically generated
+ tests are automatically generated
+ spec files are much smaller (more transparent, a lot of complexity
hidden in macros)
+ guidelines looks awesome, more descriptive in what needs to be done
(up to some parts mentioning many times the current spec files sucks a lot)
- no build-time dependencies (we can not discover missing dependencies
before installation)
- always generating all provided packages that are available (I don't
want to provide all packages always, e.g. need just a subset to build my
project, the remaining provided packages need new Go projects which adds
unnecessary complexity)
- as jcajka pointed out, it is not easy to debug the lua based scripts
- combination of bundled and un-bundled dependencies (sometimes I just
need to use one or two bundled dependencies because without generating a
tune of compat packages)
In all cases, the new approach is strongly focused on specfile
simplification.
The API backwards incompatibility problems are conveniently hidden in
the creation of Go code compatibility packages.
I have already mentions this approach in the
https://pagure.io/packaging-committee/issue/382 and the reasons why it
is not a way to go.
Even if we create a lot of compat packages we still need a mechanism
that can tell us when a given compat package is no longer needed.
I am working on a tooling (in my free time) that will provide that
information (https://github.com/gofed/symbols-extractor, feel free to
contribute to the project with ideas who we need to detect or extract
from the code to make more spec file pieces generic and macro-replaceable).
At the end, even if we do "create as many compat Go packages as needed"
or "let's use vendored packages because we can",
we will still need to maintain entire ecosystem in some reasonable way.
From my point of view, the new Go packaging guidelines make it easier
for fresh Go packager to get familiar with Go packaging practices and to
quickly package his/her Go project.
However, from the point of view of entire Go ecosystem (or any micro
ecosystem = I have more projects building from the same dependencies),
it is harder to tune individual packages for the mutual benefit.
=========================================
In overall, I don't like you degrading the current Go packaging
guidelines (although it is still a draft). All the decisions made have
historical reasons. By the time of applying the guidelines, there were
only a few go packages and the severity and impact of the practices were
not eminent nor obvious. Now we know it is not sufficient and there are
things we need to improve. Indeed, 100% agree. So please, pay at least
some respect to all who spent their time and dreamless nights coming
with quick solutions. Cause a lot of us do it in our free-time so we are
limited by time and capacity. What you are trying to solve now was known
two years ago. We already knew about that. For that reason my and my
colleagues started building tooling to detect at least some of the known
problems like API backwards incompatibilities or automation around
updating spec files.
I also suggest to remove every comparison with the Go guidelines draft
and the current state of all Go spec files. Optionally, you can put all
the mentions under "Best practices" or "Thing to avoid" section instead.
There are other parts of the new guidelines I could contradict. However,
doing that we can end up in endless discussion, annoying each other and
burning time that we can use to do important things instead.
So let's divide the workload. If you want to take care of the packaging
guidelines, feel free to do it. Want to improve the spec file? I will be
more than happy to witness that. Do you want to make the Go packaging in
Fedora (and maybe other distributions) more user friendly? Man, that is
awesome and noble. I have no intention to stop you from doing that. But
please, keep in mind, you are not the only one involved in this effort.
Some of us have families, some of us has other things/folks to take care
of. We only do what we can in the time given :). I have/had my use case
I am/was driven, you have yours. That's completely fine and appreciated.
The more use cases we can fine, the more ideas and solutions we can
combine. Let's not degrade results of our efforts, it leads nowhere.
Let's combine the results, our experience. Let's open a google document
(or different collaboration solution if you have one in mind) where we
collect all our requirements. Let's ask other Go (or wannabe) packagers
what are their requirement, what they see as drawbacks of the current
and new guidelines, who is willing to help with the packaging and
maintenance, let's ask more questions.
I am happy to take the tooling side. As I mentioned in the ticket
(https://pagure.io/packaging-committee/issue/382), I shifted my focus on
building tooling that do a lot of analysis and detection which are out
of scope of any guidelines. Even if we agree on the final form of the
guidelines, we still need tooling that will help us with the packaging.
E.g. detection of API backwards incompatibilities, spec file generators
(enhancing and improving over time), detection of go packages that are
no longer needed, running CI/CD, etc.
Regards
Jan
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
