Hello,
Let me introduce the main ideas, and scope of the ports management
system, we currently call ports 2.0.
First, the confusion may result from the name, which looks like having
ambitions to completely replace ports system and infrastructure.
If we call "infrastructure" the ports directory tree with makefiles,
options, knobs etc. we won't replace/change or even touch this in this
project. The infrastructure is not the bad in current system, but the good.
If we call management of the ports "the process" (either using make,
packages or tools like portupgrade), this is what ports 2.0 should solve
differently, and much more efficiently than any existing tool.
Here we see the main problem in recursive make, which prevents "the
process" from being parallelized and utilize resources in an optimal
way. My first draft for improvement was described here:
http://moon.felk.cvut.cz/~fischeo1/portwizard/
But now we have mouch more powerfull solution. With Aryeh, we designed
an algorithm, that has a dependency graph of packages on it's input, and
detects allways the maximum of actions, that can be processed in
parallel. And the more, it can pass those actions to several processing
thread pools and thus expolit resources optimally by type.
In other words: If we have 4CPUs and 1GB internet connection, we can use
eg. 4 parallel builds and eg. 10 parallel fetches allways, when there
exists a set of such actions, that can be done in parallel.
What is other benefit, is incrementality. You don't have to first choose
all packages to be installed/upgraded. You just choose a set, and while
"the process" running, you can add next packages with no harm to
parallel actions detection. This only needs server -> client model. The
installation/upgrade process must be one (with several threads).
Right now we are working on prove of concept of underlaying algorithm, a
prototype, and a detailed specification. The prototype will be in Java,
since it is extremely easy to implement it, but I guess, it's not
desired to have java dependency for such system management tool.
That's for introduction.
Regards
Ondrej Fischer
Aryeh M. Friedman wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
David Wood wrote:
| [Additional CCs added by Aryeh noted, but left untouched]
I removed the OP, Simon and Mezz since this has nothing to boost any
more and I changed the Alex Ryba cc to Alejandro Pulver because the
first was a misaddressing on my part
|
| I wanted to retitle this post, but couldn't come up with a summary
of what I was trying to say.
I think the subject now fits better
|
|
| In message <[EMAIL PROTECTED]>, Aryeh M. Friedman
<[EMAIL PROTECTED]> writes
|> I am top posting because my comments are general and not in
relationship to any given point. I think Mezz along with Simon both
the right way to handle it for now. There are several projects that
I am not directly involved designed to tackle this sort of issue with
ports 2.0 being the right long term solution but not needed to solve
this issue per se so I will not discuss it here. Most of the
projects can be found on Ale's PortsToDo wiki
(wiki.freebsd.org/PortsToDo) the main one not found on there is Jim
Staplton's virtual ports DB (which I think is either in the committ
queue or should be since both me an Ale have signed off on it as being
a good idea). The issue is really not the infrastruct as you state
but the more patchs like this we make the more likely it is the
infrastruct will become problematic down the road.
|
| Forgive me, Aryeh, but you've taken a specific point I made and
dismissed it saying that "the infrastructure needs a complete overhaul
and I'm the right person to head that work", though you have
acknowledged that there is other work going on.
We seem to have a fundimental failure to communicate. I was saying
*SPECFICALLY* that ports 2.0 (at least as currently envisioned) was
not the right way to handle this for the time bing (only that if and
when ports 2.0 does become adopted that it needs to handle such
situations as well or better then the current system).... my original
comment was made solely as a user of ports that depend on boost and a
desire to make it so there was no python wackiness (which Simon's and
Mezz's suggestion does just fine).
As to other people's work odds are that the very core of Ports 2.0 is
going to be based on a concept put forward by Ondrej Fischer (I will
leave it to him to discuss the details) and Ports 2.0 *ONLY* refers to
item 5 in Ale's todo list (which is less then 10% of the total list)
|
| I know you will argue you aren't making a reply to my post, so why
didn't you trim all my text instead of quoting it? I believe you are
replying - and that your reply is pretty close to being non sequitur.
You're the one who attempted to put words into my mouth (totally
without cause or instigation) by attempting to make a linkage between
my concern about a given current port and my other work in the ports
system in general. The two are completely unrelated.
|
| I am not interested in pointless points scoring; I have better
things to do. What I want to do is understand the issue..
|
|
| I am no committer - just a mere ports maintainer. Freshports doesn't
dig up any ports you maintain, at least not with an email address
beginning with aryeh or containing fried.
Since I work closely with Ale who is a committer for book keeping
simplicity he is the maintainer of record for several ports we
co-maintain: devel/aegis, devel/thistest (besides I wrote the
underlaying application), and sysutils/fusefs-ntfs
|
| I do not believe that this problem is one that brings into question
the whole infrastructure. It certainly isn't "the more patchs (sic)
that we make the more likely it is that the infrastruct (sic) will
become problematic down the road". Not all evolutionary software
engineering is bad, especially if modules are rewritten when
appropriate. Of course I accept the possibility of 'bit rot' setting
in when code has been through many hands and has many small changes
applied. In this case, I believe that things are getting better, not
worse or less maintainable.
Neither do I (at least until we break the 25k ports or so [total
guess]). As to evolutionary software development it is the primary
model I use and one of the things it teaches is to know when the
current design is starting to reach it's limits and change it either
by some structural refactoring (if you are lucky) and/or rewrite of
major subsystems (if your unlucky)... ports 2.0 takes the point of
view that the user level ports system is pretty much good the way it
is but the way depends are handled sucks and that this will evenually
make the system unworkable unless a completely new depend system is
developed and that is why I am always refering to "Recursive Make
Considered Harmful"
|
| This problem was known about some while back and there's no need to
do any fresh engineering. The delay in having a solution available was
the need for changes in the base system to support bsd.options.mk.
Base system changes mean waiting until all versions of FreeBSD that
lack the necessary support are End of Life. After 31 May, ports will
be able to process their options and set a dependency on python before
including bsd.pre.port.mk.
Please take your head out of "Aryeh is evil" and see my comments for
what they where (an attempt to ask if the solution was implemented
visa vie boost and boost only!)
|
| A simple and elegant solution will soon be available to solve what
is essentially a circular dependency at the moment.
It is already solved if you read the reply from simon.
|
|
|
| I see two issues here, both stemming from the complexities of
dependency between ports. Firstly, there's a need to look for the best
possible solutions with the tools we have now as well as to look for
continuing improvement. Secondly, there's the need to educate both
maintainers and administrators on how best to handle the complexities
that can occur.
|
| I think everyone would agree that the current documentation isn't
the best. In stating this, I believe I'm restating Mark Linimon's well
known criticism of the current Porters' Handbook. It is not my
intention to hurt anyone by what I say. The way out of this is, of
course, for people to pay attention to the documentation - but those
best placed to do that are some of the busiest people already within
the FreeBSD project. Having worked alongside technical authors, I
realise what a specialist (and impressive) skill set they have.
This is a non-Ports 2.0 work item already on Ale's todo list
spefically to create a more general guide then the porters handbook
(his working title is the ports handbook) which is meant for both
maintainers and admins.
|
| At the moment, it is hard for a maintainer to discover current best
practice. I suspect this is partly why the same issue is handled in
different ways by different ports (as I mentioned above with
Kerberos). Meanwhile, it is hard for system administrators to discover
how best to use the richness of ports.
I doubt if anyone ever even attempted to compile a best practices list
(one Ale's goals with the porters guide). As to your assertion that
all we need is to make maintainers aware of complexities is ass
backwards if you ask me when such complexities are symptoms of either
bad practices or a bad underlaying system (I personal think it is both)
|
|
| System administrators need the freedom to make whatever decisions
they feel are appropriate (as I said, I wouldn't want ports
unnecessarily depending on OpenSSL or the Cyrus SASL library).
However, features that help administrators manage this richness, such
as portconf, are not that well known about, especially by the less
technical users.
|
|
|
| Aryeh - you seem to have something against slave ports. At times,
they are very useful. They make the creation and maintenance of
client/server ports easy - for example, databases/mysql50-client is a
slave port of databases/mysql50-server.
In theory I have nothing against them since they are the best we can
do with the current system, *BUT* maintainers all too often
inadvertently create problems with them such as the old boost did.
|
| The reason for both my posts is a feeling that you haven't
understood *why* the current situation is as it is, Aryeh. A reply
along the lines of "we're working on a complete replacement of the
whole infrastructure, and that will solve it" doesn't help with
understanding the specific issue that has arisen. You can't offer any
guarantee that your new system will solve all the problems -
especially if you do not take the time to understand the weaknesses in
the current system as well as its strengths.
I think your the one who misunderstood completely by some how assuming
I was pushing a non-maintstream agenda (which I agree ports 2.0
currently is) instead of simple making a request as user of a specific
port.
|
| If you bring forward coherent proposals and a proof of concept for
Ports 2.0, I will certainly give what input I can at that stage. For
now, we live with and continue to improve what we have, rather than
looking only to what might replace it if it's found to be better after
careful evaluation.
Depend on what Ondrej Fischer feels about discussing his work we can
start to put forward such concrete stuff right now (the design is 99%
done we are just testing the underlaying alrogithms)
|
|
| Maintainers have to work with OPTIONS as they are. They are a
valuable feature, even if they don't offer all the functionality
wished for.
Where the (*&(* did you get the impression I was against OPTIONS
and/or gnobs?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (FreeBSD)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkgZHIEACgkQk8GFzCrQm4CZegCdFhZ3TD8vOpPMyzA2BBHpraLJ
Dw0Anik2dR4BuSZB2OucOzxR/Xiu3t/T
=3UyF
-----END PGP SIGNATURE-----
_______________________________________________
freebsd-ports@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
