Op 22 sep 2011, om 12:20 heeft Richard Purdie het volgende geschreven:
On Thu, 2011-09-22 at 10:28 +0200, Koen Kooi wrote:
Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende
geschreven:
On Thu, 2011-09-22 at 08:40 +0200, Koen Kooi wrote:
Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende
geschreven:
On Wed, 2011-09-21 at 21:49 +0200, Koen Kooi wrote:
Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende
geschreven:
On 9/21/11 2:38 PM, Philip Balister wrote:
I'm finally trying to get started with oe-core. I'm using a
setup
created with the setup-script created by Koen. Everything
looks ok until
it tried to build cpufrequtils. The recipe tries to fetch the
source
from a git repo at kernel.org. Obviously, the fetch fails.
What is odd, in the "good old days" I would expect this
config to fall
back and use the Angstrom source mirror, but apparently this
is not the
case with oe-core.
Is this a bug or a feature? If this is a bug, is there a plan
to fix it?
If it is a feature, how are we supposed to do builds if
servers go away?
What do we do about GPL compliance?
This is a feature, as people did not want oe-core tied to a
given mirror.
This is the "OE core doesn't generate versioned git tarballs"
problem
people keep talking about. I know I can edit each and every
recipe to
add ';rebasable=true', but I'd like a global solution before
the GPL
police starts looking at the angstrom source mirrors.
Is this a versioned tarball problem or just the tarballs not being
generated at all?
versioned tarball problem, since the source mirror won't overwrite
files with the same name, so unversioned ones are both useless and
bandwidth wasting.
They seem to be working perfectly well for Yocto...
They're also not bandwidth wasting
They are, since for the kernel you need to re-upload the 300MB
tarball
to the source mirror for mirroring to work, even if there were no
real
changes but bitbake repacked it. I know that versioned tarballs
aren't
terribly efficient either, but at least they are deterministic.
Well, it depends on your perspective, I was looking at this from a
user's point of view. The server's bandwidth is nothing compared to
the
users pulling the data.
The angstrom source mirror checks for duplicate filenames to avoid
introducing checksum changes due to upstream stupidity or corrupted
downloads.
Fair enough but you could make an exception for the git scm tarballs
(which follow a fairly identifiable pattern).
That still opens the door for people to upload "old" tarballs with
less revs in them. No solution is perfect, though.
since the fetcher is efficient and
will incrementally update something that is incomplete.
Which won't work currently since kernel.org is down.
This just means there aren't appropriate git:// -> git:// mappings in
people's mirrors files (including Yocto). If git:// -> git:// mappings
don't work, we should fix that but they should.
You mean something like git.kernel.org/scm/torvalds/linux-2.6.git ->
github.com/torvalds/linux.git ?
Anyhow, this is the first time someone has explained that the mirror
tarball improvements that were made to bitbake have not done
everything
they desire. If you want someone on the Yocto team to look at this
I'd
suggest a bug report or I'll take patches to bitbake (as always).
But you do acknowledge the problem? FWIW, I've been arguing for a
global method of enabled versioned tarballs since I first started
using oe-core (called differently back then). I know linux-yocto
stretches the fetcher to its limits, but I'd like this change for all
the recipes using a single git tree with a single srcrev.
Now someone has explained the issue, yes, I can see why it is causing
problems. I do think some of them are of people's own choosing but I'm
trying to be flexible and not make judgement on that.
So we can do something along the lines of what you after but there are
the following considerations:
a) The versioned tarballs do need to be different to what we had
before
in that the tarball will be of a .git directory containing the git
metadata, not just a checkout. Why? This is so we can incrementally
update a git repo if the user changes the source revision as one
example. We're dealing with the git fetcher and limiting ourselves
to a
flat one dimension doesn't make sense.
Agreed on that. The bitbake patch I sent does at least do that :)
b) We need to agree what order the tarballs on the server should be
searched for. The "obvious" order would be:
1. Specific Revision X tarball
2. generic mirror tarball
3. git clone
The problems come about when we have an existing git checkout which
doesn't contain the revision we want.
There currently is a bug in the fetcher that makes it hard to use now
that kernel.org is down. The situation:
I have a recipe that pulls from git (e.g. cpufrequtils.git from
kernel.org) with SRCREV fixed to e.g. deadbeef. DL_DIR/git/
git.kernel.org.utils.cpufrequtils has revision deadbeef. When I do a -
c fetch -f it will try to do an update and fail since kernel.org is
down. I would have expected it to check the existing bare clone for
deadbeef first and not try updating if it is present.
Anyway, I agree on the search order above.
Currently we can just update and
assume that will bring in the revision. If we now support people doing
rebases and other things, we now at least in theory have to:
1. See if the revision exists locally
2. Try pulling
3. If that doesn't work look for a versioned tarball
4. Blow away all fetcher git status from DL_DIR
5. Extract the tarball
Since git supports local fetches the above can be:
1) see if rev exists
2) try pulling
3) try versioned tarball
4) extract tarball into tmpdir, fetch objects into DL_DIR/git2 repo
That would stop things from being blown away. It does allow the
versioned snapshots to differ, but they will always have the rev they
were versioned with.
My big concern with this is rather than having some general
archictecture and standard way of doing this, the fetcher once again
becomes a set of special case if A do B but if C do D first then do E
but don't forget to do A if Z happens type code.
I know virtually nothing of the fetcher code, but I think treating
every tarball as a local mirror that gets fetched into DL_DIR/git2/
repodir would avoid a lot of special casing.
So what I'm saying is I don't want the fetcher to become the
unmaintainable mess the that the code paths in fetch1 were. I don't
honestly know how to add a code path like the above and avoid this :(
The patch on the bitbake list doesn't consider any of this
complexity or
the corner cases
Sadly it's the best thing I can accomplish with my python skills :( It
does solve my #1 need, which is an easy way (ls | grep) to check for
GPL compliance.
regards,
Koen
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
