** No longer affects: ubuntu
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage notifications about this bug go to:
https://bugs.la
** Changed in: linaro-ci
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage notificati
With the pipelining disabled, using the main archive for packages seems to have
fixed the issue.
Have not seen packages installation failure because of the archive problems and
the ec2 slaves deploy are quite stable now.
Closing this bug now, reopen it if it re appears.
Thanks!!!
Deepti.
** Ch
As suggested in the comment #53 have disabled pipeline in apt, and
switched to s3 mirror and I see the following error :
W: Failed to fetch http://us-
east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/dists/precise/main/source/Sources
403 Forbidden
W: Failed to fetch http://us-
east-1.ec2.ar
** Changed in: linaro-ci
Status: New => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage notifications about thi
> so, what are the exact changes on a-b* that we have that fixes this
problem ?
And it "fixed" it for Ubuntu version we use on android-build (Natty).
They may be different/additional issues with OS versions we use on ci.*.
That's why I think that we should just switch to custom AMI as a
sustainabl
> + + echo Acquire::http::Pipeline-Depth 0;^M
> tee /etc/apt/apt.conf.d/99no-pipelining^M
> Acquire::http::Pipeline-Depth 0;^M
Well, this command exactly disables pipelining.
> so, what are the exact changes on a-b* that we have that fixes this
problem ?
The best way probably will be to compare
On Wed, Jun 6, 2012 at 12:17 PM, Paul Sokolovsky
<932...@bugs.launchpad.net>wrote:
> Today we're having 403's on ci.linaro.org again, see attachment for a
> sample log.
>
>
** Attachment added: "i-18e33761.log"
>
> https://bugs.launchpad.net/linaro-android-infrastructure/+bug/932088/+attachment/3
Today we're having 403's on ci.linaro.org again, see attachment for a
sample log.
** Attachment added: "i-18e33761.log"
https://bugs.launchpad.net/linaro-android-infrastructure/+bug/932088/+attachment/3177116/+files/i-18e33761.log
--
You received this bug notification because you are a memb
This seems to be stably fixed for linaro-android-infrastructure, closing
in it (may still be pertinent for linaro-ci and ubuntu).
** Changed in: linaro-android-infrastructure
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Bugs
** Changed in: linaro-ci
Milestone: 2012.05 => 2012.06
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage notifications about t
Just for stats, Linaro's https://android-build.linaro.org/jenkins/
didn't have any issues lately (is on S3 mirror w/o pipelining).
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu
similar errors with me as well since couple of hours
W: Failed to fetch http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists
/precise-updates/universe/binary-amd64/Packages 404 Not Found
W: Failed to fetch http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists
/precise-updates/main/binary-i386/Pa
I've been having failures in cbuild starting a natty instance for the
last 24 hours. Switching to S3 with no pipelining works around the
problem.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
This exact problem a few minutes ago and I've been suffering it during
the last hour
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To ma
This exact problem is happening for me for about 8 hours now. Is anybody
else seeing this, too?
http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ natty-updates/main [...]
403 Forbidden
I also tried a workaround proposed by Scott Moser from
http://markmail.org/message/ihueaulwm7k7swkb#query:+page:1
Thanks Scott!
I've just re-enabled S3 mirror, it works ok now:
Get:23 http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/
natty/main g++-4.5 amd64 4.5.2-8ubuntu4 [6,511 kB]
Get:24 http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/
natty/main g++ amd64 4:4.5.2-1ubuntu3
On Tue, 22 May 2012, Paul Sokolovsky wrote:
> As of now (well, for couple of hours at least) we're having following
> errors. These are known to happen from time to time, and dissolve after
> some time (we usually attribute it to to ongoing package repository
> update), but this ones seems like ta
Switching back to standard EC2 mirror (from S3 one) fixed that for now.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage notifica
As of now (well, for couple of hours at least) we're having following
errors. These are known to happen from time to time, and dissolve after
some time (we usually attribute it to to ongoing package repository
update), but this ones seems like taking longer then usual, so figure
I'd post about it h
we hit again 404 Not Found error on ci.linaro.org since yesterday. As
mentioned in previous comments, "us-east-1.ec2." is duplicated. afaics,
we don't use cloud-init, only sed call manipulates sources.list.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is su
** Changed in: linaro-ci
Assignee: (unassigned) => Deepti B. Kalakeri (deeptik)
** Changed in: linaro-ci
Milestone: None => 2012.05
** Changed in: linaro-ci
Importance: Undecided => High
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscrib
** Changed in: linaro-android-infrastructure
Milestone: 2012.04 => 2012.05
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage n
On Wed, 25 Apr 2012, Deepti B. Kalakeri wrote:
> On Wed, Apr 25, 2012 at 8:15 PM, Ben Howard
> <932...@bugs.launchpad.net>wrote:
>
> > Deepti...you are using the S3 bucket of:
> > us-east-1.ec2.us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com
> >
> > Which should be:
> > us-east-1.ec2.archive.ubu
On Wed, Apr 25, 2012 at 8:15 PM, Ben Howard
<932...@bugs.launchpad.net>wrote:
> Deepti...you are using the S3 bucket of:
> us-east-1.ec2.us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com
>
> Which should be:
> us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com
>
> Note the extra "us-east-1.ec2." in
Deepti...you are using the S3 bucket of:
us-east-1.ec2.us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com
Which should be:
us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com
Note the extra "us-east-1.ec2." in the URL's from the error logs. If you
remove that, it should work for you.
--
You receiv
Deepti, Ben pointed out, the log you posted has bad apt sources lines:
http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com
not
http://us-east-1.ec2.us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is
Log (i-5adcf83d_slave.log) for the slave using the ami-b5ea34dc instance
which failed with the 404 error attached.
** Attachment added: "Log_of_slave_404_error"
https://bugs.launchpad.net/linaro-android-infrastructure/+bug/932088/+attachment/3103295/+files/i-5adcf83d_slave.log
--
You receiv
On Tue, 24 Apr 2012, Deepti B. Kalakeri wrote:
> Well I have logs for the slave that used ami instance ami-b5ea34dc and
> failed to start with the 404 failure in the morning.
> Now the same slaves using ami-b5ea34dc instance are able to start without
> the problem.
> I am not an EC2 expert so am
On Tue, Apr 24, 2012 at 6:30 PM, Scott Moser wrote:
> On Tue, 24 Apr 2012, Deepti B. Kalakeri wrote:
>
> > According to the comments in the bug above we have been able to resolve
> the 404 Not Found error and HashSum issue by using the following 2 things.
> > 1) sudo sed -i.dist 's,archive.ubunt
On Tue, 24 Apr 2012, Deepti B. Kalakeri wrote:
> According to the comments in the bug above we have been able to resolve the
> 404 Not Found error and HashSum issue by using the following 2 things.
> 1) sudo sed -i.dist
> 's,archive.ubuntu.com,archive.ubuntu.com.s3.amazonaws.com,g'
> /etc/apt/
According to the comments in the bug above we have been able to resolve the 404
Not Found error and HashSum issue by using the following 2 things.
1) sudo sed -i.dist
's,archive.ubuntu.com,archive.ubuntu.com.s3.amazonaws.com,g'
/etc/apt/sources.list
2) echo "Acquire::http::Pipeline-Depth 0;" |
Update on android-build.linaro.org (uses S3):
$ grep -l -a -E '^Failed to fetch.+s3' * | tail -n3
2012-04-03-slave-i-69bb4e0e.log
2012-04-04-slave-i-0f32c068.log
2012-04-04-slave-i-a5a153c2.log
$ grep -l -a "Hash Sum mismatch" * | tail -n3
2012-04-04-slave-i-25629242.log
2012-04-04-slave-i-2d1beb
But on ci.linaro.org (non-S3 mirror):
$ grep -l -a "Hash Sum mismatch" *|wc
40 401309
So, I'm going to leverage same config to ci.linaro.org and see how it
goes (it uses different set of AMIs, so results may not match android-
build).
--
You received this bug notification because
Yes, those slave log links expire with slaves, that's why I pasted it
in . https://bugs.launchpad.net/linaro-android-
infrastructure/+bug/932088/comments/27/+download should give the same
info as if it was attached.
And yes, this latest issue and log from ci.linaro.org, which is not yet
uses S3 m
Paul,
I also just noticed, you're not using the S3 backed mirrors:
sudo sed -i.dist
's,archive.ubuntu.com,archive.ubuntu.com.s3.amazonaws.com,g'
/etc/apt/sources.list
I'm not saying there was not a problem, only that I thought (and hoped)
you were using the s3 mirrors. I've just tried ri
Paul, your link to https://ci.linaro.org/jenkins/computer/i-185d097f/log
does not work for me, could you attach full log ?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 pack
Also, if you look that log carefully, you can see following weird thing:
we do two "apt-get update"s in row, with sleep 2 inbetween. First update
hits *only*
http://security.ubuntu.com
http://archive.ubuntu.com
doesn't hit in-EC2 mirror at all. Second updates starts to hit http
://us-east-1.ec2.a
That's with ami-b5ea34dc just in case.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage notifications about this bug go to:
https
We just had a case of 10+ broken builds slave being started up in short
time. Typical Jenkins instance init log:
https://ci.linaro.org/jenkins/computer/i-185d097f/log
As you can see, "Acquire::http::Pipeline-Depth 0;" is set, but still
there's hashsum mismatch error (but the fact that dozen of sl
The 403s are in fact weird, but they are explainable. The reason for
them is that the 'Sources', and 'Packages' files do not exist in
uncompressed form on S3 *or* on the *.archive.ubuntu.com mirrors (see
http://us.archive.ubuntu.com/ubuntu/dists/natty-updates/universe/source/
)
The reason you se
Ok, now I eyeballed few logs and tentative conclusion - thank you very
much, turning off pipelining seem to have solved our issues! Not even
those weird 403's don't pop up after 2012-04-04. If you're interested
what it was here's example:
2012-04-04-slave-i-a5a153c2.log:W: Failed to fetch
http://
On the other hand:
$ grep -l -a -E '^Failed to fetch.+s3' * | tail -n3
2012-04-03-slave-i-69bb4e0e.log
2012-04-04-slave-i-0f32c068.log
2012-04-04-slave-i-a5a153c2.log
Last line matches the line in comment #15 which I made shortly before
turning off pipelining. Which means we didn't have (that spe
I personally not sure what happens here, but:
Just opening http://us-
east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/dists/natty-
updates/ in a browser leads to 403 XML. Ok, suppose it's limited to only
EC2. From out EC2 machine:
ubuntu@ip-10-243-34-224:~$ wget
http://us-east-1.ec2.archiv
James, do you have some logs? I am not able to replicate this.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage notifications abo
James, Thanks for the info, we're looking into it.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage notifications about this bug
Our slaves only seem to be failing when accessing http://us-
east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/dists/natty-
updates/ now. ubuntu/dists/natty is fine. Is there a permission problem
with that directory?
--
You received this bug notification because you are a member of Ubuntu
Bug
On Wed, 4 Apr 2012, Paul Sokolovsky wrote:
> Previous note is about applying to android-build.linaro.org. Now also
> applied to ci.linaro.org, where have slave apt-get issues, albeit errors
> we get there are different %).
Thank you.
--
You received this bug notification because you are a membe
On Wed, 4 Apr 2012, Paul Sokolovsky wrote:
> $ grep -l -a -E '^Failed to fetch.+s3' *
> ...
> 2012-03-06-slave-i-63954a07.log
> 2012-04-02-slave-i-7fff1c18.log
> 2012-04-04-slave-i-0f32c068.log
> 2012-04-04-slave-i-a5a153c2.log
>
> So yes, we're still having it almost everyday, it's just auto-ret
Previous note is about applying to android-build.linaro.org. Now also
applied to ci.linaro.org, where have slave apt-get issues, albeit errors
we get there are different %).
** Changed in: linaro-android-infrastructure
Status: Fix Released => In Progress
** Also affects: linaro-ci
Impo
echo "Acquire::http::Pipeline-Depth 0;" | sudo tee /etc/apt/apt.conf.d
/99no-pipelining - was added to our EC2 instance init scripts, I'll be
watching how it goes.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.
$ grep -l -a -E '^Failed to fetch.+s3' *
...
2012-03-06-slave-i-63954a07.log
2012-03-06-slave-i-7d12c119.log
2012-03-06-slave-i-e1c51a85.log
2012-03-07-slave-i-31c66355.log
2012-03-07-slave-i-fbd9029f.log
2012-03-08-slave-i-5961c23d.log
2012-03-08-slave-i-5f40ee3b.log
2012-03-08-slave-i-c97cdfad.lo
Sorry for delays with responses. The story on our side was that we
implemented auto-retries for failed apt-get invocations in our scripts
with exponential back-off on 2012-03-06, and that "fixed" it for us, at
least I never saw slaves failed to launch. Soon after that we were
swamped into other urg
Hi Paul,
Some more debugging info is in bug 948461, but it looks like, for the moment
if you set apt to use 'Acquire::http::Pipeline-Depth=0', then many of your
errors may go away.
Could you please try in your builds to do (prior to even apt-get update):
$ echo "Acquire::http::Pipeline-Depth
** Tags added: mirrors
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage notifications about this bug go to:
https://bugs.launchpa
I found another bug in APT that appears to explain the hashsum/size
mismatches. See Bug 948461
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failu
The "416" errors are caused by Bug 798023.
I did find a minor bug in the flipping code that activates the S3 meta-
data. It should shake out by later tonight. Can you let me know what the
results of the next nights tests are?
--
You received this bug notification because you are a member of Ubun
Here's good AWS forum thread from 2008:
https://forums.aws.amazon.com/thread.jspa?threadID=21514 . According to
it, "403 Forbidden" in AWS-speak means "there may be unusually long
propagation delay".
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribe
This latest round of issues is tracked in our bug lp:941784, which may
have more info.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To
We worked that around by adding retries for apt-get operations with
exponential backoff and eventual abort after 6 retries (64s wait last
time). Of 51 instances which were launched since that fix was deployed,
18 had retries kick in. And 10 still failed in the end (i.e. after 6
retries). However, a
We started to see following issues with beta S3 mirror:
Get:40 http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/
natty/main zip amd64 3.0-3build1 [265 kB]
Failed to fetch
http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/d/dpkg/libdpkg-perl_1.16.0~ubuntu7.1
Have been working ok for android-build.linaro.org , and at the end of
our monthly cycle, we'd like to close this in linaro-android-
infrastructure project. We still hope this we be open in Ubuntu for
reference and to allow to communicate any issues/observations. Thanks.
** Changed in: linaro-andr
Deployed, I mean, source.list editing per
http://groups.google.com/group/ec2ubuntu/browse_thread/thread/507f58dd51b6b631
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 packag
Ok this was deployed for https://android-build.linaro.org/ . Watching
builds, it all goes well.
We'll share issues/comments (if any) of how it goes for Ubuntu EC2 team
info here.
** Changed in: linaro-android-infrastructure
Milestone: None => 2012.02
** Changed in: linaro-android-infrastruc
** Branch linked: lp:~pfalcon/linaro-android-build-tools/ec2-s3-mirror
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/932088
Title:
Ubuntu EC2 package mirror intermitent failures
To manage notificat
** Changed in: linaro-android-infrastructure
Importance: Undecided => High
** Changed in: linaro-android-infrastructure
Status: New => Triaged
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/9
Thanks for quick reply, Scott. I'll RFC switching to new S3 sources, and
unless our service stakeholder raise serious concerns with that, going
to try them soon.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.ne
Hi,
We're hoping to [very soon] retire the old EC2 mirror infrastructure and
replace it with mirrors backed by S3 [1].
When they go live, the rollover will be transparent to the user, but right
now you can selectively choose the S3 mirrors, which should provide better
uptime and speed.
If
** Project changed: ubuntu-on-ec2 => ubuntu
** Tags added: ec2-images
** Changed in: ubuntu
Importance: Undecided => High
** Changed in: ubuntu
Status: New => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https
69 matches
Mail list logo
