We tested that patch from aarondav's branch, and are no longer seeing that
deadlock. Seems to have solved the problem, at least for us.
On Mon, Jul 14, 2014 at 7:22 PM, Patrick Wendell wrote:
> Andrew and Gary,
>
> Would you guys be able to test
> https://github.com/apache/spark/pull/1409/file
Hey Andrew,
Yeah, that would be preferable. Definitely worth investigating both,
but the regression is more pressing at the moment.
- Patrick
On Mon, Jul 14, 2014 at 10:02 PM, Andrew Ash wrote:
> I don't believe mine is a regression. But it is related to thread safety on
> Hadoop Configuration
I don't believe mine is a regression. But it is related to thread safety on
Hadoop Configuration objects. Should I start a new thread?
On Jul 15, 2014 12:55 AM, "Patrick Wendell" wrote:
> Andrew is your issue also a regression from 1.0.0 to 1.0.1? The
> immediate priority is addressing regression
Andrew is your issue also a regression from 1.0.0 to 1.0.1? The
immediate priority is addressing regressions between these two
releases.
On Mon, Jul 14, 2014 at 9:05 PM, Andrew Ash wrote:
> I'm not sure either of those PRs will fix the concurrent adds to
> Configuration issue I observed. I've got
I'm not sure either of those PRs will fix the concurrent adds to
Configuration issue I observed. I've got a stack trace and writeup I'll
share in an hour or two (traveling today).
On Jul 14, 2014 9:50 PM, "scwf" wrote:
> hi，Cody
> i met this issue days before and i post a PR for this(
> https:/
hi，Cody
i met this issue days before and i post a PR for this(
https://github.com/apache/spark/pull/1385)
it's very strange that if i synchronize conf it will deadlock but it is ok when
synchronize initLocalJobConfFuncOpt
Here's the entire jstack output.
On Mon, Jul 14, 2014 at 4:44 PM, P
The patch won't solve the problem where two people try to add a
configuration option at the same time, but I think there is currently an
issue where two people can try to initialize the Configuration at the same
time and still run into a ConcurrentModificationException. This at least
reduces (sligh
We'll try to run a build tomorrow AM.
On Mon, Jul 14, 2014 at 7:22 PM, Patrick Wendell wrote:
> Andrew and Gary,
>
> Would you guys be able to test
> https://github.com/apache/spark/pull/1409/files and see if it solves
> your problem?
>
> - Patrick
>
> On Mon, Jul 14, 2014 at 4:18 PM, Andrew As
Andrew and Gary,
Would you guys be able to test
https://github.com/apache/spark/pull/1409/files and see if it solves
your problem?
- Patrick
On Mon, Jul 14, 2014 at 4:18 PM, Andrew Ash wrote:
> I observed a deadlock here when using the AvroInputFormat as well. The
> short of the issue is that t
I observed a deadlock here when using the AvroInputFormat as well. The
short of the issue is that there's one configuration object per JVM, but
multiple threads, one for each task. If each thread attempts to add a
configuration option to the Configuration object at once you get issues
because HashM
HI Patrick, I'm not aware of another place where the access happens, but
it's possible that it does. The original fix synchronized on the
broadcastConf object and someone reported the same exception.
On Mon, Jul 14, 2014 at 3:57 PM, Patrick Wendell wrote:
> Hey Nishkam,
>
> Aaron's fix should p
We use the Hadoop configuration inside of our code executing on Spark as we
need to list out files in the path. Maybe that is why it is exposed for us.
On Mon, Jul 14, 2014 at 6:57 PM, Patrick Wendell wrote:
> Hey Nishkam,
>
> Aaron's fix should prevent two concurrent accesses to getJobConf (a
Hey Nishkam,
Aaron's fix should prevent two concurrent accesses to getJobConf (and
the Hadoop code therein). But if there is code elsewhere that tries to
mutate the configuration, then I could see how we might still have the
ConcurrentModificationException.
I looked at your patch for HADOOP-10456
Hi Aaron, I'm not sure if synchronizing on an arbitrary lock object would
help. I suspect we will start seeing the ConcurrentModificationException
again. The right fix has gone into Hadoop through 10456. Unfortunately, I
don't have any bright ideas on how to synchronize this at the Spark level
with
The full jstack would still be useful, but our current working theory is
that this is due to the fact that Configuration#loadDefaults goes through
every Configuration object that was ever created (via
Configuration.REGISTRY) and locks it, thus introducing a dependency from
new Configuration to old,
Hey Cody,
This Jstack seems truncated, would you mind giving the entire stack
trace? For the second thread, for instance, we can't see where the
lock is being acquired.
- Patrick
On Mon, Jul 14, 2014 at 1:42 PM, Cody Koeninger
wrote:
> Hi all, just wanted to give a heads up that we're seeing a
Hi all, just wanted to give a heads up that we're seeing a reproducible
deadlock with spark 1.0.1 with 2.3.0-mr1-cdh5.0.2
If jira is a better place for this, apologies in advance - figured talking
about it on the mailing list was friendlier than randomly (re)opening jira
tickets.
I know Gary had
17 matches
Mail list logo
