Hi Matt,
I’m not sure whether those tests will actually find this specific issue. The
tests that I linked to test Spark’s Zookeeper-based multi-master mode, whereas
it sounds like you’re seeing this issue in regular standalone cluster. In
those tests, the workers disconnect from the master be
Thanks Josh! These tests seem to cover the cases I'm looking for already =).
What's interesting though is that we still ran into SPARK-3736 despite such
integration tests being in place to catch it - specifically, the case when
the master disconnects and reconnects, the workers should reconnect to
There are some end-to-end integration tests of Master <-> Worker
fault-tolerance in
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/FaultToleranceTest.scala
I’ve actually been working to develop a more generalized Docker-based
integration-testing framewor
I think on a higher level I also want to ask why such unit testing has not
actually been done in this codebase. If it's not a common practice to test
message passing then I'm fine with leaving out the unit test, however I'm
more curious as to why such testing was not done before.
On Wed, Oct 15, 2
You can call resolve method on ActorSelection.resolveOne() to see if the
actor is still there or the path is correct. The method returns a future
and you can wait for it with timeout. This way, you know the actor is live
or already dead or incorrect.
Another way, is to send Identify method to Acto
What's happening when I do this is that the Worker tries to get the Master
actor by calling context.actorSelection(), and the RegisterWorker message
gets sent to the dead letters mailbox instead of being picked up by
expectMsg. I'm new to Akka and I've tried various ways to registering a
"mock" mas
I don’t think there are test cases for Worker itself
You can
val actorRef = TestActorRef[Master](Props(classOf[Master], ...))(actorSystem)
actorRef.underlyingActor.receive(Heartbeat)
and use expectMsg to test if Master can reply correct message by assuming
Worker is absolutely correct
Thanks, the example was helpful.
However, testing the Worker itself is a lot more complicated than
WorkerWatcher, since the Worker class is quite a bit more complex. Are
there any tests that inspect the Worker itself?
Thanks,
-Matt Cheah
On Tue, Oct 14, 2014 at 6:40 PM, Nan Zhu wrote:
> You c
You can use akka testkit
Example:
https://github.com/apache/spark/blob/ef4ff00f87a4e8d38866f163f01741c2673e41da/core/src/test/scala/org/apache/spark/deploy/worker/WorkerWatcherSuite.scala
--
Nan Zhu
On Tuesday, October 14, 2014 at 9:17 PM, Matthew Cheah wrote:
> Hi everyone,
>
> I’m ad
Hi everyone,
I’m adding some new message passing between the Master and Worker actors in
order to address https://issues.apache.org/jira/browse/SPARK-3736 .
I was wondering if these kinds of interactions are tested in the automated
Jenkins test suite, and if so, where I could find some examples t
10 matches
Mail list logo
