Thanks.  That looks pretty similar to what I'm doing, with the
difference being getPeers vs getMemoryStatus.  Seems like they're both
backed by the same blockManagerInfo, but getPeers is filtering in a
way that looks close to what I need.  Is there a reason to prefer
getMemoryStatus?

On Thu, Mar 3, 2016 at 5:15 PM, Shixiong(Ryan) Zhu
<shixi...@databricks.com> wrote:
> You can take a look at
> "org.apache.spark.streaming.scheduler.ReceiverTracker#getExecutors"
>
> On Thu, Mar 3, 2016 at 3:10 PM, Reynold Xin <r...@databricks.com> wrote:
>>
>> What do you mean by consistent? Throughout the life cycle of an app, the
>> executors can come and go and as a result really has no consistency. Do you
>> just need it for a specific job?
>>
>>
>>
>> On Thu, Mar 3, 2016 at 3:08 PM, Cody Koeninger <c...@koeninger.org> wrote:
>>>
>>> I need getPreferredLocations to choose a consistent executor for a
>>> given partition in a stream.  In order to do that, I need to know what
>>> the current executors are.
>>>
>>> I'm currently grabbing them from the block manager master .getPeers(),
>>> which works, but I don't know if that's the most reasonable way to do
>>> it.
>>>
>>> Relevant code:
>>>
>>>
>>> https://github.com/koeninger/spark-1/blob/aaef0fc6e7e3aae18e4e03271bc0707d09d243e4/external/kafka-beta/src/main/scala/org/apache/spark/streaming/kafka/KafkaRDD.scala#L107
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>>> For additional commands, e-mail: dev-h...@spark.apache.org
>>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Reply via email to