That error message occurs when the Mesos scheduler tries to accept an offer
that doesn't exist. It should never happen. Can you submit a JIRA and cc
me to it? Also, what libmesos and mesos master version are you running?
On Wed, Aug 17, 2016 at 9:23 AM, vr spark <vrspark...@gmail.com> wrote:
> My code is very simple, if i use other hive tables, my code works fine. This
> particular table (virtual view) is huge and might have more metadata.
>
> It has only two columns.
>
> virtual view name is : cluster_table
>
> # col_name data_type
>
> ln string
>
> parti int
>
>
> here is snippet...
>
> from pyspark import SparkConf, SparkContext
>
> from pyspark.sql import HiveContext
>
> import pyspark.sql
>
> import json
>
> myconf=SparkConf().setAppName("sql")
>
> spcont=SparkContext(conf=myconf)
>
> sqlcont=HiveContext(spcont)
>
> res=sqlcont.sql("select parti FROM h.cluster_table WHERE parti > 408910
> and parti <408911 limit 10")
>
> print res.printSchema()
>
> print 'res'
>
> print res
>
> df=res.collect()
>
> print 'after collect'
>
> print df
>
>
> Here is the ouput after i submit the job
>
> I0817 09:18:40.606465 31409 sched.cpp:262] New master detected at
> master@x.y.17.56:6750
>
> I0817 09:18:40.607461 31409 sched.cpp:272] No credentials provided.
> Attempting to register without authentication
>
> I0817 09:18:40.612763 31409 sched.cpp:641] Framework registered with
> b859f2f3-7484-482d-8c0d-35bd91c1ad0a-6336
>
> 16/08/17 09:18:57 WARN ObjectStore: Version information not found in
> metastore. hive.metastore.schema.verification is not enabled so recording
> the schema version 1.2.0
>
> 16/08/17 09:18:57 WARN ObjectStore: Failed to get database default,
> returning NoSuchObjectException
>
> root
>
> |-- parti: integer (nullable = true)
>
>
> None
>
> res
>
> DataFrame[partition_epoch_hourtenth: int]
>
> 2016-08-17 09:19:20,648:31315(0x7fafebfb1700):ZOO_WARN@
> zookeeper_interest@1557: Exceeded deadline by 19ms
>
> 2016-08-17 09:19:30,662:31315(0x7fafebfb1700):ZOO_WARN@
> zookeeper_interest@1557: Exceeded deadline by 13ms
>
> W0817 09:20:01.715824 31412 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676564
>
> W0817 09:20:01.716455 31412 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676630
>
> W0817 09:20:01.716645 31412 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676713
>
> W0817 09:20:01.724409 31412 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676554
>
> W0817 09:20:01.724728 31412 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676555
>
> W0817 09:20:01.724936 31412 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676556
>
> W0817 09:20:01.725126 31412 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676557
>
> W0817 09:20:01.725309 31412 sched.cpp:1195] Attempting to accept an
> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O168676558.
>
> and many more lines like this on the screen with similar message....
>
>
>
> On Wed, Aug 17, 2016 at 9:08 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
>> Please include user@ in your reply.
>>
>> Can you reveal the snippet of hive sql ?
>>
>> On Wed, Aug 17, 2016 at 9:04 AM, vr spark <vrspark...@gmail.com> wrote:
>>
>>> spark 1.6.1
>>> mesos
>>> job is running for like 10-15 minutes and giving this message and i
>>> killed it.
>>>
>>> In this job, i am creating data frame from a hive sql. There are other
>>> similar jobs which work fine
>>>
>>> On Wed, Aug 17, 2016 at 8:52 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>>>
>>>> Can you provide more information ?
>>>>
>>>> Were you running on YARN ?
>>>> Which version of Spark are you using ?
>>>>
>>>> Was your job failing ?
>>>>
>>>> Thanks
>>>>
>>>> On Wed, Aug 17, 2016 at 8:46 AM, vr spark <vrspark...@gmail.com> wrote:
>>>>
>>>>>
>>>>> W0816 23:17:01.984846 16360 sched.cpp:1195] Attempting to accept an
>>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910492
>>>>>
>>>>> W0816 23:17:01.984987 16360 sched.cpp:1195] Attempting to accept an
>>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910493
>>>>>
>>>>> W0816 23:17:01.985124 16360 sched.cpp:1195] Attempting to accept an
>>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910494
>>>>>
>>>>> W0816 23:17:01.985339 16360 sched.cpp:1195] Attempting to accept an
>>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910495
>>>>>
>>>>> W0816 23:17:01.985508 16360 sched.cpp:1195] Attempting to accept an
>>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910496
>>>>>
>>>>> W0816 23:17:01.985651 16360 sched.cpp:1195] Attempting to accept an
>>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910497
>>>>>
>>>>> W0816 23:17:01.985801 16360 sched.cpp:1195] Attempting to accept an
>>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910498
>>>>>
>>>>> W0816 23:17:01.985961 16360 sched.cpp:1195] Attempting to accept an
>>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910499
>>>>>
>>>>> W0816 23:17:01.986121 16360 sched.cpp:1195] Attempting to accept an
>>>>> unknown offer b859f2f3-7484-482d-8c0d-35bd91c1ad0a-O162910500
>>>>>
>>>>> 2016-08-16 23:18:41,877:16226(0x7f71271b6
>>>>> 700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 13ms
>>>>>
>>>>> 2016-08-16 23:21:12,007:16226(0x7f71271b6
>>>>> 700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 11ms
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
--
Michael Gummelt
Software Engineer
Mesosphere
