HBase has its own Java API and Scala API: you can use what you like. Btw, which Spark-Hbase connector do you use? Cloudera or Hortonworks?
On Wed, Oct 11, 2017 at 3:01 PM, Amine CHERIFI < cherifimohamedam...@gmail.com> wrote: > it seems that the job block whene we call newAPIHadoopRDD to get data from > Hbase. it may be the issue !! > is there another api to load date from hbase ? > > > <https://mailtrack.io/> Sent with Mailtrack > <https://chrome.google.com/webstore/detail/mailtrack-for-gmail-inbox/ndnaehgpjlnokgebbaldlmgkapkpjkkb?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality> > <#m_758372304976785313_> > > 2017-10-11 14:45 GMT+02:00 Sebastian Piu <sebastian....@gmail.com>: > >> We do have this issue randomly too, so interested in hearing if someone >> was able to get to the bottom of it >> >> On Wed, 11 Oct 2017, 13:40 amine_901, <cherifimohamedam...@gmail.com> >> wrote: >> >>> We encounter a problem on a Spark job 1.6(on yarn) that never ends, whene >>> several jobs launched simultaneously. >>> We found that by launching the job spark in yarn-client mode we do not >>> have >>> this problem, unlike launching it in yarn-cluster mode. >>> it could be a trail to find the cause. >>> >>> we changed the code to add a sparkContext.stop () >>> Indeed, the SparkContext was created (val sparkContext = >>> createSparkContext) >>> but not stopped. this solution has allowed us to decrease the number of >>> jobs >>> that remains blocked but nevertheless we still have some jobs blocked. >>> >>> by analyzing the logs we have found this log that repeats without >>> stopping: >>> /17/09/29 11:04:37 DEBUG SparkEventPublisher: Enqueue >>> SparkListenerExecutorMetricsUpdate(1,WrappedArray()) >>> 17/09/29 11:04:41 DEBUG ApplicationMaster: Sending progress >>> 17/09/29 11:04:41 DEBUG ApplicationMaster: Number of pending allocations >>> is >>> 0. Sleeping for 5000. / >>> >>> Does someone have an idea about this issue ? >>> Thank you in advance >>> >>> >>> >>> -- >>> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>> >>> > > > -- > CHERIFI Mohamed Amine > Développeur Big data/Data scientist > 07 81 65 17 03 >
