spark hbase

Hi, I have tried multiple ways to use hbase-spark and none of them works as expected. SHC and hbase-spark library are loading all the data on executors and it is running for ever. https://ramottamado.dev/how-to-use-hbase-fuzzyrowfilter-in-spark/ Above link has the solution that I am looking for

Spark Hbase Hive error in EMR

Hi, I am trying to connect hbase which sits on top of hive as external table. I am getting below exception. Am I missing anything to pass here? 21/04/09 18:08:11 INFO ZooKeeper: Client environment:user.dir=/ 21/04/09 18:08:11 INFO ZooKeeper: Initiating client connection, connectString=localhost:

spark hbase connector

Hi, My spark jobs need to talk to hbase and I am not sure which spark hbase connector is recommended: https://hortonworks.com/blog/spark-hbase-dataframe-based-hbase-connector/ https://phoenix.apache.org/phoenix_spark.html Or there is any other better solutions. Appreciate any guidance.

Hortonworks Spark-Hbase-Connector does not read zookeeper configurations from spark session config ??(Spark on Yarn)

Hi I am trying to write a spark code that reads data from Hbase and store it in DataFrame. I am able to run it perfectly with hbase-site.xml in $spark-home/conf folder. But I am facing few issues Here. Issue 1: Passing hbase-site.xml location with --file parameter submitted through client mode(It

Financial fraud detection using streaming RDBMS data into Spark & Hbase

I am not talking about Credit Card fraud etc. In the complex fraud cases like that one in UBS , the rogue trader over a period of time manipulated the figures. Although there is a lot of talk about using elaborate set-ups to predict unu

Issue with Spark HBase connector streamBulkGet method

ark-user-list.1001560.n3.nabble.com/Issue-with-Spark-HBase-connector-streamBulkGet-method-tp27613.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Spark HBase bulk load using hfile format

Hi i am doing bulk load into HBase as HFileFormat, by using saveAsNewAPIHadoopFile when i try to write i am getting an exception java.io.IOException: Added a key not lexically larger than previous. following is the code snippet case class HBaseRow(rowKey: ImmutableBytesWritable, kv: KeyValue)

A problem involving Spark & HBase.

1560.n3.nabble.com/A-problem-involving-Spark-HBase-tp26602.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail:

Re: Generalised Spark-HBase integration

Cool, will revisit, is your latest code visible publicly somewhere ? On 28 July 2015 at 17:14, Ted Malaska wrote: > Yup you should be able to do that with the APIs that are going into HBase. > > Let me know if you need to chat about the problem and how to implement it > with the HBase apis. > >

Re: Generalised Spark-HBase integration

rget/classes > at 1438099569598 > [ERROR] > /home/hbase/s-on-hbase/src/main/scala/org/apache/spark/hbase/examples/simple/HBaseTableSimple.scala:36: > error: type mismatch; > [INFO] found : Int > [INFO] required: Short > [INFO] while (

Re: Generalised Spark-HBase integration

Hi Ted, yes, cloudera blog and your code was my starting point - but I needed something more spark-centric rather than on hbase. Basically doing a lot of ad-hoc transformations with RDDs that were based on HBase tables and then mutating them after series of iterative (bsp-like) steps. On 28 July 2

Re: Generalised Spark-HBase integration

I got a compilation error: [INFO] /home/hbase/s-on-hbase/src/main/scala:-1: info: compiling [INFO] Compiling 18 source files to /home/hbase/s-on-hbase/target/classes at 1438099569598 [ERROR] /home/hbase/s-on-hbase/src/main/scala/org/apache/spark/hbase/examples/simple/HBaseTableSimple.scala:36

Generalised Spark-HBase integration

Hi all, last couple of months I've been working on a large graph analytics and along the way have written from scratch a HBase-Spark integration as none of the ones out there worked either in terms of scale or in the way they integrated with the RDD interface. This week I have generalised it into a

Re: Spark + HBase + Kerberos

: Eric Walk Cc: user@spark.apache.org;Bill Busch Subject: Re: Spark + HBase + Kerberos Are hbase config / keytab files deployed on executor machines ? Consider adding -Dsun.security.krb5.debug=true for debug purpose. Cheers On Wed, Mar 18, 2015 at 11:39 AM, Eric Walk mailto:eric.w...@perficie

Re: Spark + HBase + Kerberos

Are hbase config / keytab files deployed on executor machines ? Consider adding -Dsun.security.krb5.debug=true for debug purpose. Cheers On Wed, Mar 18, 2015 at 11:39 AM, Eric Walk wrote: > Having an issue connecting to HBase from a Spark container in a Secure > Cluster. Haven’t been able to

Re: Spark Hbase

/examples/pythonconverters/HBaseConverters.scala > > Cheers > > On Wed, Sep 24, 2014 at 9:39 AM, Madabhattula Rajesh Kumar < > mrajaf...@gmail.com> wrote: > >> Hi Team, >> >> Could you please point me the example program for Spark HBase to read >> columns and values >> >> Regards, >> Rajesh >> > >

Re: Spark Hbase

Cheers On Wed, Sep 24, 2014 at 9:39 AM, Madabhattula Rajesh Kumar < mrajaf...@gmail.com> wrote: > Hi Team, > > Could you please point me the example program for Spark HBase to read > columns and values > > Regards, > Rajesh >

Spark Hbase

Hi Team, Could you please point me the example program for Spark HBase to read columns and values Regards, Rajesh

Re: Spark Hbase job taking long time

ime(Bytes.toLong(v1._2.getValue( >>> >>> Bytes.toBytes("si"), Bytes.toBytes("at"; >>> >>> return person; >>> } >>>

Re: Spark Hbase job taking long time

t; JavaSchemaRDD schemaPeople = sqlCtx.applySchema(people, Person.class); >> schemaPeople.registerAsTable("people"); >> >> // SQL can be run over RDDs that have been registered as >> tables. >> JavaSchemaRDD t

Spark Hbase job taking long time

ark using start-all.sh script with 2 workers Any pointers will be of a great help Regards, -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Hbase-job-taking-long-time-tp11541.html Sent from the Apache Spark User List mailing list archive at Nabble.com. -

Re: Need help on spark Hbase

Hi Rajesh, I saw : Warning: Local jar /home/rajesh/hbase-0.96.1.1-hadoop2/lib/hbase -client-0.96.1.1-hadoop2.jar, does not exist, skipping. in your log. I believe this jar contains the HBaseConfiguration. I'm not sure what went wrong in your case but can you try without spaces in --jars i.e. --j

Re: Need help on spark Hbase

Hi Team, Now i've changed my code and reading configuration from hbase-site.xml file(this file is in classpath). When i run this program using : mvn exec:java -Dexec.mainClass="com.cisco.ana.accessavailability.AccessAvailability". It is working fine. But when i run this program from spark-submit i

Re: Need help on spark Hbase

Good catch. I thought the largest port number is 65535. Cheers On Tue, Jul 15, 2014 at 4:33 PM, Spark DevUser wrote: > Are you able to launch *hbase shell* and run some commands (list, > describe, scan, etc)? Seems *configuration.set("hbase.**master", > "localhost:60")* is wrong. > > > On

Re: Need help on spark Hbase

Are you able to launch *hbase shell* and run some commands (list, describe, scan, etc)? Seems *configuration.set("hbase.**master", "localhost:60")* is wrong. On Tue, Jul 15, 2014 at 3:00 PM, Tathagata Das wrote: > Also, it helps if you post us logs, stacktraces, exceptions, etc. > > TD > >

Re: Need help on spark Hbase

Also, it helps if you post us logs, stacktraces, exceptions, etc. TD On Tue, Jul 15, 2014 at 10:07 AM, Jerry Lam wrote: > Hi Rajesh, > > I have a feeling that this is not directly related to spark but I might be > wrong. The reason why is that when you do: > >Configuration configuration =

Re: Need help on spark Hbase

Hi Rajesh, I have a feeling that this is not directly related to spark but I might be wrong. The reason why is that when you do: Configuration configuration = HBaseConfiguration.create(); by default, it reads the configuration files hbase-site.xml in your classpath and ... (I don't remember a

Re: Need help on spark Hbase

One vector to check is the HBase libraries in the --jars as in : spark-submit --class --master --jars hbase-client-0.98.3-hadoop2.jar,commons-csv-1.0-SNAPSHOT.jar,hbase-common-0.98.3-hadoop2.jar,hbase-hadoop2-compat-0.98.3-hadoop2.jar,hbase-it-0.98.3-hadoop2.jar,hbase-protocol-0.98.3-hadoop2.jar,

Re: Need help on spark Hbase

Hi Nathan and Jerry, Thank you for the details. Jerry, I've installed Spark, Hbase and Hadoop in same machine. Please let me know do you need more information. I'm not able to identify the issue why it is not connected to Hbase when i use spark-submit Do you have a example progr

Re: Need help on spark Hbase

Hi Rajesh, can you describe your spark cluster setup? I saw localhost:2181 for zookeeper. Best Regards, Jerry On Tue, Jul 15, 2014 at 9:47 AM, Madabhattula Rajesh Kumar < mrajaf...@gmail.com> wrote: > Hi Team, > > Could you please help me to resolve the issue. > > *Issue *: I'm not able to co

Need help on spark Hbase

Hi Team, Could you please help me to resolve the issue. *Issue *: I'm not able to connect HBase from Spark-submit. Below is my code. When i execute below program in standalone, i'm able to connect to Hbase and doing the operation. When i execute below program using spark submit ( ./bin/spark-su