NullPointerException when scanning HBase table

2018-04-30 Thread Huiliang Zhang
Hi, In my spark job, I need to scan HBase table. I set up a scan with custom filters. Then I use newAPIHadoopRDD function to get a JavaPairRDD variable X. The problem is when no records inside HBase matches my filters, the call X.isEmpty() or X.count() will cause a java.lang.NullPointerException

Resource manage inside map function

2018-03-30 Thread Huiliang Zhang
Hi, I have a spark job which needs to access HBase inside a mapToPair function. The question is that I do not want to connect to HBase and close connection each time. As I understand, PairFunction is not designed to manage resources with setup() and close(), like Hadoop reader and writer. Does s