Hello everyone, I am trying to get benefits of DataFrames (to perform all SQL BASED operations like 'Where Clause', Joining etc.) as mentioned in https://spark.apache.org/docs/1.5.1/api/java/org/apache/spark/sql/DataFrame.html.
I am using, Aerospike and Spark (1.4.1) Java Client in Spring Framework . My scenario as below: (My apparent guide is : http://www.programcreek.com/java-api-examples/index.php?source_dir=deep-examples-master/deep-aerospike/) 1. I have a DeepSparkContext. 2. Preparing DataFrame from Aerospike in this way: public DataFrame createDataFrame (String setName, String[] outputColumns, String userId ){ DataFrame frame = null; ExtractorConfig setCells = new ExtractorConfig<>(Cells.class); setCells.putValue(ExtractorConstants.HOST, configurationHandler.aerospikeHost()) .putValue(ExtractorConstants.PORT, configurationHandler.aerospikePort()) .putValue(ExtractorConstants.NAMESPACE, configurationHandler.getAerospikeNamespace()) .putValue(ExtractorConstants.SET, setName) .putValue(ExtractorConstants.INPUT_COLUMNS, outputColumns) setCells.setExtractorImplClass(AerospikeCellExtractor.class); try { frame= deepSparkContext.createJavaSchemaRDD(setCells); } catch (UnsupportedDataTypeException undt){ log.error(undt.getMessage()); } return frame; } 3. In another method, I am processing DataFrame in this way: public void getActivePush (String userId, Integer status){ String[] activePushColumns = new String[] {"clientId", "userId", "status"}; DataFrame activePushFrame = createDataFrame(configurationHandler.getAerospikePushActivationSet(), activePushColumns, userId); activePushFrame.registerTempTable("activePush"); DataFrame messagesFiltered = deepSparkContext.sql("SELECT * FROM activePush"); log.debug("first : " + messagesFiltered.first()); } The above method gives successfully the first element. 4. But, If I use : DataFrame messagesFiltered = deepSparkContext.sql("SELECT * FROM activePush WHERE userId = 'user1'"); It gives me the following exception: [2016-03-22 13:40:15.413] boot - 10493 ERROR [main] --- SpringApplication: Application startup failed org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'MyAnalyzer': Invocation of init method failed; nested exception is java.lang.NullPointerException at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:408) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1564) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:755) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:690) at org.springframework.boot.SpringApplication.run(SpringApplication.java:322) springframework.boot.SpringApplication.run(SpringApplication.java:970) at org.springframework.boot.SpringApplication.run(SpringApplication.java:959) at com.matchinguu.analytics.AnalyticalEngineApplication.main(AnalyticalEngineApplication.java:14) Caused by: java.lang.NullPointerException Please guide me that how to apply where clause conditions/filters in DataFrame? I have also a plan to create multiple DataFrames and to apply JOIN operation between them, it will be good if u give me some good examples about it. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Issue-wihle-applying-filters-conditions-in-DataFrame-in-Spark-tp26560.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
