Stream-static join : Refreshing subset of static data / Connection pooling

2020-11-26 Thread Geervan Hayatnagarkar
Hi We intend to do a stream-static join where kafka is a streaming source and RDBMS is a static source. e.g. User activity data is coming in as a stream from Kafka source and we need to pull User personal details from PostgreSQL. Because PostgreSQL is a static source, the entire "User-Personal-D

Re: Stream-static join : Refreshing subset of static data / Connection pooling

2020-11-26 Thread Geervan Hayatnagarkar
able as a DataFrame and persist it in the > memory. You may also need to repartition the DataFrame > > > On 11/26/2020 21:44,Geervan Hayatnagarkar > wrote: > > Hi > > We intend to do a stream-static join where kafka is a streaming source and > RDBMS is a static source. > &

Re: Stream-static join : Refreshing subset of static data / Connection pooling

2020-11-29 Thread Geervan Hayatnagarkar
he data to > hbase by flink/spark streaming。So you can join the data in hbase > directly. in consideration of the particularly big table, the scan > performance in hbase is much better than PostgreSql. > > > > > > *From: *German Schiavon > *Date: *Friday, November 27