I have a table of size 5GB, and want to load selective rows into dataframe
instead of loading the entire table in memory,
For me memory is a constraint hence , and i would like to peridically load
few set of rows and perform dataframe operations on it,
,
for the "dbtable" is there a way to perform select * from master_schema
where 'TID' = '100_0';
which can load only this to memory as dataframe .
Currently I'm using code as below
val df = sqlContext.read .format("jdbc")
.option("url", url)
.option("dbtable", "master_schema").load()
Thansk,
Sujeet