Also I wanted to run get() and set() from mapPartitions (from spark workers and not master)...
To be able to do that I think I have to create a separate spark context for the cache... But I am not sure how SparkContext from job1 can access SparkContext from job2 ! On Tue, Feb 10, 2015 at 3:25 PM, Debasish Das <[email protected]> wrote: > Thanks...this is what I was looking for... > > It will be great if Ankur can give brief details about it...Basically how > does it contrast with memcached for example... > > On Tue, Feb 10, 2015 at 2:32 PM, Michael Armbrust <[email protected]> > wrote: > >> You should look at https://github.com/amplab/spark-indexedrdd >> >> On Tue, Feb 10, 2015 at 2:27 PM, Debasish Das <[email protected]> >> wrote: >> >>> Hi Michael, >>> >>> I want to cache a RDD and define get() and set() operators on it. >>> Basically like memcached. Is it possible to build a memcached like >>> distributed cache using Spark SQL ? If not what do you suggest we should >>> use for such operations... >>> >>> Thanks. >>> Deb >>> >>> On Fri, Jul 18, 2014 at 1:00 PM, Michael Armbrust < >>> [email protected]> wrote: >>> >>>> You can do insert into. As with other SQL on HDFS systems there is no >>>> updating of data. >>>> On Jul 17, 2014 1:26 AM, "Akhil Das" <[email protected]> >>>> wrote: >>>> >>>>> Is this what you are looking for? >>>>> >>>>> >>>>> https://spark.apache.org/docs/1.0.0/api/java/org/apache/spark/sql/parquet/InsertIntoParquetTable.html >>>>> >>>>> According to the doc, it says "Operator that acts as a sink for >>>>> queries on RDDs and can be used to store the output inside a directory of >>>>> Parquet files. This operator is similar to Hive's INSERT INTO TABLE >>>>> operation in the sense that one can choose to either overwrite or append >>>>> to >>>>> a directory. Note that consecutive insertions to the same table must have >>>>> compatible (source) schemas." >>>>> >>>>> Thanks >>>>> Best Regards >>>>> >>>>> >>>>> On Thu, Jul 17, 2014 at 11:42 AM, Hu, Leo <[email protected]> wrote: >>>>> >>>>>> Hi >>>>>> >>>>>> As for spark 1.0, can we insert and update a table with SPARK SQL, >>>>>> and how? >>>>>> >>>>>> >>>>>> >>>>>> Thanks >>>>>> >>>>>> Best Regard >>>>>> >>>>> >>>>> >>> >> >
