if you are doing a lot of small updates on a huge amount of data & need to get real time response on the output spark is probably not a good fit. If you are dong small updates on your rdd but need to materialize the final rdd with all the changes every 1 day or so then probably spark can fit with some data modelling.
Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi <https://twitter.com/mayur_rustagi> On Thu, May 1, 2014 at 6:31 PM, NN Murthy <nn.mur...@cmcltd.com> wrote: > Thanks a lot for very prompt response. Then next questions are the > following. > > 1. Can we conclude that Spark is NOT the solution for our > requirement? Or > > 2. Is there a design approach to meet such requirements using > Spark? > > > > *From:* Mayur Rustagi [mailto:mayur.rust...@gmail.com] > *Sent:* 01 May 2014 18:22 > *To:* user@spark.apache.org > *Cc:* u...@spark.incubator.apache.org > *Subject:* Re: update of RDDs > > > > RDD are immutable so cannot be updated. You can create new RDD containing > updated entries(often not what you want to do). > > > > > > > Mayur Rustagi > Ph: +1 (760) 203 3257 > > http://www.sigmoidanalytics.com > > @mayur_rustagi <https://twitter.com/mayur_rustagi> > > > > > > On Thu, May 1, 2014 at 4:42 AM, narayanabhatla NarasimhaMurthy < > nn.mur...@cmcltd.com> wrote: > > In our application, we need distributed RDDs containing key-value maps. We > have operations that update RDDs by way of adding entries to the map, > delete > entries from the map as well as update value part of maps. > We also have map reduce functions that operate on the RDDs.The questions > are > the following. > 1. Can RDDs be updated? if Yes, what rae the methods? > 2. If we update RDDs, will it happen in place or does it create new RDDs > with almost double the original RDD size (original+newly created RDD)? > Thank you very much. > N.N.Murthy > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/update-of-RDDs-tp5132.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > DISCLAIMER > The information contained in this e-mail message and/or attachments to it > may contain confidential or privileged information. If you are not the > intended recipient, any dissemination, use, review, distribution, printing > or copying of the information contained in this e-mail message and/or > attachments to it are strictly prohibited. If you have received this > communication in error, please notify us by reply e-mail or directly to > netsupp...@cmcltd.com or telephone and immediately and permanently delete > the message and any attachments. Thank you. > This email has been scrubbed for your protection by SecureMX. For more > information visit securemx.in >
