Thanks, Well I think that will be my final option. Right now I am running the script twice and is ok for my performance requirement. When the amount of data increases I might have to write a custom store function.
Vishnu Viswanath > On 16-Jan-2015, at 13:51, Pradeep Gollakota <[email protected]> wrote: > > Actually, there is one more option. You copy the code of the LoadStoreFunc > and modify it to push the collection name from a config property into the > location URL. But this is more involved engineering wise than splitting it up > into two scripts. > > It's up to you. > >> On Thu, Jan 15, 2015 at 11:59 PM, Pradeep Gollakota <[email protected]> >> wrote: >> It looks like your only option then is to use two separate scripts. It's not >> ideal because you have twice the I/O, but it should work. >> >> P.S. make sure to guy reply all so the list is kept in the loop. >> >>> On Jan 15, 2015 11:41 PM, "Vishnu Viswanath" <[email protected]> >>> wrote: >>> Thanks Pradeep for the suggestion. >>> >>> I am using zookeeper to store into SOLR. So my location is the zookeeper >>> server. I followed this link for doing the same: >>> https://docs.lucidworks.com/plugins/servlet/mobile#content/view/24380610 >>> >>> Is there a better way of doing it if I am using zookeeper? >>> >>> Regards, >>> Vishnu Viswanath >>> >>> >>> > On 16-Jan-2015, at 12:34, Pradeep Gollakota <[email protected]> wrote: >>> > >>> > Just out of curiosity, why are you using SET to set the solr collection? >>> > I'm not sure if you're using an out of the box Load/Store Func, but if I >>> > were to design it, I would use the "location" of a Load/Store Func to >>> > specify which solr collection to write to. >>> > >>> > Is it possible for you to redesign this way? >>> > >>> > On Thu, Jan 15, 2015 at 9:41 PM, Vishnu Viswanath < >>> > [email protected]> wrote: >>> > >>> >> Thanks >>> >> >>> >> SET sets the SOLR collection name. When the STORE is invoked, the data >>> >> will be ingested into the collection name set before. >>> >> >>> >> So, the problem must be because the second set is overriding the >>> >> collection name and the STORE is failing. >>> >> >>> >> Is there any way to overcome this? Because most of the processing time is >>> >> taken in the load and I don't want to do it twice. >>> >> >>> >> Regards, >>> >> Vishnu Viswanath >>> >> >>> >>> On 16-Jan-2015, at 09:29, Cheolsoo Park <[email protected]> wrote: >>> >>> >>> >>> What does "SET" do for Solr? Pig pre-processes all the set commands in >>> >> the >>> >>> entire script before executing any query, and values are overwritten if >>> >> the >>> >>> same key is set more than once. In your example, you have two set >>> >> commands. >>> >>> If you're thinking that different values will be applied in each >>> >>> section, >>> >>> that's not the case. e) will overwrite a). >>> >>> >>> >>> >>> >>> On Thu, Jan 15, 2015 at 7:46 PM, Vishnu Viswanath < >>> >>> [email protected]> wrote: >>> >>> >>> >>>> Hi All, >>> >>>> >>> >>>> I am in indexing data into solr using pig script. >>> >>>> I have two such scripts, and I tried combining these two scripts into a >>> >>>> single one. >>> >>>> >>> >>>> i.e., i have script 1 that does >>> >>>> -------------------- >>> >>>> a)SET solr collection info for collection 1 >>> >>>> b)LOAD data >>> >>>> c)FILTER data for SOLR collection number 1 >>> >>>> d)STORE data to solr >>> >>>> >>> >>>> >>> >>>> and script 2 that does >>> >>>> ------------------- >>> >>>> a)SET solr collection info for collection 2 >>> >>>> b)LOAD data >>> >>>> c)FILTER data for SOLR collection number 2 >>> >>>> d)STORE data to solr >>> >>>> >>> >>>> >>> >>>> combined script looks something like >>> >>>> -------------- >>> >>>> a)SET solr collection info for collection 1 >>> >>>> b)LOAD data >>> >>>> c)FILTER data from (b) for SOLR collection number 1 >>> >>>> d)STORE data to solr >>> >>>> e)SET solr collection info for collection 2 >>> >>>> f)FILTER data from (b) for SOLR collection number 2 >>> >>>> g)STORE data to solr >>> >>>> >>> >>>> But the store function fails when I run the combined script where as it >>> >>>> runs fine if I run scripts 1 and 2 separately. >>> >>>> >>> >>>> Any idea? >>> >>>> >>> >>>> Regards, >>> >>>> Vishnu >>> > >>> > --001a11c13bfcdc3d7f050cbf93c1-- >
