Actually, there is one more option. You copy the code of the LoadStoreFunc and modify it to push the collection name from a config property into the location URL. But this is more involved engineering wise than splitting it up into two scripts.
It's up to you. On Thu, Jan 15, 2015 at 11:59 PM, Pradeep Gollakota <[email protected]> wrote: > It looks like your only option then is to use two separate scripts. It's > not ideal because you have twice the I/O, but it should work. > > P.S. make sure to guy reply all so the list is kept in the loop. > On Jan 15, 2015 11:41 PM, "Vishnu Viswanath" <[email protected]> > wrote: > >> Thanks Pradeep for the suggestion. >> >> I am using zookeeper to store into SOLR. So my location is the zookeeper >> server. I followed this link for doing the same: >> https://docs.lucidworks.com/plugins/servlet/mobile#content/view/24380610 >> >> Is there a better way of doing it if I am using zookeeper? >> >> Regards, >> Vishnu Viswanath >> >> >> > On 16-Jan-2015, at 12:34, Pradeep Gollakota <[email protected]> >> wrote: >> > >> > Just out of curiosity, why are you using SET to set the solr collection? >> > I'm not sure if you're using an out of the box Load/Store Func, but if I >> > were to design it, I would use the "location" of a Load/Store Func to >> > specify which solr collection to write to. >> > >> > Is it possible for you to redesign this way? >> > >> > On Thu, Jan 15, 2015 at 9:41 PM, Vishnu Viswanath < >> > [email protected]> wrote: >> > >> >> Thanks >> >> >> >> SET sets the SOLR collection name. When the STORE is invoked, the data >> >> will be ingested into the collection name set before. >> >> >> >> So, the problem must be because the second set is overriding the >> >> collection name and the STORE is failing. >> >> >> >> Is there any way to overcome this? Because most of the processing time >> is >> >> taken in the load and I don't want to do it twice. >> >> >> >> Regards, >> >> Vishnu Viswanath >> >> >> >>> On 16-Jan-2015, at 09:29, Cheolsoo Park <[email protected]> wrote: >> >>> >> >>> What does "SET" do for Solr? Pig pre-processes all the set commands in >> >> the >> >>> entire script before executing any query, and values are overwritten >> if >> >> the >> >>> same key is set more than once. In your example, you have two set >> >> commands. >> >>> If you're thinking that different values will be applied in each >> section, >> >>> that's not the case. e) will overwrite a). >> >>> >> >>> >> >>> On Thu, Jan 15, 2015 at 7:46 PM, Vishnu Viswanath < >> >>> [email protected]> wrote: >> >>> >> >>>> Hi All, >> >>>> >> >>>> I am in indexing data into solr using pig script. >> >>>> I have two such scripts, and I tried combining these two scripts >> into a >> >>>> single one. >> >>>> >> >>>> i.e., i have script 1 that does >> >>>> -------------------- >> >>>> a)SET solr collection info for collection 1 >> >>>> b)LOAD data >> >>>> c)FILTER data for SOLR collection number 1 >> >>>> d)STORE data to solr >> >>>> >> >>>> >> >>>> and script 2 that does >> >>>> ------------------- >> >>>> a)SET solr collection info for collection 2 >> >>>> b)LOAD data >> >>>> c)FILTER data for SOLR collection number 2 >> >>>> d)STORE data to solr >> >>>> >> >>>> >> >>>> combined script looks something like >> >>>> -------------- >> >>>> a)SET solr collection info for collection 1 >> >>>> b)LOAD data >> >>>> c)FILTER data from (b) for SOLR collection number 1 >> >>>> d)STORE data to solr >> >>>> e)SET solr collection info for collection 2 >> >>>> f)FILTER data from (b) for SOLR collection number 2 >> >>>> g)STORE data to solr >> >>>> >> >>>> But the store function fails when I run the combined script where as >> it >> >>>> runs fine if I run scripts 1 and 2 separately. >> >>>> >> >>>> Any idea? >> >>>> >> >>>> Regards, >> >>>> Vishnu >> > >> > --001a11c13bfcdc3d7f050cbf93c1-- >> >
