Hi Cheolsoo, Thank you very much for the reply and interest in this. I actually did opened a jira issue for this a few weeks ago - PIG-3798<https://issues.apache.org/jira/browse/PIG-3798>
Thanks, Dotan On Wed, Mar 26, 2014 at 8:50 PM, Cheolsoo Park <[email protected]> wrote: > Hi Dotan, > > Very sorry for the late reply. > > >> Debuging it, I can see that the same jar from the local file system is > being > added multiple times to the PigContext skipJars member. > > From a brief look, skipJars is updated by > JobControlCompiler< > https://github.com/apache/pig/blob/trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java#L1612 > >, > and I can see it's called every time when a new MR job is compiled by > getJob()< > https://github.com/apache/pig/blob/trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java#L605 > >method. > A > simple fix might be to not add a jar if it is already present in the > skipJars list. Do you mind filing a jira? > > Thanks, > Cheolsoo > > > > > On Wed, Mar 5, 2014 at 7:28 AM, Dotan Patrich <[email protected]> > wrote: > > > Hi, > > I've run a pig script the has REGISTER command in it using java code that > > uses the PigServer.registerScript method. > > On each execution of the pig script (using the same java process) the > > registered jar is added to the classpath of the taskjvm.sh file. > > This is duplicated until we get an error for having a classpath which is > > too long... > > Debuging it, I can see that the same jar from the local file system is > > being added multiple times to the PigContext skipJars member. > > > > Does anyone know of an open issue regarding this? am I using the > PigServer > > wrong? > > I couldn't find any open issue regarding this, so the current workaround > > was to remove the register statement from the pig script and call > > PigServer.registerJar() once when my java process starts. > > > > Thanks for the help, > > Dotan > > >
