Hi Dotan, Very sorry for the late reply.
>> Debuging it, I can see that the same jar from the local file system is being added multiple times to the PigContext skipJars member. >From a brief look, skipJars is updated by JobControlCompiler<https://github.com/apache/pig/blob/trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java#L1612>, and I can see it's called every time when a new MR job is compiled by getJob()<https://github.com/apache/pig/blob/trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java#L605>method. A simple fix might be to not add a jar if it is already present in the skipJars list. Do you mind filing a jira? Thanks, Cheolsoo On Wed, Mar 5, 2014 at 7:28 AM, Dotan Patrich <[email protected]> wrote: > Hi, > I've run a pig script the has REGISTER command in it using java code that > uses the PigServer.registerScript method. > On each execution of the pig script (using the same java process) the > registered jar is added to the classpath of the taskjvm.sh file. > This is duplicated until we get an error for having a classpath which is > too long... > Debuging it, I can see that the same jar from the local file system is > being added multiple times to the PigContext skipJars member. > > Does anyone know of an open issue regarding this? am I using the PigServer > wrong? > I couldn't find any open issue regarding this, so the current workaround > was to remove the register statement from the pig script and call > PigServer.registerJar() once when my java process starts. > > Thanks for the help, > Dotan >
