Hello,
I am getting following error sometimes in local mode
java.lang.Exception: java.lang.ClassCastException:
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSplit
incompatible with org.apache.hadoop.mapreduce.lib.input.FileSplit
at
org.apache.hadoop.mapred.LocalJobRunne
Hi
I launched the attached pig job on pig-12 with hadoop MRv1 with the
attached data, but the FILTER function causes the job to get stuck in an
infinite loop.
pig -p pPeriod=201312 -f test.pig
The thread in question seems to be stuck forever inside while loop of
runPipeline method.
stack trace:
This is the pig script -
%default previousPeriod $pPeriod
tWeek = LOAD '/tmp/test_data.txt' USING PigStorage ('|') AS (WEEK:int,
DESCRIPTION:chararray, END_DATE:chararray, PERIOD:int);
gTWeek = FOREACH tWeek GENERATE WEEK AS WEEK, PERIOD AS PERIOD;
*pWeek = FILTER gTWeek BY PERIOD == $previousP
I seem to be hitting this issue in pig-0.12 although it claims to be fixed
in pig-0.12
https://issues.apache.org/jira/browse/PIG-3395
Large filter expression makes Pig hang
Cheers,
Suhas.
On Thu, Mar 6, 2014 at 4:26 PM, Suhas Satish wrote:
> This is the pig script -
>
> %default previousPeriod
Hi Suhas,
No. The issue with PIG-3461 is that Pig hangs at the query compilation with
a big filter expression before the job is submitted.
In addition, the filter extractor was totally rewritten in 0.12.
https://issues.apache.org/jira/browse/PIG-3461
Where exactly is your job hanging? Backend or
Hi Cheolsoo,
This is where its hanging -
*pWeek = FILTER gTWeek BY PERIOD == $previousPeriod;*
org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/
PigGenericMapBase.java:
protected void *runPipeline*(PhysicalOperator leaf) throws IOException,
InterruptedException {
while(true){
So that's backend. It has nothing to do with the filter extractor. The
filter extractor is for predicate push down on the frontend.
The code that you're showing is the entry point where Pig mapper begins. So
it doesn't tell us much. The mapper is given a segment of physical plan
(pipeline), and th
The example that reproduces the issue along with data is attached in the
very first email on this thread
On Thursday, March 6, 2014, Cheolsoo Park wrote:
> So that's backend. It has nothing to do with the filter extractor. The
> filter extractor is for predicate push down on the frontend.
>
> Th
