I think the problem there is # of unique keys -- one winds up creating way too many filehandles all at the same time. I may be misunderstanding the nature of the bug. If I do understand it correctly, it's endemic to the whole concept of MultiStorage; creating 7K files * # reducers sounds like a really bad thing to do; if you are running into the problem, you probably shouldn't be using MultiStorage.
Or am I misreading what's happening? D On Thu, Mar 31, 2011 at 9:12 AM, Jonathan Holloway < [email protected]> wrote: > Hi all, > > I'm working with some data at the moment, for which I needed to generate > multiple reports for a given grouped set of data by name. > I wasn't initially sure about how to do this, I came across MultiStorage in > Pig contrib, but a little worried about the 7k limit there at > the moment due to a bug: > > https://issues.apache.org/jira/browse/PIG-1547 > > Does anybody know what the issue here is - I can take a look at this if > necessary and someone can point me in the right way in > terms of fixing it? I've currently hacked MultiStorage to take a bag and > the contained tuples and spit out the tuples with a tab > delimiter between them. Is this the best way to go? > > Just looking for some feedback. > > Cheers, > Jon. >
