Re: Question about Utilities#getInputPaths

2016-10-05 Thread Gopal Vijayaraghavan
(always helpful to call out a version, I'm going to assume 1.2) >select * from (select count(1) from T union all select count(1) from T2) x; >I have to admit that I don't quite understand that. Would it mean that we'd > only get a single row if we left out this empty path? AFAIK, this

Question about Utilities#getInputPaths

2016-10-05 Thread Lars Francke
Hi everyone, I've encountered a performance issue at multiple customers now. The problem is the processing of input paths when there are lots of partitions. We check each directory if it's empty. This alone can take minutes. There is a comment in Utilities: "We need to add a empty file, it is n