OK, thanks for the pointer.
If I wind up rolling our own code to handle this I'll make sure to
contribute it.
DR
Aaron Kimball wrote:
There is no technical limit that prevents Hadoop from operating in this
fashion; it's simply the case that the included InputFormat implementations
do not do so. This behavior has been set in this fashion for a long time, so
it's unlikely that it will change soon, as that might break existing
applications.
But you can write your own subclass of TextInputFormat or
SequenceFileInputFormat that overrides the getSplits() method to recursively
descend through directories and search for files.
- Aaron
On Tue, Jun 2, 2009 at 1:22 PM, David Rosenstrauch <[email protected]>wrote:
As per a previous list question (
http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%[email protected]%3e)
it looks as though it's not possible for hadoop to traverse input
directories recursively in order to discover input files.
