Yes it does. :-) I have implemented it with Hadoop1 and Hadoop2.
Essentially I have extended the HadoopOutputFormat reusing part of the code
of the HadoopOutputFormatBase, and set the MongoOutputCommiter to replace
the FileOutputCommitter.
saluti,
Stefano
Stefano Bortoli, PhD
*ENS Technical Di
Does this make the MongoHadoopOutputFormat work for you?
On Thu, Jul 23, 2015 at 12:44 PM, Stefano Bortoli
wrote:
> https://issues.apache.org/jira/browse/FLINK-2394?filter=-2
>
> Meanwhile, I have implemented the MongoHadoopOutputFormat overriding open,
> close and globalFinalize methods.
>
> sa
https://issues.apache.org/jira/browse/FLINK-2394?filter=-2
Meanwhile, I have implemented the MongoHadoopOutputFormat overriding open,
close and globalFinalize methods.
saluti,
Stefano
2015-07-22 17:11 GMT+02:00 Stephan Ewen :
> Thank's for reporting this, Stefano!
>
> Seems like the HadoopOutpu
A simple solution would be to:
1 - create a constructor of HadoopOutputFormatBase and HadoopOutputFormat
that gets the OutputCommitter as a parameter
2 - change the outputCommitter field of HadoopOutputFormatBase to be a
generic OutputCommitter
3 - remove the default assignment in the open() and f
Thank's for reporting this, Stefano!
Seems like the HadoopOutputFormat wrapper is pretty much specialized on
File Output Formats.
Can you open an issue for that? Someone will need to look into this...
On Wed, Jul 22, 2015 at 4:48 PM, Stefano Bortoli
wrote:
> In fact, on close() of the HadoopOu
In fact, on close() of the HadoopOutputFormat the fileOutputCommitter
returns false on if
(this.fileOutputCommitter.needsTaskCommit(this.context)) returns false.
i/**
* commit the task by moving the output file out from the temporary
directory.
* @throws java.io.IOException
*/
Debugging, it seem the commitTask method of the MongoOutputCommitter is
never called. Is it possible that this 'bulk' approach of mongo-hadoop 1.4
does not fit the task execution method of Flink?
any idea? thanks a lot in advance.
saluti,
Stefano
Stefano Bortoli, PhD
*ENS Technical Director *__
Hi,
I am trying to analyze and update a MongoDB collection with Apache Flink
0.9.0 and Mongo Hadoop 1.4.0 Hadoop 2.6.0.
The process is fairly simple, and the MongoInputFormat works smoothly,
however it does not write back to the collection. The process works,
because the writeAsText works as expe
