Thanks for all your directions. BucketingSink worked.
Isuru
On Thu, Oct 12, 2017 at 9:05 AM, Piotr Nowojski
wrote:
> I think the issue might be that writeAsText (TextOutputFormat) doesn’t
> flush the data anywhere (only on close, which in streaming doesn’t happen).
> You would need to use custo
I think the issue might be that writeAsText (TextOutputFormat) doesn’t flush
the data anywhere (only on close, which in streaming doesn’t happen). You would
need to use custom output format, but as Aljoscha pointed out BucketingSink
makes more sense for streaming applications.
Piotrek
> On 12
Hi,
Maybe this is an access rights issue? Could you try to create and write to same
file (same directory) in some other way (manually?), using the same user and
the same machine as would Flink job do?
Maybe there will be some hint in hdfs logs?
Piotrek
> On 12 Oct 2017, at 00:19, Isuru Suriar
Hi all,
I'm just trying to use an HDFS file as the sink for my flink stream job. I
use the following line to do so.
stream.writeAsText("hdfs://hadoop-master:9000/user/isuru/foo");
I have not set "fs.hdfs.hadoopconf" in my flink configuration as it should
work with the full hdfs file name accord
