createInput creates a stream out of a file and can be used for HBase,
correct?
But I do not want to read from HBase. I only want to write to HBase. For
that I implemented an HBaseOutputFormat which I pass to the writeToFile
method of the dataStream. Then, the results of the stream processsing
are written into HBase.
Am 20.05.2015 um 14:26 schrieb Robert Metzger:
Hi,
great to hear that it is working.
If the PR is going to be only about adding the "write()" method, you
probably don't need to open the PR.
https://github.com/apache/flink/pull/521 is going to add a method called:
public<OUT>DataStreamSource<OUT>createInput(InputFormat<OUT,
?>inputFormat, TypeInformation<OUT>typeInfo)
That's exactly what you need, right?
The issue with that pull request is probably only that we have to wait
for another week until its merged.
On Wed, May 20, 2015 at 2:23 PM, Hilmi Yildirim
<hilmi.yildi...@neofonie.de <mailto:hilmi.yildi...@neofonie.de>> wrote:
Hi,
I've changed "writeToFile" to public and then I implemented an
Outputformat to write the stream into the HBase. This is working
very well. I will do later a pull request.
Maybe the method name "writeToFile" should be changed in, for
example, "write".
Alternatively, I can create a method writeToHBase in the
DataStream class.
Best Regards,
Hilmi
Am 20.05.2015 um 11:15 schrieb Robert Metzger:
There is this pending pull request which is addressing exactly
the issues I've mentioned (wrong naming, private method):
https://github.com/apache/flink/pull/521
I'll see whats blocking the PR ...
On Wed, May 20, 2015 at 11:11 AM, Robert Metzger
<rmetz...@apache.org <mailto:rmetz...@apache.org>> wrote:
Maybe we can also use the Batch HBase OutputFormat.
In the DataStream API there is a private method:
privateDataStreamSink<OUT>writeToFile(OutputFormat<OUT>
format,longmillis) {
which seems to allow batch output formats.
The naming of the method seems weird because its called
"toFile" but its expecting an OutputFormat instead of a
FileOutputFormat.
Maybe its worth trying to see how far we can get with this
method.
On Wed, May 20, 2015 at 11:00 AM, Robert Metzger
<rmetz...@apache.org <mailto:rmetz...@apache.org>> wrote:
Hi,
I agree with Hilmi, Flavio's examples are for batch.
I'm not aware of a StreamingHBaseSink for Flink yet.
I'll filed a JIRA for the feature request:
https://issues.apache.org/jira/browse/FLINK-2055
Are you interested in implementing this?
On Wed, May 20, 2015 at 10:50 AM, Hilmi Yildirim
<hilmi.yildi...@neofonie.de
<mailto:hilmi.yildi...@neofonie.de>> wrote:
Thank you Flavio,
these are examples for Batch Processing. But I want
to write a continuous stream into an HBase within a
StreamExecutionEnvironment instead of a
ExecutionEnvironment.
Best Regards,
Hilmi
Am 20.05.2015 um 10:42 schrieb Flavio Pompermaier:
I've added an example of HBase writing
at
flink-staging/flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseWriteExample.java.
Otherwise you can look at these 2 URLs:
*
https://github.com/fpompermaier/flink/blob/hbaseOutExample/flink-staging/flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseWriteExample.java
*
https://github.com/apache/flink/blob/57615aaa19e9933e43ed0431a78dd231bf98b103/flink-staging/flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseWriteExample.java
Best,
Flavio
On Wed, May 20, 2015 at 10:16 AM, Hilmi Yildirim
<hilmi.yildi...@neofonie.de
<mailto:hilmi.yildi...@neofonie.de>> wrote:
Hi,
I want to write a stream continuously into an
HBase. For example, I have 1 source and 4
workers. I want that each worker writes
autonomously into HBase. Is there a proper way
to do it?
Best Regards,
--
--
Hilmi Yildirim
Software Developer R&D
T: +49 30 24627-281 <tel:%2B49%2030%2024627-281>
hilmi.yildi...@neofonie.de
<mailto:hilmi.yildi...@neofonie.de>
http://www.neofonie.de
Besuchen Sie den Neo Tech Blog für Anwender:
http://blog.neofonie.de/
Folgen Sie uns:
https://plus.google.com/+neofonie
http://www.linkedin.com/company/neofonie-gmbh
https://www.xing.com/companies/neofoniegmbh
Neofonie GmbH | Robert-Koch-Platz 4 | 10115 Berlin
Handelsregister Berlin-Charlottenburg: HRB 67460
Geschäftsführung: Thomas Kitlitschko
--
--
Hilmi Yildirim
Software Developer R&D
T:+49 30 24627-281 <tel:%2B49%2030%2024627-281>
hilmi.yildi...@neofonie.de <mailto:hilmi.yildi...@neofonie.de>
http://www.neofonie.de
Besuchen Sie den Neo Tech Blog für Anwender:
http://blog.neofonie.de/
Folgen Sie uns:
https://plus.google.com/+neofonie
http://www.linkedin.com/company/neofonie-gmbh
https://www.xing.com/companies/neofoniegmbh
Neofonie GmbH | Robert-Koch-Platz 4 | 10115 Berlin
Handelsregister Berlin-Charlottenburg: HRB 67460
Geschäftsführung: Thomas Kitlitschko
--
--
Hilmi Yildirim
Software Developer R&D
T:+49 30 24627-281 <tel:%2B49%2030%2024627-281>
hilmi.yildi...@neofonie.de <mailto:hilmi.yildi...@neofonie.de>
http://www.neofonie.de
Besuchen Sie den Neo Tech Blog für Anwender:
http://blog.neofonie.de/
Folgen Sie uns:
https://plus.google.com/+neofonie
http://www.linkedin.com/company/neofonie-gmbh
https://www.xing.com/companies/neofoniegmbh
Neofonie GmbH | Robert-Koch-Platz 4 | 10115 Berlin
Handelsregister Berlin-Charlottenburg: HRB 67460
Geschäftsführung: Thomas Kitlitschko
--
--
Hilmi Yildirim
Software Developer R&D
T: +49 30 24627-281
hilmi.yildi...@neofonie.de
http://www.neofonie.de
Besuchen Sie den Neo Tech Blog für Anwender:
http://blog.neofonie.de/
Folgen Sie uns:
https://plus.google.com/+neofonie
http://www.linkedin.com/company/neofonie-gmbh
https://www.xing.com/companies/neofoniegmbh
Neofonie GmbH | Robert-Koch-Platz 4 | 10115 Berlin
Handelsregister Berlin-Charlottenburg: HRB 67460
Geschäftsführung: Thomas Kitlitschko
