Re: Write Stream to HBase

Hilmi Yildirim Wed, 20 May 2015 05:23:59 -0700

Hi,

I've changed "writeToFile" to public and then I implemented anOutputformat to write the stream into the HBase. This is working verywell. I will do later a pull request.

Maybe the method name "writeToFile" should be changed in, for example,"write".


Alternatively, I can create a method writeToHBase in the DataStream class.

Best Regards,
Hilmi

Am 20.05.2015 um 11:15 schrieb Robert Metzger:

There is this pending pull request which is addressing exactly theissues I've mentioned (wrong naming, private method):https://github.com/apache/flink/pull/521


I'll see whats blocking the PR ...

On Wed, May 20, 2015 at 11:11 AM, Robert Metzger <rmetz...@apache.org<mailto:rmetz...@apache.org>> wrote:


    Maybe we can also use the Batch HBase OutputFormat.

    In the DataStream API there is a private method:

    privateDataStreamSink<OUT>writeToFile(OutputFormat<OUT> format,longmillis) {

    which seems to allow batch output formats.
    The naming of the method seems weird because its called "toFile"
    but its expecting an OutputFormat instead of a FileOutputFormat.

    Maybe its worth trying to see how far we can get with this method.


    On Wed, May 20, 2015 at 11:00 AM, Robert Metzger
    <rmetz...@apache.org <mailto:rmetz...@apache.org>> wrote:

        Hi,
        I agree with Hilmi, Flavio's examples are for batch.

        I'm not aware of a StreamingHBaseSink for Flink yet.
        I'll filed a JIRA for the feature request:
        https://issues.apache.org/jira/browse/FLINK-2055

        Are you interested in implementing this?

        On Wed, May 20, 2015 at 10:50 AM, Hilmi Yildirim
        <hilmi.yildi...@neofonie.de
        <mailto:hilmi.yildi...@neofonie.de>> wrote:

            Thank you Flavio,
            these are examples for Batch Processing. But I want to
            write a continuous stream into an HBase within a
            StreamExecutionEnvironment instead of a ExecutionEnvironment.

            Best Regards,
            Hilmi


            Am 20.05.2015 um 10:42 schrieb Flavio Pompermaier:

            I've added an example of HBase writing
            at 
flink-staging/flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseWriteExample.java.

            Otherwise you can look at these 2 URLs:

              * 
https://github.com/fpompermaier/flink/blob/hbaseOutExample/flink-staging/flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseWriteExample.java
              * 
https://github.com/apache/flink/blob/57615aaa19e9933e43ed0431a78dd231bf98b103/flink-staging/flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseWriteExample.java

            Best,
            Flavio

            On Wed, May 20, 2015 at 10:16 AM, Hilmi Yildirim
            <hilmi.yildi...@neofonie.de
            <mailto:hilmi.yildi...@neofonie.de>> wrote:

                Hi,
                I want to write a stream continuously into an HBase.
                For example, I have 1 source and 4 workers. I want
                that each worker writes autonomously into HBase. Is
                there a proper way to do it?

                Best Regards,

----

                Hilmi Yildirim
                Software Developer R&D

                T: +49 30 24627-281 <tel:%2B49%2030%2024627-281>
                hilmi.yildi...@neofonie.de
                <mailto:hilmi.yildi...@neofonie.de>

                http://www.neofonie.de

                Besuchen Sie den Neo Tech Blog für Anwender:
                http://blog.neofonie.de/

                Folgen Sie uns:
                https://plus.google.com/+neofonie
                http://www.linkedin.com/company/neofonie-gmbh
                https://www.xing.com/companies/neofoniegmbh

                Neofonie GmbH | Robert-Koch-Platz 4 | 10115 Berlin
                Handelsregister Berlin-Charlottenburg: HRB 67460
                Geschäftsführung: Thomas Kitlitschko

----

            Hilmi Yildirim
            Software Developer R&D

            T:+49 30 24627-281  <tel:%2B49%2030%2024627-281>
            hilmi.yildi...@neofonie.de  <mailto:hilmi.yildi...@neofonie.de>

            http://www.neofonie.de

            Besuchen Sie den Neo Tech Blog für Anwender:
            http://blog.neofonie.de/

            Folgen Sie uns:
            https://plus.google.com/+neofonie
            http://www.linkedin.com/company/neofonie-gmbh
            https://www.xing.com/companies/neofoniegmbh

            Neofonie GmbH | Robert-Koch-Platz 4 | 10115 Berlin
            Handelsregister Berlin-Charlottenburg: HRB 67460
            Geschäftsführung: Thomas Kitlitschko


--
--
Hilmi Yildirim
Software Developer R&D

T: +49 30 24627-281
hilmi.yildi...@neofonie.de

http://www.neofonie.de

Besuchen Sie den Neo Tech Blog für Anwender:
http://blog.neofonie.de/

Folgen Sie uns:
https://plus.google.com/+neofonie
http://www.linkedin.com/company/neofonie-gmbh
https://www.xing.com/companies/neofoniegmbh

Neofonie GmbH | Robert-Koch-Platz 4 | 10115 Berlin
Handelsregister Berlin-Charlottenburg: HRB 67460
Geschäftsführung: Thomas Kitlitschko

Re: Write Stream to HBase

Reply via email to