Looking at my previous mail which mentions changes to API, optimizer, and
runtime code of the DataSet API this would be a major and non-trivial
effort and also require that a committer spends a good amount of time for
this.


2018-01-16 10:07 GMT+01:00 Flavio Pompermaier <pomperma...@okkam.it>:

> Do you think is that complex to support it? I think we can try to
> implement it if someone could give us some support (at least some big
> picture)
>
> On Tue, Jan 16, 2018 at 10:02 AM, Fabian Hueske <fhue...@gmail.com> wrote:
>
>> No, I'm not aware of anybody working on extending the Hadoop
>> compatibility support.
>> I'll also have no time to work on this any time soon :-(
>>
>> 2018-01-13 1:34 GMT+01:00 Flavio Pompermaier <pomperma...@okkam.it>:
>>
>>> Any progress on this Fabian? HBase bulk loading is a common task for us
>>> and it's very annoying and uncomfortable to run a separate YARN job to
>>> accomplish it...
>>>
>>> On 10 Apr 2015 12:26, "Flavio Pompermaier" <pomperma...@okkam.it> wrote:
>>>
>>> Great! That will be awesome.
>>> Thank you Fabian
>>>
>>> On Fri, Apr 10, 2015 at 12:14 PM, Fabian Hueske <fhue...@gmail.com>
>>> wrote:
>>>
>>>> Hmm, that's a tricky question ;-) I would need to have a closer look.
>>>> But getting custom comparators for sorting and grouping into the Combiner
>>>> is not that trivial because it touches API, Optimizer, and Runtime code.
>>>> However, I did that before for the Reducer and with the recent addition of
>>>> groupCombine the Reducer changes might be just applied to combine.
>>>>
>>>> I'll be gone next week, but if you want to, we can have a closer look
>>>> at the problem after that.
>>>>
>>>> 2015-04-10 12:07 GMT+02:00 Flavio Pompermaier <pomperma...@okkam.it>:
>>>>
>>>>> I think I could also take care of it if somebody can help me and guide
>>>>> me a little bit..
>>>>> How long do you think it will require to complete such a task?
>>>>>
>>>>> On Fri, Apr 10, 2015 at 12:02 PM, Fabian Hueske <fhue...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> We had an effort to execute any HadoopMR program by simply specifying
>>>>>> the JobConf and execute it (even embedded in regular Flink programs).
>>>>>> We got quite far but did not complete (counters and custom grouping /
>>>>>> sorting functions for Combiners are missing if I remember correctly).
>>>>>> I don't think that anybody is working on that right now, but it would
>>>>>> definitely be a cool feature.
>>>>>>
>>>>>> 2015-04-10 11:55 GMT+02:00 Flavio Pompermaier <pomperma...@okkam.it>:
>>>>>>
>>>>>>> Hi guys,
>>>>>>>
>>>>>>> I have a nice question about Hadoop compatibility.
>>>>>>> In https://flink.apache.org/news/2014/11/18/hadoop-compatibilit
>>>>>>> y.html you say that you can reuse existing mapreduce programs.
>>>>>>> Could it be possible to manage also complex mapreduce programs like
>>>>>>> HBase BulkImport that use for example a custom partioner
>>>>>>> (org.apache.hadoop.mapreduce.Partitioner)?
>>>>>>>
>>>>>>> In the bulk-import examples the call 
>>>>>>> HFileOutputFormat2.configureIncrementalLoadMap
>>>>>>> that sets a series of job parameters (like partitioner, mapper, 
>>>>>>> reducers,
>>>>>>> etc) -> http://pastebin.com/8VXjYAEf.
>>>>>>> The full code of it can be seen at https://github.com/apache/h
>>>>>>> base/blob/master/hbase-server/src/main/java/org/apache/hadoo
>>>>>>> p/hbase/mapreduce/HFileOutputFormat2.java.
>>>>>>>
>>>>>>> Do you think there's any change to make it run in flink?
>>>>>>>
>>>>>>> Best,
>>>>>>> Flavio
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>
> --
> Flavio Pompermaier
> Development Department
>
> OKKAM S.r.l.
> Tel. +(39) 0461 041809 <+39%200461%20041809>
>

Reply via email to