Cool, thanks!

I am able to verify the Execution Query Plan on this example:
https://github.com/felipegutierrez/flink-first/blob/master/src/main/java/flink/example/streaming/SocketWindowWordCountFilterJava.java
I am also going to build a little POC like you said.

Thanks,
Felipe
*--*
*-- Felipe Gutierrez*

*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
<https://felipeogutierrez.blogspot.com>*


On Fri, Nov 30, 2018 at 11:33 AM Fabian Hueske <fhue...@gmail.com> wrote:

> Hi Felipe,
>
> You can define TableSources (for SQL, Table API) that support filter
> push-down.
> The optimizer will figure out this opportunity and hand filters to a
> custom TableSource.
>
> I should add that AFAIK this feature is not used very often (expect some
> rough edges) and that the API is likely to change in the future.
> But it might be enough for a simple POC.
>
> Best, Fabian
>
> Am Fr., 30. Nov. 2018 um 10:13 Uhr schrieb Felipe Gutierrez <
> felipe.o.gutier...@gmail.com>:
>
>> I guess this message from 2016 is very related of what I am looking for (
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Execution-Plan-td4290.html).
>> I am posting here for future references.
>>
>> I am going to implement a toy example to visualize this. Do you guys see
>> this description as actual of latest Flink source code?
>> *--*
>> *-- Felipe Gutierrez*
>>
>> *-- skype: felipe.o.gutierrez*
>> *--* *https://felipeogutierrez.blogspot.com
>> <https://felipeogutierrez.blogspot.com>*
>>
>>
>> On Thu, Nov 29, 2018 at 12:01 PM Felipe Gutierrez <
>> felipe.o.gutier...@gmail.com> wrote:
>>
>>> thanks Kostas for the quick reply,
>>>
>>> yes. It is related to my previous question.
>>>
>>> When you said "But if you know what operation to push down" -> This is
>>> what I am trying to search on Flink code. I want to know the operation on
>>> the fly.
>>> The component on Flink that will say to me that there is a filter on the
>>> query specified by the user. I want to get this metadata and send a message
>>> to my RPi through a Flink connector (I guess this is the way to do) and the
>>> data stream will come to Flink already filtered.
>>>
>>> I intend to start with a simple and naive example. Do you know which
>>> component on Flink I can get the operations on the fly that are
>>> running inside a query?
>>>
>>> thanks
>>> *--*
>>> *-- Felipe Gutierrez*
>>>
>>> *-- skype: felipe.o.gutierrez*
>>> *--* *https://felipeogutierrez.blogspot.com
>>> <https://felipeogutierrez.blogspot.com>*
>>>
>>>
>>> On Thu, Nov 29, 2018 at 11:18 AM Kostas Kloudas <
>>> k.klou...@data-artisans.com> wrote:
>>>
>>>> Hi again,
>>>>
>>>> I forgot to say that, unfortunately, I am not familiar with Apache
>>>> Edgent, but if you can write your filter in Edgent's programming model,
>>>> Then you can push your data from Edgent to a third party storage system
>>>> (e.g. Kafka, HDFS, etc) and use Flink's connectors, instead of
>>>> having to implement a custom source.
>>>>
>>>> Cheers,
>>>> Kostas
>>>>
>>>> On Thu, Nov 29, 2018 at 11:08 AM Kostas Kloudas <
>>>> k.klou...@data-artisans.com> wrote:
>>>>
>>>>> Hi Felipe,
>>>>>
>>>>> This seems related to your previous question about a custom scheduler
>>>>> that knows which task to run on which machine.
>>>>> As Chesnay said, this is a rather involved and laborious task, if you
>>>>> want to do it as a general framework.
>>>>>
>>>>> But if you know what operation to push down, then why not decoupling
>>>>> the two and implementing the filtering as a separate job
>>>>> running on your Raspberry and a new job which consumes the output of
>>>>> the first and does the analytics?
>>>>>
>>>>> Cheers,
>>>>> Kostas
>>>>>
>>>>> On Thu, Nov 29, 2018 at 10:23 AM Felipe Gutierrez <
>>>>> felipe.o.gutier...@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am trying to design a little prototype with Flink and Apache Edgent
>>>>>> (http://edgent.apache.org/) and I would like some help on the
>>>>>> direction for it. I am running Flink at my laptop and Edgent on my
>>>>>> Raspberry Pi with a simple filter for a proximity sensor (
>>>>>> https://github.com/felipegutierrez/explore-rpi/blob/master/src/main/java/org/sense/edgent/app/UltrasonicEdgentApp.java
>>>>>> ).
>>>>>>
>>>>>> My idea is to push down the filter operator from Flink to the
>>>>>> Raspberry Pi which is running Apache Edgent. With this in mind, where do
>>>>>> you guys advise me to start?
>>>>>>
>>>>>> I have some ideas to study...
>>>>>> 1 - Try to get the list of operators that Flink is about to execute
>>>>>> on the JobManager. source:
>>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/internals/job_scheduling.html
>>>>>> 2 - Implement a connector to Apache Edgent in order to exchange
>>>>>> messages between Flink and Edgent.
>>>>>>
>>>>>> Do you guys think in another source that is interesting regarding my
>>>>>> prototype?
>>>>>>
>>>>>> Thanks,
>>>>>> Felipe
>>>>>> *--*
>>>>>> *-- Felipe Gutierrez*
>>>>>>
>>>>>> *-- skype: felipe.o.gutierrez*
>>>>>> *--* *https://felipeogutierrez.blogspot.com
>>>>>> <https://felipeogutierrez.blogspot.com>*
>>>>>>
>>>>>

Reply via email to