thanks Kostas for the quick reply, yes. It is related to my previous question.
When you said "But if you know what operation to push down" -> This is what I am trying to search on Flink code. I want to know the operation on the fly. The component on Flink that will say to me that there is a filter on the query specified by the user. I want to get this metadata and send a message to my RPi through a Flink connector (I guess this is the way to do) and the data stream will come to Flink already filtered. I intend to start with a simple and naive example. Do you know which component on Flink I can get the operations on the fly that are running inside a query? thanks *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com <https://felipeogutierrez.blogspot.com>* On Thu, Nov 29, 2018 at 11:18 AM Kostas Kloudas <k.klou...@data-artisans.com> wrote: > Hi again, > > I forgot to say that, unfortunately, I am not familiar with Apache Edgent, > but if you can write your filter in Edgent's programming model, > Then you can push your data from Edgent to a third party storage system > (e.g. Kafka, HDFS, etc) and use Flink's connectors, instead of > having to implement a custom source. > > Cheers, > Kostas > > On Thu, Nov 29, 2018 at 11:08 AM Kostas Kloudas < > k.klou...@data-artisans.com> wrote: > >> Hi Felipe, >> >> This seems related to your previous question about a custom scheduler >> that knows which task to run on which machine. >> As Chesnay said, this is a rather involved and laborious task, if you >> want to do it as a general framework. >> >> But if you know what operation to push down, then why not decoupling the >> two and implementing the filtering as a separate job >> running on your Raspberry and a new job which consumes the output of the >> first and does the analytics? >> >> Cheers, >> Kostas >> >> On Thu, Nov 29, 2018 at 10:23 AM Felipe Gutierrez < >> felipe.o.gutier...@gmail.com> wrote: >> >>> Hi, >>> >>> I am trying to design a little prototype with Flink and Apache Edgent ( >>> http://edgent.apache.org/) and I would like some help on the direction >>> for it. I am running Flink at my laptop and Edgent on my Raspberry Pi with >>> a simple filter for a proximity sensor ( >>> https://github.com/felipegutierrez/explore-rpi/blob/master/src/main/java/org/sense/edgent/app/UltrasonicEdgentApp.java >>> ). >>> >>> My idea is to push down the filter operator from Flink to the Raspberry >>> Pi which is running Apache Edgent. With this in mind, where do you guys >>> advise me to start? >>> >>> I have some ideas to study... >>> 1 - Try to get the list of operators that Flink is about to execute on >>> the JobManager. source: >>> https://ci.apache.org/projects/flink/flink-docs-stable/internals/job_scheduling.html >>> 2 - Implement a connector to Apache Edgent in order to exchange messages >>> between Flink and Edgent. >>> >>> Do you guys think in another source that is interesting regarding my >>> prototype? >>> >>> Thanks, >>> Felipe >>> *--* >>> *-- Felipe Gutierrez* >>> >>> *-- skype: felipe.o.gutierrez* >>> *--* *https://felipeogutierrez.blogspot.com >>> <https://felipeogutierrez.blogspot.com>* >>> >>
