Thanks Igal for the detailed explanantion. I know that this was only a demo, I just wanted to reason a bit on the pros and cons of sending data to an UI from Flink.
Best, Flavio Il Ven 1 Nov 2019, 12:21 Igal Shilman <i...@ververica.com> ha scritto: > Hi Flavio, let me try to clarify: > > The intention of this example is to demonstrate how > different entities (drivers, passengers, etc') participates in a protocol > (ride matching). For that we have the stateful functions application, and a > standalone java application that just generates the events to trigger the > simulation (i.e. a passenger requests a ride would be an event that the > simulator emits). > The visualization aspect of this example is not a part of what we are > trying to demonstrate but rather, an out-of-band peek into the live system > (websocket that duplicates the events directly to the UI) which we thought > is a nice addition to the example that visualizes how cars are moving on > the grid :-) > > If your goal is to create a scalable visualization of car locations, then > I guess your suggestion to break the grid into distinct geo-regions can > work. For that you would need: > 1. An additional egress backed by a Kafka topic. > 2. Modify the driver function to send location changes to that egress, > where you'd use the region id (geohash with a resolution to your liking) as > key [1]. > 3. At the consuming side subscribe to the partition that holds the region > of interest. > > But, realistically speaking for this kind of a visualization you would be > better off consuming from the locations update topic (from-driver topic) > and populating an optimized geo-indexed store, > and querying it directly. > > I hope this clarifies things, > Igal > > [1] > https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L121 > > On Thu, Oct 31, 2019 at 6:21 PM Flavio Pompermaier <pomperma...@okkam.it> > wrote: > >> Thanks Igal, this is more or less what I was expecting..this implies that >> ALL events are received on the UI side. >> I was concerned about the tradeoffs of this choice: when I zoom on the >> map I could simply ignore messages outside the boundaries (but I still >> spend many cpu resource in the reading of useless messages). >> >> In the case of a worldwide company (eg uber or simolar) it's probably better >> to create a topic per geographical arean..but also in this case then the UI >> should know when to attach or detach from queue topics when it reach a >> section of the map served by different topics..in another case I could just >> have too many events also in the single map section (let's think about some >> big city with many user). >> >> Is there any talk at FF ir someone else that faced those issues too? >> >> Il Gio 31 Ott 2019, 17:44 Igal Shilman <i...@ververica.com> ha scritto: >> >>> For that particular example, the simulator [1] is responsible for >>> simulating physical drivers and passengers that interact with their >>> corresponding >>> stateful functions [2]. >>> The interaction between the simulator and the stateful functions is >>> happening via four Kafka topics: >>> * to-driver - messages that are sent from the physical drivers (the >>> simulator [1]) to the stateful functions. The messages always carry a >>> driver-id which acts as a routing key (I think that this is what you mean >>> by correlation id) to a specific driver stateful function (FnDriver) >>> * from-driver - messages that are sent from a stateful function with a >>> specific driver id to the simulator >>> * to-passenger - symmetric to to-driver >>> * from-passenger - symmetric to from-driver. >>> The ingress and egress definition are specified here [3], and you may >>> want to checkout how to router is defined as well [4][5]. >>> >>> In addition the simulator is also feeding the UI directly by duplicating >>> the messages to a web socket (see [6]) >>> >>> I hope this clarifies the examples. >>> >>> Igal. >>> >>> [1] >>> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator >>> [2] >>> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions >>> [3] >>> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L43 >>> [4] >>> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/Module.java#L33 >>> [5] >>> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/InboundDriverRouter.java#L26 >>> [6] >>> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator/simulation/Driver.java#L70 >>> >>> On Thu, Oct 31, 2019 at 4:15 PM Flavio Pompermaier <pomperma...@okkam.it> >>> wrote: >>> >>>> Yes, I'm interested in how to read data from a UI..which egress should >>>> I use? If we use a kafka queue, how to filter data received in the topic? >>>> Should I use a correlation id or use a new topic per user? >>>> >>>> Il Gio 31 Ott 2019, 16:08 Igal Shilman <i...@ververica.com> ha scritto: >>>> >>>>> Hi Flavio, >>>>> >>>>> We haven't included the UI source code just yet, we've only used it >>>>> for demos and talks. >>>>> >>>>> The reason is that (1) we didn't put a lot of effort and time there >>>>> (2) didn't check the time to go through the individual dependencies and >>>>> licences. >>>>> But we will add that very soon. >>>>> >>>>> Would having the UI code there would improve your understanding? or is >>>>> there another reason? >>>>> >>>>> Thanks, >>>>> Igal >>>>> >>>>> On Thu, Oct 31, 2019 at 5:44 AM Flavio Pompermaier < >>>>> pomperma...@okkam.it> wrote: >>>>> >>>>>> Hi Vino, >>>>>> I already checked that code but I can't find the UI part :( >>>>>> >>>>>> On Thu, Oct 31, 2019 at 12:32 PM vino yang <yanghua1...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Flavio, >>>>>>> >>>>>>> Please see this link.[1] >>>>>>> >>>>>>> Best, >>>>>>> Vino >>>>>>> >>>>>>> [1]: >>>>>>> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example >>>>>>> >>>>>>> Flavio Pompermaier <pomperma...@okkam.it> 于2019年10月31日周四 下午4:53写道： >>>>>>> >>>>>>>> Hi to all, >>>>>>>> yould it be possible to provide also the source code of the UI part >>>>>>>> of the ride sharing example? It would be interesting to me how the UI >>>>>>>> is >>>>>>>> reading the data from the Kafka egress. >>>>>>>> >>>>>>>> Best, >>>>>>>> Flavio >>>>>>>> >>>>>>> >>>>>>
