Nice paper to read and cool usage of Kafka. Thanks for sharing Afonso :)
Guozhang On Mon, Mar 12, 2018 at 1:13 PM, Afonso Mukai <afonso.mu...@esss.se> wrote: > Hi Hannes, > > We will use Kafka here at the European Spallation Source (the facility is > currently under construction) to stream data from neutron detectors and > other experimental station equipment to consumers (the EPICS forwarding > software mentioned by Eric covers the sources other than detectors). From > Kafka the data follow two different paths: > > - For online reduction and visualisation, data will be consumed directly > from Kafka topics by the applications performing these tasks. > - A separate consumer (https://github.com/ess-dmsc/kafka-to-nexus) > subscribes to these data and writes them to HDF5 files, for subsequent > offline reduction and analysis. > > An overview of the system architecture is given in > http://accelconf.web.cern.ch/AccelConf/icalepcs2017/papers/tupha177.pdf > > Best regards, > Afonso > > > On 12/03/2018, 19:49, "Berryman, Eric" <berry...@frib.msu.edu> wrote: > > > The European Spallation Source [1] seems to be using it for this case > [2]. > > > I am also using this code [2], but only for visualization in another > "data center". > > > > [1] https://europeanspallationsource.se/ > > > [2] https://github.com/ess-dmsc/forward-epics-to-kafka > > > Thank you! > > Eric > > > > > ________________________________ > From: Hannes Petri <hannespe...@gmail.com> > Sent: Monday, March 12, 2018 1:37:29 PM > To: users@kafka.apache.org > Subject: Using Kafka to build a streaming platform for a research > facility? > > Hi, > > I work at a research facility where numerous hi-res detectors produce > thousands of GB of data every day. We want to build a highly flexible and > performant streaming platform for storing, transmitting and routing the > data. For example, detector output needs to end up: > > 1. In permanent storage systems > 2. In realtime or semi-realtime visualization software > 3. In post-processing and analysis software > 4. In metrics software > > ...and possibly more. Now I'm exploring Kafka as an option to back > such a platform. > > Would Kafka be a good fit? The reason I'm asking is because among the > use cases, I've mostly seen Kafka being used with more lightweight data, > geared towards business events and high frequency streams of text and > scalars. In other words, *more* but *smaller* messages. > > In my situation, we'd be looking at low frequency but huge files > (typically these detectors produce one large file at a time). In order not > to flood the storage, raw data topics would need to have a very short > retention time (hours to days). > > Does anybody have experience in using Kafka in a similar scenario? > What are your thoughts about the situation I describe? Would we benefit > from using Kafka? > > I highly appreciate any input on this. Many thanks in advance. > > Best regards > Hannes > > Sent from my iPhone > > > -- -- Guozhang
