Hi Boris, Example usage of flink sources and sink is available in the documentation[1].
[1] https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/io-module/flink-connectors.html On Wed, May 27, 2020 at 1:08 PM Boris Lublinsky < boris.lublin...@lightbend.com> wrote: > Thats not exactly the usage question that I am asking > When I am writing IO module I have to write Ingress and Egress spec. > You have an example for Kafka, which looks like > > def getIngressSpec: IngressSpec[GreetRequest] = > KafkaIngressBuilder.forIdentifier(GREETING_INGRESS_ID) > .withKafkaAddress(kafkaAddress) > .withTopic("names") > .withDeserializer(classOf[GreetKafkaDeserializer]) > .withProperty(ConsumerConfig.GROUP_ID_CONFIG, "greetings") > .build > > def getEgressSpec: EgressSpec[GreetResponse] = > KafkaEgressBuilder.forIdentifier(GREETING_EGRESS_ID) > .withKafkaAddress(kafkaAddress) > .withSerializer(classOf[GreetKafkaSerializer]) > .build > > How is it going to look if I am using SourceSinkModule? > Do I just specify stream names? Something else? > > > > > > On May 27, 2020, at 11:29 AM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> > wrote: > > > > On Thu, May 28, 2020, 12:19 AM Boris Lublinsky < > boris.lublin...@lightbend.com> wrote: > >> I think I figured this out. >> The project seems to be missing >> >> resources >> <https://github.com/ververica/flink-statefun-workshop/tree/master/statefun-workshop-functions/src/main/resources> >> /META-INF >> <https://github.com/ververica/flink-statefun-workshop/tree/master/statefun-workshop-functions/src/main/resources/META-INF> >> /services directory, which should contain services >> > > Yes, the functions / ingresses / regresses etc. are not discoverable if > the service file isnt present in the classpath. > > For the examples, if you are running it straight from the repo, should all > have that service file defined and therefore readily runnable. > > If you are creating your own application project, you'll have to add that > yourself. > > >> Another question: >> I see org.apache.flink.statefun.flink.io.datastream.SourceSinkModule >> >> Class, which I think allows to use existing data streams as >> ingress/egress. >> >> Are there any examples of its usage >> > > On the Harness class, there is a withFlinkSourceFunction method in which > you can directly add a Flink source function as the ingress. > > If you want to use that directly in a normal application (not just > execution in IDE with the Harness), you can define your ingesses/egresses > by binding SourceFunctionSpec / SinkFunctionSpec. > Please see how they are being used in the Harness class for examples. > > Gordon > > >> >> On May 27, 2020, at 11:10 AM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> >> wrote: >> >> Hi, >> >> The example is working fine on my side (also using IntelliJ). >> This could most likely be a problem with your project setup in the IDE, >> where the classpath isn't setup correctly. >> >> What do you see when you right click on the >> statefun-flink-harness-example directory (in the IDE) --> Open Module >> Settings, and then under the "Sources" / "Dependencies" tab? >> Usually this should all be automatically setup correctly when importing >> the project. >> >> Gordon >> >> On Wed, May 27, 2020 at 11:46 PM Boris Lublinsky < >> boris.lublin...@lightbend.com> wrote: >> >>> The project >>> https://github.com/apache/flink-statefun/tree/release-2.0/statefun-examples/statefun-flink-harness-example >>> Does not work in Intellij. >>> >>> The problem is that when running in Intellij, method public static Modules >>> loadFromClassPath() { >>> Does not pick up classes, which are local in Intellij >>> >>> Any work arounds? >>> >>> >>> >>> >>> On May 22, 2020, at 12:03 AM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> >>> wrote: >>> >>> Hi, >>> >>> Sorry, I need to correct my comment on using the Kafka ingress / egress >>> with the Harness. >>> >>> That is actually doable, by adding an extra dependency to >>> `statefun-flink-distribution` in your Harness program. >>> That pulls in all the other required dependencies required by the Kafka >>> ingress / egress, such as the source / sink providers and Flink Kafka >>> connectors. >>> >>> Cheers, >>> Gordon >>> >>> On Fri, May 22, 2020 at 12:04 PM Tzu-Li (Gordon) Tai < >>> tzuli...@apache.org> wrote: >>> >>>> Are you getting an exception from running the Harness? >>>> The Harness should already have the required configurations, such as >>>> the parent first classloading config. >>>> >>>> Otherwise, if you would like to add your own configuration, use the >>>> `withConfiguration` method on the `Harness` class. >>>> >>>> On Fri, May 22, 2020 at 7:19 AM Boris Lublinsky < >>>> boris.lublin...@lightbend.com> wrote: >>>> >>>>> Also, where do I put flint-conf.yaml in Idea to add additional >>>>> required config parameter: >>>>> >>>>> classloader.parent-first-patterns.additional: >>>>> org.apache.flink.statefun;org.apache.kafka;com.google.protobuf >>>>> >>>>> >>>>> >>>>> On May 21, 2020, at 12:22 PM, Boris Lublinsky < >>>>> boris.lublin...@lightbend.com> wrote: >>>>> >>>>> Hi, >>>>> I am trying to run >>>>> https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-greeter-example >>>>> locally >>>>> using >>>>> https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-flink-harness-example >>>>> >>>>> And have several questions. >>>>> 1. It seems fairly straightforward to use it with in memory message >>>>> generators, but I can’t figure out how to add Kafka ingress/Egress so that >>>>> I can use it with Kafk >>>>> 2. GreetingModule already creates StatefulFunctionUniverse and so >>>>> does Harness. Is there a way to short circuit it and have Harness get >>>>> StatefulFunctionUniverse directly >>>>> 3. Is there an example on how to write Flink main for stageful >>>>> function? >>>>> 4. Is there an example anywhere on how to run such examples in the IDE >>>>> with Kafka? >>>>> 5 There is a great stateful functions example >>>>> https://github.com/ververica/flink-statefun-workshop, but its readme >>>>> does not really describe implementation and neither does this article, >>>>> referencing it >>>>> https://dev.to/morsapaes/flink-stateful-functions-where-to-start-2j39. >>>>> Is there anything that describes this implementation? >>>>> >>>>> >>>>> >>> >> >
