Thanks Seth Will take a look.
> On May 27, 2020, at 3:15 PM, Seth Wiesman <sjwies...@gmail.com> wrote: > > Hi Boris, > > Example usage of flink sources and sink is available in the documentation[1]. > > [1] > https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/io-module/flink-connectors.html > > <https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.0/io-module/flink-connectors.html> > On Wed, May 27, 2020 at 1:08 PM Boris Lublinsky > <boris.lublin...@lightbend.com <mailto:boris.lublin...@lightbend.com>> wrote: > Thats not exactly the usage question that I am asking > When I am writing IO module I have to write Ingress and Egress spec. > You have an example for Kafka, which looks like > > def getIngressSpec: IngressSpec[GreetRequest] = > KafkaIngressBuilder.forIdentifier(GREETING_INGRESS_ID) > .withKafkaAddress(kafkaAddress) > .withTopic("names") > .withDeserializer(classOf[GreetKafkaDeserializer]) > .withProperty(ConsumerConfig.GROUP_ID_CONFIG, "greetings") > .build > > def getEgressSpec: EgressSpec[GreetResponse] = > KafkaEgressBuilder.forIdentifier(GREETING_EGRESS_ID) > .withKafkaAddress(kafkaAddress) > .withSerializer(classOf[GreetKafkaSerializer]) > .build > How is it going to look if I am using SourceSinkModule? > Do I just specify stream names? Something else? > > > > > >> On May 27, 2020, at 11:29 AM, Tzu-Li (Gordon) Tai <tzuli...@apache.org >> <mailto:tzuli...@apache.org>> wrote: >> >> >> >> On Thu, May 28, 2020, 12:19 AM Boris Lublinsky >> <boris.lublin...@lightbend.com <mailto:boris.lublin...@lightbend.com>> wrote: >> I think I figured this out. >> The project seems to be missing >> >> resources >> <https://github.com/ververica/flink-statefun-workshop/tree/master/statefun-workshop-functions/src/main/resources>/META-INF >> >> <https://github.com/ververica/flink-statefun-workshop/tree/master/statefun-workshop-functions/src/main/resources/META-INF>/services >> directory, which should contain services >> >> Yes, the functions / ingresses / regresses etc. are not discoverable if the >> service file isnt present in the classpath. >> >> For the examples, if you are running it straight from the repo, should all >> have that service file defined and therefore readily runnable. >> >> If you are creating your own application project, you'll have to add that >> yourself. >> >> >> Another question: >> I see org.apache.flink.statefun.flink.io.datastream.SourceSinkModule >> >> Class, which I think allows to use existing data streams as ingress/egress. >> >> Are there any examples of its usage >> >> On the Harness class, there is a withFlinkSourceFunction method in which you >> can directly add a Flink source function as the ingress. >> >> If you want to use that directly in a normal application (not just execution >> in IDE with the Harness), you can define your ingesses/egresses by binding >> SourceFunctionSpec / SinkFunctionSpec. >> Please see how they are being used in the Harness class for examples. >> >> Gordon >> >> >> >>> On May 27, 2020, at 11:10 AM, Tzu-Li (Gordon) Tai <tzuli...@apache.org >>> <mailto:tzuli...@apache.org>> wrote: >>> >>> Hi, >>> >>> The example is working fine on my side (also using IntelliJ). >>> This could most likely be a problem with your project setup in the IDE, >>> where the classpath isn't setup correctly. >>> >>> What do you see when you right click on the statefun-flink-harness-example >>> directory (in the IDE) --> Open Module Settings, and then under the >>> "Sources" / "Dependencies" tab? >>> Usually this should all be automatically setup correctly when importing the >>> project. >>> >>> Gordon >>> >>> On Wed, May 27, 2020 at 11:46 PM Boris Lublinsky >>> <boris.lublin...@lightbend.com <mailto:boris.lublin...@lightbend.com>> >>> wrote: >>> The project >>> https://github.com/apache/flink-statefun/tree/release-2.0/statefun-examples/statefun-flink-harness-example >>> >>> <https://github.com/apache/flink-statefun/tree/release-2.0/statefun-examples/statefun-flink-harness-example> >>> Does not work in Intellij. >>> >>> The problem is that when running in Intellij, method public static Modules >>> loadFromClassPath() { >>> Does not pick up classes, which are local in Intellij >>> >>> Any work arounds? >>> >>> >>> >>> >>>> On May 22, 2020, at 12:03 AM, Tzu-Li (Gordon) Tai <tzuli...@apache.org >>>> <mailto:tzuli...@apache.org>> wrote: >>>> >>>> Hi, >>>> >>>> Sorry, I need to correct my comment on using the Kafka ingress / egress >>>> with the Harness. >>>> >>>> That is actually doable, by adding an extra dependency to >>>> `statefun-flink-distribution` in your Harness program. >>>> That pulls in all the other required dependencies required by the Kafka >>>> ingress / egress, such as the source / sink providers and Flink Kafka >>>> connectors. >>>> >>>> Cheers, >>>> Gordon >>>> >>>> On Fri, May 22, 2020 at 12:04 PM Tzu-Li (Gordon) Tai <tzuli...@apache.org >>>> <mailto:tzuli...@apache.org>> wrote: >>>> Are you getting an exception from running the Harness? >>>> The Harness should already have the required configurations, such as the >>>> parent first classloading config. >>>> >>>> Otherwise, if you would like to add your own configuration, use the >>>> `withConfiguration` method on the `Harness` class. >>>> >>>> On Fri, May 22, 2020 at 7:19 AM Boris Lublinsky >>>> <boris.lublin...@lightbend.com <mailto:boris.lublin...@lightbend.com>> >>>> wrote: >>>> Also, where do I put flint-conf.yaml in Idea to add additional required >>>> config parameter: >>>> classloader.parent-first-patterns.additional: >>>> org.apache.flink.statefun;org.apache.kafka;com.google.protobuf >>>> >>>> >>>>> On May 21, 2020, at 12:22 PM, Boris Lublinsky >>>>> <boris.lublin...@lightbend.com <mailto:boris.lublin...@lightbend.com>> >>>>> wrote: >>>>> >>>>> Hi, >>>>> I am trying to run >>>>> https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-greeter-example >>>>> >>>>> <https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-greeter-example> >>>>> locally >>>>> using >>>>> https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-flink-harness-example >>>>> >>>>> <https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-flink-harness-example> >>>>> >>>>> And have several questions. >>>>> 1. It seems fairly straightforward to use it with in memory message >>>>> generators, but I can’t figure out how to add Kafka ingress/Egress so >>>>> that I can use it with Kafk >>>>> 2. GreetingModule already creates StatefulFunctionUniverse and so does >>>>> Harness. Is there a way to short circuit it and have Harness get >>>>> StatefulFunctionUniverse directly >>>>> 3. Is there an example on how to write Flink main for stageful function? >>>>> 4. Is there an example anywhere on how to run such examples in the IDE >>>>> with Kafka? >>>>> 5 There is a great stateful functions example >>>>> https://github.com/ververica/flink-statefun-workshop >>>>> <https://github.com/ververica/flink-statefun-workshop>, but its readme >>>>> does not really describe implementation and neither does this article, >>>>> referencing it >>>>> https://dev.to/morsapaes/flink-stateful-functions-where-to-start-2j39 >>>>> <https://dev.to/morsapaes/flink-stateful-functions-where-to-start-2j39>. >>>>> Is there anything that describes this implementation? >>>>> >>>> >>> >> >
