Thats not exactly the usage question that I am asking
When I am writing IO module I have to write Ingress and Egress spec.
You have an example for Kafka, which looks like
def getIngressSpec: IngressSpec[GreetRequest] =
KafkaIngressBuilder.forIdentifier(GREETING_INGRESS_ID)
.withKafkaAddress(kafkaAddress)
.withTopic("names")
.withDeserializer(classOf[GreetKafkaDeserializer])
.withProperty(ConsumerConfig.GROUP_ID_CONFIG, "greetings")
.build
def getEgressSpec: EgressSpec[GreetResponse] =
KafkaEgressBuilder.forIdentifier(GREETING_EGRESS_ID)
.withKafkaAddress(kafkaAddress)
.withSerializer(classOf[GreetKafkaSerializer])
.build
How is it going to look if I am using SourceSinkModule?
Do I just specify stream names? Something else?
> On May 27, 2020, at 11:29 AM, Tzu-Li (Gordon) Tai <[email protected]> wrote:
>
>
>
> On Thu, May 28, 2020, 12:19 AM Boris Lublinsky <[email protected]
> <mailto:[email protected]>> wrote:
> I think I figured this out.
> The project seems to be missing
>
> resources
> <https://github.com/ververica/flink-statefun-workshop/tree/master/statefun-workshop-functions/src/main/resources>/META-INF
>
> <https://github.com/ververica/flink-statefun-workshop/tree/master/statefun-workshop-functions/src/main/resources/META-INF>/services
> directory, which should contain services
>
> Yes, the functions / ingresses / regresses etc. are not discoverable if the
> service file isnt present in the classpath.
>
> For the examples, if you are running it straight from the repo, should all
> have that service file defined and therefore readily runnable.
>
> If you are creating your own application project, you'll have to add that
> yourself.
>
>
> Another question:
> I see org.apache.flink.statefun.flink.io.datastream.SourceSinkModule
>
> Class, which I think allows to use existing data streams as ingress/egress.
>
> Are there any examples of its usage
>
> On the Harness class, there is a withFlinkSourceFunction method in which you
> can directly add a Flink source function as the ingress.
>
> If you want to use that directly in a normal application (not just execution
> in IDE with the Harness), you can define your ingesses/egresses by binding
> SourceFunctionSpec / SinkFunctionSpec.
> Please see how they are being used in the Harness class for examples.
>
> Gordon
>
>
>
>> On May 27, 2020, at 11:10 AM, Tzu-Li (Gordon) Tai <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Hi,
>>
>> The example is working fine on my side (also using IntelliJ).
>> This could most likely be a problem with your project setup in the IDE,
>> where the classpath isn't setup correctly.
>>
>> What do you see when you right click on the statefun-flink-harness-example
>> directory (in the IDE) --> Open Module Settings, and then under the
>> "Sources" / "Dependencies" tab?
>> Usually this should all be automatically setup correctly when importing the
>> project.
>>
>> Gordon
>>
>> On Wed, May 27, 2020 at 11:46 PM Boris Lublinsky
>> <[email protected] <mailto:[email protected]>> wrote:
>> The project
>> https://github.com/apache/flink-statefun/tree/release-2.0/statefun-examples/statefun-flink-harness-example
>>
>> <https://github.com/apache/flink-statefun/tree/release-2.0/statefun-examples/statefun-flink-harness-example>
>> Does not work in Intellij.
>>
>> The problem is that when running in Intellij, method public static Modules
>> loadFromClassPath() {
>> Does not pick up classes, which are local in Intellij
>>
>> Any work arounds?
>>
>>
>>
>>
>>> On May 22, 2020, at 12:03 AM, Tzu-Li (Gordon) Tai <[email protected]
>>> <mailto:[email protected]>> wrote:
>>>
>>> Hi,
>>>
>>> Sorry, I need to correct my comment on using the Kafka ingress / egress
>>> with the Harness.
>>>
>>> That is actually doable, by adding an extra dependency to
>>> `statefun-flink-distribution` in your Harness program.
>>> That pulls in all the other required dependencies required by the Kafka
>>> ingress / egress, such as the source / sink providers and Flink Kafka
>>> connectors.
>>>
>>> Cheers,
>>> Gordon
>>>
>>> On Fri, May 22, 2020 at 12:04 PM Tzu-Li (Gordon) Tai <[email protected]
>>> <mailto:[email protected]>> wrote:
>>> Are you getting an exception from running the Harness?
>>> The Harness should already have the required configurations, such as the
>>> parent first classloading config.
>>>
>>> Otherwise, if you would like to add your own configuration, use the
>>> `withConfiguration` method on the `Harness` class.
>>>
>>> On Fri, May 22, 2020 at 7:19 AM Boris Lublinsky
>>> <[email protected] <mailto:[email protected]>>
>>> wrote:
>>> Also, where do I put flint-conf.yaml in Idea to add additional required
>>> config parameter:
>>> classloader.parent-first-patterns.additional:
>>> org.apache.flink.statefun;org.apache.kafka;com.google.protobuf
>>>
>>>
>>>> On May 21, 2020, at 12:22 PM, Boris Lublinsky
>>>> <[email protected] <mailto:[email protected]>>
>>>> wrote:
>>>>
>>>> Hi,
>>>> I am trying to run
>>>> https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-greeter-example
>>>>
>>>> <https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-greeter-example>
>>>> locally
>>>> using
>>>> https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-flink-harness-example
>>>>
>>>> <https://github.com/apache/flink-statefun/tree/master/statefun-examples/statefun-flink-harness-example>
>>>>
>>>> And have several questions.
>>>> 1. It seems fairly straightforward to use it with in memory message
>>>> generators, but I can’t figure out how to add Kafka ingress/Egress so that
>>>> I can use it with Kafk
>>>> 2. GreetingModule already creates StatefulFunctionUniverse and so does
>>>> Harness. Is there a way to short circuit it and have Harness get
>>>> StatefulFunctionUniverse directly
>>>> 3. Is there an example on how to write Flink main for stageful function?
>>>> 4. Is there an example anywhere on how to run such examples in the IDE
>>>> with Kafka?
>>>> 5 There is a great stateful functions example
>>>> https://github.com/ververica/flink-statefun-workshop
>>>> <https://github.com/ververica/flink-statefun-workshop>, but its readme
>>>> does not really describe implementation and neither does this article,
>>>> referencing it
>>>> https://dev.to/morsapaes/flink-stateful-functions-where-to-start-2j39
>>>> <https://dev.to/morsapaes/flink-stateful-functions-where-to-start-2j39>.
>>>> Is there anything that describes this implementation?
>>>>
>>>
>>
>
