Hi Dan, let me try to answer your questions:
> I guess my question is if one can
> freely mix Flink core with SF's code with regards to performance,
> fault-tolerance, and checkpointing?
The main limitations at the moment is that, currently SF requires a
processing time watermark semantics only
@Galen FYI: the upcoming StateFun release would use Scala2.12
On Thu, Sep 10, 2020 at 5:14 PM Seth Wiesman wrote:
> @glen
>
> Yes, we would absolutely migrate statefun. StateFun can be compiled with
> Scala 2.12 today, I'm not sure why it's not cross released.
>
> @aljoscha :)
>
> @mathieu Its
Hi,
Your observation is correct, currently the only way to invoke a remote
function is trough an HTTP POST request to a service that exposes a
StateFun endpoint.
The endpoint must implement the client side of a the “RequestReply”
protocol as defined by StateFun (basically an invocation contains t
Hi Tim,
You are correct, currently the argument to a remote function must be a
Protobuf Any, however StateFun doesn't interpret the contents of that Any,
and it would be passed as-is to the remote function.
As you mentioned in your email you can interpret the bytes as the bytes of
a JSON string.
Hi John,
Thank you for sharing your interesting use case!
Let me start from your second question:
> Are stateful functions available to all Flink jobs within a cluster?
Yes, the remote functions are some logic exposed behind an HTTP endpoint,
and Flink would forward any message addressed to th
> infrastructure. With that said, gRPC is also part of our requirements, thus
> motivation for the question.
>
>
>
> I’d love to hear more about plans to implement support for gRPC and
> perhaps become an early adopter.
>
>
>
> I hope this helps with understanding o
fig I
> can set in the "module.yaml" to have it just pack arbitrary bytes into the
> Any?
>
> Thanks,
>
> Tim
>
> On Wed, Sep 23, 2020 at 7:01 AM Igal Shilman wrote:
>
>> Hi Tim,
>>
>> You are correct, currently the argument to a remote funct
Hi Mazen,
What are the differences between Flink stateful functions and Event driven
> microservices are they almost the same concept
>
You can think of Stateful Functions as an API and a runtime that helps
building event driven microservices. It addresses some of the hardest parts
of composing s
Hi Dan,
I'm assuming that you have different Kafka topics, and each topic contains
messages of a single protobuf type.
In that case, you have to specify the mapping between a topic name to it's
Protobuf message type.
To do that, assume that you have a Kafka topic *A* that contains protobuf
messag
Hi Lian,
Good to hear that you are learning about StateFun, and I'd be happy to
answer any of your questions while doing so :-)
Perhaps in the future it would be best if you start a new email thread, so
that it would be easier to spot your question.
The following is completely thread safe:
final
Hi Jan,
The architecture outlined by you, sounds good and we've run successfully
mixed architectures like this.
Let me try to address your questions:
1)
To enable checkpointing you need to set the relevant values in your
flink-conf.yaml file.
execution.checkpointing.interval: (see [1])
state.che
>
> Could you give a quick opinion if these fears are even true and if so,
> what would be a recommended setup to satisfy these use cases?
>
>
> Best regards
>
> Jan
>
>
> On 05.11.20 17:02, Igal Shilman wrote:
>
> Hi Jan,
>
> The architecture outlined
Hi Lian,
If you are using the statefun-sdk directly (an embedded mode) then, most
likely is that you are missing a
META-INF/services/org.apache.flink.statefun.sdk.spi.StatefulFunctionModule
file that would point to your modules class. We are using Java SPI [1] to
load all the stateful functions mo
Glad to hear that it worked out!
On Wed, Nov 11, 2020 at 9:07 AM Lian Jiang wrote:
> Just realized making autoservice class discoverable also solved "There are
> no routers defined" mentioned by Puneet. Yes, harness does test statefun
> module discovery. Thanks.
>
> On Tue, Nov 10, 2020 at 9:57
Hi George,
Specifically for StateFun, we have the following Helm charts [1] to help
you deploy Stateful Functions on k8s.
The greeter example's docker-compose file also includes Kafka (and hence
Zookeeper).
Indeed the Flink cluster is "included" in the master/worker stateful
functions docker image
Glad to hear that you were able to resolve the issue!
One comment tho: I would really encourage you to upgrade to statefun 2.2.1,
which was released recently and it fixes a checkpointing related issue.
Kind regards,
Igal.
On Tue, Nov 24, 2020 at 10:10 PM Lian Jiang wrote:
> Probolved solved. I
Hello Jan,
The next release candidate for stateful functions is expected in mid
February, and the release itself would follow shortly thereafter.
This release will be based on Flink 1.12.
Cheers,
Igal.
On Tue, Dec 15, 2020 at 4:19 PM Jan Brusch
wrote:
> Hi,
>
> just a quick question: Is there
Hi Le,
You can attach many different functions in a single StateFun builder, and
let them message each other.
In your example, you can make the "Greet" function message Greet2 directly
(in addition to emitting a message as an egress).
Embedding multiple copies of StateFun within a Datastream appli
Hello :-)
It seems like in your attached pom you are not bundling the dependencies.
Check out the docs here [1].
[1]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/deployment-and-operations/packaging.html#flink-jar
On Wed, Dec 23, 2020 at 3:07 AM Le Xu wrote:
> Hello!
>
https://gist.github.com/flint-stone/059f00832d8b99af433a446771f4f740#file-pom-xml-L64
>
> Thanks!
>
> Le
>
> On Sat, Dec 26, 2020 at 2:23 PM Igal Shilman wrote:
>
>> Hello :-)
>>
>> It seems like in your attached pom you are not bundling the dependencies.
>> Check out the docs h
sendDownstream(FeedbackUnionOperator.java:186)
>>> at
>>> org.apache.flink.statefun.flink.core.feedback.FeedbackUnionOperator.processElement(FeedbackUnionOperator.java:86)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emit
Hi Le,
Let me try to answer to your multiple questions, one by one:
> I'm trying to understand the internal mechanism used by Flink Statefun to
> dispatch functions to Flink cluster. In particular, I was trying to find a
> good example demonstrating Statefun's "Logical Co-location, Physical
> Sep
orImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:288)
> ... 8 more
>
>
> Thanks for the help!
>
> Le
>
>
> On Mon, Dec 28, 2020 at 6:24 AM Igal Shilman wrot
Hello!
Is it possible to dynamically associate Remote Modules with Remote Function
> Endpoints to an existing/already running Flink stateful application?
>
Yes, in the upcoming StateFun release we are introducing exactly that :)
As of the upcoming version, we are adding a capability to dynamicall
k-core/src/test/resources/module-v3_0/module.yaml#L29
On Tue, Jan 12, 2021 at 7:16 PM Ahmad Alkilani wrote:
> That's awesome, thank you! JIRA I can follow?
>
> On Tue, Jan 12, 2021 at 9:01 AM Igal Shilman wrote:
>
>> Hello!
>>
>> Is it possible to dynamically
Hi Martijn,
I'm glad you like it! and we are always happy to learn about new use cases
:)
* How is the feedback stream exactly merged with the input stream from
> ingresses?
>
First, I'd like to refer you to this talk, that has a peek under the hood
part[1] (it starts at ~26min) where I outline
Hi,
I don't know what is the status of iterations at the moment, and whatever
the community has plans to work at that,
But I would like to point you to Flink Stateful Functions [1], a recent
contribution to Apache Flink that allows building
applications composed of stateful functions that can invok
Hi All,
One way to try to think about it with StateFun, is to represent the Graph
vertices as stateful functions instances. Unlike other frameworks an
instance of a function does not take any resources while idle, and
potentially you can have many millions of those.
A state for each vertex might be
Hi,
I'm glad to hear that your PoC with StateFun functions has turned out to be
successful, even
if it is for verifying external systems are integrating with each other
correctly.
I hope that eventually StateFun would replace the 3 external systems :-)
Good luck,
Igal.
On Fri, Apr 10, 2020 at 3
Hi,
I'm not familiar with the other library that you have mentioned, and indeed
using Flink CEP from within a stateful function is not possible within a
single Flink job, as Gordon mentioned.
I'm wondering what aspects of CEP are you interested in?
Because essentially a stateful function can be
Hi Annemarie,
There are plans to make stateful functions more easily embeddable within a
Flink Job,
perhaps skipping ingress/egress routing abstracting all together and
basically exposing the
core Flink job that is the heart of stateful functions.
Although these plans are not concrete yet I believe
Hi Max,
Sorry for not being clearer earlier, by splitting the pipeline I mean:
having a pre-processing job that does whatever transformations necessary
with the DataStream and outputs to Kafka / Kinesis,
and then having a separate StateFun deployment that consumes from that
transformed Kafka / Kin
Hi!
Glad to see that you are trying out StateFun!
The exception means that in your application you are missing a router
definition, this might happen if either:
1) indeed you did not bind any router
2) just forgot to include the stateful functions module. If you have copied
the code from an examp
Hi Wouter!
Glad to read that you are using Flink for quite some time, and also
exploring with StateFun!
1) yes it is correct and you can follow the Dockerhub contribution PR at [1]
2) I’m not sure I understand what do you mean by trigger from the browser.
If you mean, for testing / illustration
Hi all,
Data stream windows are not yet supported in statefun, but it seems like
the main motivation here
is to purge old edges?
If this is the case perhaps we need to integrate state TTL [1] into
persisted values/persistedtables.
An alternative approach would be to implement a thumbling window p
n.
I hope this helps,
Good luck!
Igal.
[1] https://issues.apache.org/jira/browse/FLINK-17644
On Fri, May 8, 2020 at 1:00 PM m@xi wrote:
> Dear Igal, Very insightful answer. Thanks.
>
> Igal Shilman wrote
> An alternative approach would be to implement a *thumbling window* per
>
ant to
>> experiment running stateful functions behind a front-end (which should be
>> able to trigger a function), but we're a bit afraid that using Kafka
>> doesn't scale well if on the frontend side a user has to consume all Kafka
>> messages to find the correct
parallelism, but I
> couldn't really find that option yet. Is there some value I need to set in
> the module.yaml.
>
> Once again, thanks for the help so far. It has been useful.
>
> Regards,
> Wouter
>
> Op wo 13 mei 2020 om 00:03 schreef Igal Shilman :
>
&
Hi Dinesh,
They are different projects that aim to solve different problems.
The streaming ledger goal is to bring serializable ACID Transactions to
streaming pipelines, while statefun's aim is to simplify building
distributed stateful applications in the serverless model.
To get more information
Hi,
Can you verify that your jar contains the following file
META-INF/services/org.apache.flink.statefun.sdk.spi.StatefulFunctionModule ?
Thanks,
Igal.
On Tue, May 26, 2020 at 11:49 AM C DINESH wrote:
> Hi Gordon,
>
> Thanks for your response.
>
> After adding this conf to flink-yml.
>
> `class
Hi Francesco,
It is absolutely possible to deploy some functions as embedded and some as
remote, and scale them independently, while technically being part of the
same
stateful function application instance (I think that what you meant by
"sharing the same master").
One possible way to do it in k
Hi Jan,
Judging by the exception message it seems like the function type
"demo/eventCounterPython" is not known to stateful functions.
This could happen if the module.yaml (provided in your email) was
accidentally excluded from the resulting artifact (Docker image or a
jar-with-dependencies)
Can yo
Hi Jan,
Stateful functions would look at the java class path for the module.yaml,
So one way would be including the module.yaml in your src/main/resources/
directory.
Good luck,
Igal.
On Mon, Jul 6, 2020 at 12:39 PM Jan Brusch
wrote:
> Hi,
>
> quick question about Deploying a Flink Stateful F
nt = 1
> else:
> state.count += 1
> context.state('count').pack(state)
>
>
> envelope = Any()
> envelope.Pack(state)
> context.reply(envelope)
>
>
>
> module.yaml
&g
Adding user@flink (was accidentally omitted previously)
On Fri, Jan 29, 2021 at 5:18 PM Igal Shilman wrote:
> Hi Jan,
>
> Glad to hear that 3.71 and 3.3.0 works for you. You can still include
> protobuf in your project, but the version needs to be compatible with
> what you will
Hi Frédérique!
Thank you for your kind words! let me try to answer your questions:
>From the email thread, it looks like there’s going to be support for
> dynamic function dispatch by name patterns which is pretty cool, but it
> sounds like you still need to redeploy if you add a new ingress or e
Hi Jan,
StateFun enables object reuse automatically, and it can't be disabled with
a configuration.
There is a technical reason for that that has to do with how we translate
StateFun concepts to Flink concepts.
I've created an issue to remove this limitation [1].
I might come up with a workaround
Hello Martijn,
Great to hear that you are exploring StateFun as part of your university
project!
Can you please clarify:
- how do you measure throughput?
- by co-located functions, do you mean a remote function on the same
machine?
- Can you share a little bit more about your functions, what are
Hi Stephan,
I think that what you are trying to achieve is very interesting, and
possibly other users might find that useful as well
and we will definitely add that to our roadmap.
I think that Gordon's suggestion of using the state processor API to
examine a savepoint, makes a lot of sense in th
Hello,
I believe that your assembly plugin configuration doesn't merge files under
META-INF/services.
Can you unzip your jar and examin manually the contents of:
META-INF/services/org.apache.flink.statefun.flink.io.spi.FlinkIoModule
It should include at least the following lines:
org.apache.flink
Hello Stephan,
The values are loaded lazily, upon access. And this applies to both
PersistedTable and PersistedValue.
For a PersistedTable in particular, calling get/set/remove will only affect
the specific key, and calling entries() will fetch the entire map.
Cheers,
Igal.
On Sun, Feb 14, 202
Hi Miguel,
I think that there are a couple of ways to achieve this, and it really
depends on your specific use case, and the trade-offs
that you are willing to accept.
For example, one way to approach this:
- Suppose you have an external service somewhere that returns a
representation of the logi
for this information, Piotr.
>
> The comment from Igal Shilman in FLINK-18894 issue says, "Obtaining a
> MAX_PRIO mailbox from StreamTask, solves this issue." I'm unclear what this
> means -- is this a workaround I can leverage?
>
> Dylan
> ---
Hello Marco!
Your understanding is correct, but in addition
You can also use StateFun within a DataStream application [1]
[1]
https://ci.apache.org/projects/flink/flink-statefun-docs-master/docs/sdk/flink-datastream/
On Wed, Apr 7, 2021 at 2:49 AM Marco Villalobos
wrote:
> Upon reading about s
question not addressed:
>
> Can a stateful function be called by a process function?
>
> On Wed, Apr 7, 2021 at 8:19 AM Igal Shilman wrote:
>
>> Hello Marco!
>>
>> Your understanding is correct, but in addition
>> You can also use StateFun within
Hi Tim!
Indeed the StateFun SDK / StateFun runtime, has an internal concept of
batching, that kicks in the presence of a slow
/congested remote function. Keep in mind that under normal circumstances
batching does not happen (effectively a batch of size 1 will be sent). [1]
This batch is not curren
e:
> "base64bytes"}
> {"type": "DeleteState", "namespace": "foo", "type": "bar", "id": "1"}
>
> Also having opentracing integration where Statefun passes b3 headers with
> each request so we can tr
s://issues.apache.org/jira/browse/FLINK-22389
>> * Support for OpenTracing:
>> https://issues.apache.org/jira/browse/FLINK-22390
>> * Support for State Introspection:
>> https://issues.apache.org/jira/browse/FLINK-22391
>>
>> Please feel free to edit, comment on these iss
Hello Cliff,
You are right, indeed defining custom metrics is not supported at the
moment.
I will file a JIRA issue so we can track this, and we will try to
prioritize this feature up.
Meanwhile, there are a lot of metrics that StateFun defines, like
invocations rates etc' perhaps you can find it
Hello!
Your analysis is correct, indeed what is passed is whatever is being handed
to withMessageBody(..).
Starting with StateFun 3.0, if you need to send a message to a remote
function the message needs to be a TypedValue.
You can create an instance of TypedValue manually, or you can add a
depen
namesIngress)
> .withRequestReplyRemoteFunction(
> RequestReplyFunctionBuilder.requestReplyFunctionBuilder(
> REMOTE_GREET, URI.create(
> "http://localhost:5000/statefun"; <http://localhost:5000/statefun>))
> .wit
Hi Jan,
I haven't stumbled upon this but I will try to reconstruct that scenario
with a stress test and report back.
Can you share a little bit about your environment. For example do you use
gunicorn, ngnix, aiohttp/or flask perhaps?
I'd suggest maybe checking for request size limits parameters
timeouts [1]
[1]
https://ci.apache.org/projects/flink/flink-statefun-docs-release-3.0/docs/deployment/module/#timeouts
On Fri, May 21, 2021 at 10:53 AM Igal Shilman wrote:
> Hi Jan,
>
> I haven't stumbled upon this but I will try to reconstruct that scenario
> with a stress tes
Hi Bonino,
If you've also included the statefun-sdk-java into the class path then,
make sure that you are not using accidentally the shaded version of a
TypedValue (check the package name)
Let me know if that helped, and we'll try to debug this together otherwise.
Cheers,
Igal.
On Mon, May 24,
d this leads to the reported error.
>
> Any idea on how to overcome this issue?
>
> Cheers,
>
> Dario
> On 5/25/21 11:22 AM, Igal Shilman wrote:
>
> Hi Bonino,
>
> If you've also included the statefun-sdk-java into the class path then,
> make sure that yo
it should be fine.
Good luck,
Igal.
On Tue, May 25, 2021 at 12:04 PM Igal Shilman wrote:
> Do you have the statefun-sdk-java as a dependency, if you are not using it
> can you remove it?
>
> On Tue, May 25, 2021 at 11:28 AM Bonino Dario
> wrote:
>
>> Dear Igal,
>>
ines to understand when the dependency is mandatory
> and how to solve conflicts as such in those cases?
>
> Best regards
>
> Dario
> On 5/25/21 12:04 PM, Igal Shilman wrote:
>
> Do you have the statefun-sdk-java as a dependency, if you are not using it
> can you remove it?
Hi Tim,
Any additional logs from before are highly appreciated, this would help us
to trace this issue.
By the way, do you see something in the JobManager's UI?
On Fri, May 28, 2021 at 9:06 AM Tzu-Li (Gordon) Tai
wrote:
> Hi Timothy,
>
> It would indeed be hard to figure this out without any sta
ing option is to attach a
>>> debugger [1]. I'd probably add a breakpoint to
>>> LegacySourceFunctionThread#run and see what happens. If the issue is in
>>> recovery, you should add a breakpoint to StreamTask#beforeInvoke.
>>>
>>> [1]
>>> http
Hello Jessy,
Currently StateFun Kafka ingress interprets the keys of the record as the
destination address.
So you'd have to attach a key to use that specific ingress.
If this is not an option for you, you can consider @Tim's suggestion or
create a JIRA with a feature request,
which we will be ha
Hi Flavio,
We haven't included the UI source code just yet, we've only used it for
demos and talks.
The reason is that (1) we didn't put a lot of effort and time there (2)
didn't check the time to go through the individual dependencies and
licences.
But we will add that very soon.
Would having t
w to filter data received in the topic?
> Should I use a correlation id or use a new topic per user?
>
> Il Gio 31 Ott 2019, 16:08 Igal Shilman ha scritto:
>
>> Hi Flavio,
>>
>> We haven't included the UI source code just yet, we've only used it for
>> de
t topics..in another case I could just
> have too many events also in the single map section (let's think about some
> big city with many user).
>
> Is there any talk at FF ir someone else that faced those issues too?
>
> Il Gio 31 Ott 2019, 17:44 Igal Shilman ha scritto:
&g
Hi Dan,
You can learn more about Flink’s metrics system at [1]
You would be able to either setup a reporter that would export the metrics
to an external system, or query the metrics via the REST API, or simply use
Flink’s web ui to obtain them.
If I understand the second part of your question cor
Hi Dan,
Having a class that defines only the function types indeed makes sense,
this would lower the coupling between the (maven) module that contains the
function implementation and the (maven) module that uses it.
You can peek here for example:
https://github.com/ververica/stateful-functions/bl
Hi Dan,
Yes. You should be able to store “wheels” in “Vehicle”.
Igal.
On Monday, January 6, 2020, Dan Pettersson
wrote:
> Hello again :-)
>
> When using an abstract class should the instance variables be stored in
> PersistenceValues
> to conform to the fault tolerance?
>
> Exempel Car, Truck a
Hi Dan,
The initialization and usage of a function is handled by the runtime by a
single thread.
Can you share more details? For example:
1. Any Stacktrace
2. How does your StatefulFunctionProvider looks like?
Do you cache a single function instance, or return a new one in the
provider ?
3. Are yo
-docs-release-1.13/docs/dev/datastream/fault-tolerance/state/#cleanup-during-rocksdb-compaction
On Wed, Jul 14, 2021 at 1:59 PM Igal Shilman wrote:
> Hi Ammon,
>
> The duration is per item, and the cleanup happens transparently and
> incrementally via RocksDB (background compactions w
Hi Omid,
I'm glad to hear that you are evaluating StateFun in your company! let me
try to answer your questions:
1. In version 2.x, StateFun only supported messages of type
com.google.protobuf.Any, and we had a tiny optimization that
reads type hints and unpacked the real message out of the Any m
Not yet unfortunately,
But I'd be very much happy to work with the community on a JS SDK.
On Tue, Jul 20, 2021 at 4:32 PM Omid Bakhshandeh
wrote:
> Igal,
>
> Thanks for the answers. Is there any JS SDK available?
>
> Best,
> --Omid
>
> On Tue, Jul 20, 2021 at
Hi Deniz,
Sorry for the late reply!
Let me first say that if your use case is a simple enrichment/side-lookup
then perhaps it is simpler to use Flink's Async operator that
calls into your HTTP service. StateFun comes with a programming model, an
SDK, and few limitations (for example only supports
The Apache Flink community is very happy to announce the release of Apache
Flink Stateful Functions (StateFun) 3.1.0.
StateFun is a cross-platform stack for building Stateful Serverless
applications, making it radically simpler to develop scalable, consistent,
and elastic distributed applications.
Hi Barry,
I've forward your email to the user mailing list as it is more suitable
here :-)
Your question definitely makes sense, and let me try to provide you with
some pointers:
1. The architecture that you've outlined has many advantages and is
desirable if you can
afford that. Some of them are
Hello Barry,
I assume that by "we don't need another installation of Flink to manage the
stateful functions." You mean that you already have a running Flink cluster
and you would like to submit an additional Flink Job that executes a
Stateful functions application?
Then perhaps just try to submit
istribution dependency
> were shipped with the required value in it by default?
> Thanks a million,
> Barry
>
> On 2021/09/10 12:37:32, Igal Shilman wrote:
> > Hello Barry,
> >
> > I assume that by "we don't need another installation of Flink to manage
>
ter.
> I was only able to get the code running through IntelliJ by following the
> steps below. Once I reverted the hack and changed the config on our
> cluster, everything worked perfectly.
> Sorry for the confusion and thanks for all your help.
> Barry
>
> On 2021/09/23 10:57:36,
t,
>
> Christian
>
> -Ursprüngliche Nachricht-
> Von: Roman Khachatryan
> Gesendet: Dienstag, 28. September 2021 00:31
> An: Christian Krudewig (Corporate Development) <
> christian.krude...@dpdhl.com>; Igal Shilman
> Cc: user@flink.apache.org
> Betreff
Hello Le,
Currently the only way to execute a Python function with StateFun is
through a remote function.
This means that you need to host the function separately. [1]
Good luck!
Igal
[1]
https://nightlies.apache.org/flink/flink-statefun-docs-master/docs/modules/http-endpoint/
On Thu, Oct 7, 20
Hello Christian,
The challenge with generic back pressure and remote functions, is that
StateFun doesn't know if it targets a single process or a fleet of
processes behind a load balancer and an autoscaler.
Triggering back pressure too early might never kick in the autoscaling.
Indeed that parame
Hi Mark,
For communicating with remote functions we use the default trust manager,
so I believe that if you add the self signed certificate to
the container the JVM will pick it up automatically.
I haven't done it myself, but I've found this blog post that explains how
to do it [1]
Let me know if
Forgot to include the user mailing list in my previous email.
On Fri, Oct 15, 2021 at 12:27 PM Igal Shilman wrote:
> Hello,
>
> Is there a specific reason you are using the 2.x branch? This is quite old
> and most importantly it is not compatible with the 3.x branch.
> If you are
Hello Fil,
Indeed what you are describing is exactly what a remote function does.
I am curious to learn more about the current performance limitations that
you encounter with the remote functions.
One thing to try in combination with the async transport, is to increase
the total number of in fig
ay to achieve this
>> 'sequencial-per-key' behaviour with the use of embedded functions? Those
>> seem to be rock-solid. Maybe there are some internal classes that would at
>> least provide a template on how to do it? I have a naive implementation
>> ready (the one I
Hello Hady,
Glad to see that you are testing StateFun!
Regarding that exception, I think that this is not the root cause. The root
cause is as you wrote that the StateFun job failed because it wasn't able
to deliver a message to a remote function in the given time frame.
If you look at the logs yo
Hi Fil,
The default Kafka ingress that ships with StateFun indeed doesn't bundle
the headers with the incoming message, so there is no way of getting them
at the moment, without doing some work :(
I'd be also happy to kick off the discussion (I guess JIRA would be the
right place) about supporting
Hello Mingmin,
Your described scenario and use case indeed seems like a good fit for
StateFun.
Also your analysis is correct, StateFun is executed as a very specific
Flink application (this is what you call "state engine"), and remote
functions are effectively a stateless service.
* when a host i
Hi Deniz,
StateFun would be looking for module.yaml(s) in the classpath.
If you are submitting the job to an existing Flink cluster this really
means that it needs to be either:
1. packaged with the jar (like you are already doing)
2. be present at the classpath, this means that you can place your
Hi Deniz,
My apologies for the late reply, I assume that by now you have figured this
out since I've seen your followup question :-)
StateFun uses the trust store configured in the JVM, so if you can install
your certificate there, StateFun should transparently pick it up.
Good luck,
Igal.
On Fr
Hello,
Glad to hear that you've successfully deployed a remote function with
StateFun :-)
It's not clear to me if the only way to deploy a custom egress is to use
> and embedded module because documentation states:
Indeed currently the only way to define a custom egress is by writing an
embedde
; > > ===
> > >
> > > Therefore, it's easier to use single jar in our pipelines instead of
> > > creating a different jar file for each env. (at least for development
> > > and production).
> > >
> > > For solution 2, you refer fli
1 - 100 of 118 matches
Mail list logo
