Re: Is possible that make two operators always locate in same taskmanager?

2020-11-15 Thread Si-li Liu
Thanks, I'll try it. Matthias Pohl 于2020年11月14日周六 上午12:53写道: > Hi Si-li, > trying to answer your initial question: Theoretically, you could try using > the co-location constraints to achieve this. But keep in mind that this > might lead to multiple Join operators running in the same JVM reducing

Re:Re: Flink 1.11 not showing logs

2020-11-15 Thread 马阳阳
Hi Yang, We run a self-compiled Flink-1.12-SNAPSHOT, and could not see any taskmanager/jobmanager logs. I have checked the log4j.properties file, and it's in the right format. And the FLINK_CONF_DIR is set. When checking the java dynamic options of task manager, I found that the log related opti

Re: Adding keyed state to test harness before calling process function.

2020-11-15 Thread Guowei Ma
Hi, Macro I do not find the direct way for doing this.(Maybe other guys know) A possible way might be 1. Build the expected keyed state and get the `OperatorSubtaskState` from an `xxOperatorTestHarness`. 2. Use the `OperatorSubtaskState` to initialize the `xxOperatorTestHarness` that is needed to

Re: Flink AutoScaling EMR

2020-11-15 Thread Xintong Song
> > Is there a way to make the new yarn job only on the new hardware? I think you can simply decommission the nodes from Yarn, so that new containers will not be allocated from those nodes. You might also need a large decommission timeout, upon which all the remaining running contains on the decom

Re: Hi all I'm having trouble with spinning up native Kubernetes cluster

2020-11-15 Thread Yang Wang
Since the fabric8 Kubernetes client does not build "org.bouncycastle" dependencies in the main jar, maybe we need to have the them in the classpath. Could you also put the bcpkix-jdk15on-1.67.jar in the client image, not only container(aka application) image? I think you need to build you own Flin

Dynamic ad hoc query deployment strategy

2020-11-15 Thread lalala
Hi all, I would like to consult with you regarding deployment strategies. We have +250 Kafka topics that we want users of the platform to submit SQL queries that will run indefinitely. We have a query parsers to extract topic names from user queries, and the application locally creates Kafka tabl