Re: Using redis cache in flink

2020-01-12 Thread Yun Tang
Hi Navneeth You cannot easily create single specific instance per TM since Flink would not allow user defined object binned with the life cycle of task manager. However, you can ensure all the operators of the same class could share some single object when initializing operators. You could use

Re: Fail to deploy flink on k8s in minikube

2020-01-12 Thread 刘建刚
Thank you for your help. Yang Wang 于2020年1月13日周一 下午12:53写道: > Hi, Jiangang > > Glad to hear that you are looking to run Flink on Kubernetes. > > It just because you are using the new Kubernetes version.The > extensions/v1beta1 > has been removed since v1.16. Please use apps/v1 instead. The apps/

Re: Fail to deploy flink on k8s in minikube

2020-01-12 Thread Yang Wang
Hi, Jiangang Glad to hear that you are looking to run Flink on Kubernetes. It just because you are using the new Kubernetes version.The extensions/v1beta1 has been removed since v1.16. Please use apps/v1 instead. The apps/v1 is introduced from v1.9.0. I will create a ticket fix the documentation.

Fail to deploy flink on k8s in minikube

2020-01-12 Thread 刘建刚
I fail to deploy flink on k8s referring to https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/kubernetes.html When I run the command 'kubectl create -f jobmanager-deployment.yaml', following error is reported: [image: image.png] I am new to k8s. Our team want to

Re: Using redis cache in flink

2020-01-12 Thread Navneeth Krishnan
Hi Yun, Thanks for the update. I can definitely use a redis cluster but what I don't understand is if I use a custom operator then redis cache will instantiated per operator instance. What I would like to ideally have is one redis cache instance per TM JVM. Since there isn't anyway to share data b

Re: Apache Flink - Sharing state in processors

2020-01-12 Thread Yun Tang
Hi Mans What's the meaning of 'processor' you defined here? A user defined function? When talking about share state, I'm afraid it's not so easy to implement in Flink. As no matter keyed state or operator state, they're both instantiated, used and only thread-safe in operator scope. The only wa

Re:Re: Null result cannot be used for atomic types

2020-01-12 Thread sunfulin
Hi, Thanks for the reply. Tends out that I am using table2datastream and tableEnv.sqlUpdate in the seem time and the exception thus is thrown. My mistake. At 2020-01-10 17:11:02, "Jingsong Li" wrote: Hi sunfulin, Looks like the error is happened in sink instead of source. Caused b

Re: Please suggest helpful tools

2020-01-12 Thread Kurt Young
Hi, You can try to filter NULL values with an explicit condition like " is not NULL". Best, Kurt On Sat, Jan 11, 2020 at 4:10 AM Eva Eva wrote: > Thank you both for the suggestions. > I did a bit more analysis using UI and identified at least one > problem that's occurring with the job rn

[ANNOUNCE] Weekly Community Update 2020/02

2020-01-12 Thread Konstantin Knauf
Dear community, happy new year from my side, too, and thanks a lot to Hequn for helping out with the weekly updates during the last three weeks! I enjoyed reading these myself for a change. This week's community digest features an update on Flink 1.10 release testing, a proposal for a SQL catalog

Re: Yarn Kerberos issue

2020-01-12 Thread Rong Rong
Hi Juan, I have some time to dig deeper into the code, It seems like the HADOOP_TOKEN_FILE_LOCATION is actually a static environment variable field that the UserGroupInformation will read. Interestingly Flink's Hadoop security module actually treats it differently depending on whether this is set,

Re: are blink changes merged into flink 1.9?

2020-01-12 Thread Benchao Li
Sorry for the missing link. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/streaming/temporal_tables.html#temporal-table Benchao Li 于2020年1月12日周日 下午9:06写道: > Hi kant, > > In my understanding, "treat batch as a special case of streaming" just > means runtime implementat

Re: are blink changes merged into flink 1.9?

2020-01-12 Thread Benchao Li
Hi kant, In my understanding, "treat batch as a special case of streaming" just means runtime implementation. So currently, we cannot consume a database and it's changes as a stream. But You can put the change log into massage queue system like Kafka using a CDC system like Maxwell or Canal. Then

Re: Initialization of a Stateful function

2020-01-12 Thread Dan Pettersson
Hi Igal, If you have the time, the easiest way to troubleshoot and recreate this race condition is to run class FlowTest in repo: https://github.com/danp11/stateful-functions (I've upgraded to Java11) I've created a "stockmarket-example" under examples/ directory and the test is running via Harn

Re: are blink changes merged into flink 1.9?

2020-01-12 Thread kant kodali
Hi, Thanks for the response! The Documentation says Blink will treat batch as a special case of streaming. if so, can I stream from a database? or put it in another way. If I have two streams A & B where Stream A is writing to Database and Stream B reading from the database. will stream B keeps ru

Re: Initialization of a Stateful function

2020-01-12 Thread Igal Shilman
Hi Dan, The initialization and usage of a function is handled by the runtime by a single thread. Can you share more details? For example: 1. Any Stacktrace 2. How does your StatefulFunctionProvider looks like? Do you cache a single function instance, or return a new one in the provider ? 3. Are yo