Sink metric numRecordsIn drops temporarily

Hi there, I witnessed an interesting behaviour on my Grafana dashboard where sink related metrics would show activity where there should not be any. I am saying this because for this particular sink activity is triggered by data being ingested through a cronjob at a particular time, however the das

Re: Sink metric numRecordsIn drops temporarily

Missed to cc to user@flink.apache.org Hi, > > Just wondering what is the value of that counter (wo/ applying > NonNegativeDerivative > function) when you observe the spikes? If I remember correctly, Grafana > is known to aggregate those values by averaging them across the time > duration selected

Re: Sink metric numRecordsIn drops temporarily

Hi, thank you for your answer. So for September 11th which is shown on the screenshot I had the counter sitting at 26.91k where when the drop happened it was going down to 26.01k. This happened 3 times during that day and it was always going back to the same value. Philipp -- Sent from: http://a

ARIMA Model

is there a ARIMA(Autoregressive integrated moving average ) module available in Flink machine learning?

Re: ETL with changing reference data

Hi Peter, If I understand correctly, I think you are facing a delima of having efficient dynamic referencing as well as salable processing. I don't have answer to how thing would work for your specific case. Yet this is just interesting topic to discuss. Fabian provides insights and I would like

Re: Sink metric numRecordsIn drops temporarily

Hi, Thank you for your description. What I tried to understand is what the counter value is at that moment of spikes. Grafana would take the average out of a continuous data values before rendering result to UI. That is, if the metrics value is not transmitted continuously, where at some data poin

Re: Queryable State

No, it doesn't work even if I increase the timeout. The state being fetched is a Map of data and has around 100 entries in it. I have a single job manager and 3 task managers with 16 slots each running on AWS EC2. final TypeSerializer keySerializer = TypeInformation.of(new TypeHint() {}).

need instruction on how the Flink metric works

I'm new to flink and I have read https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html, I am still unclear where do I read the metrics I added. for example, public static void main(String[] args) throws Exception { StreamExecutionEnvironment env = StreamExe

Re: need instruction on how the Flink metric works

Hi, You may enable metrics reporter to see the output of your metrics; counter in your example. There is a brief documentation regarding to metrics and reporter setup at link . The easiest approach, in my opinion

Can't send data to another service in addSink

Hi In my flink program, I want to send data to a server via MQTT protocol, here is my code https://gist.github.com/duytruong/d240958d2c4140b554b604cbef05edc7 The problem is it ran well on IntelliJ, but when submitted to cluster (bin/start-local.sh), it could connect but failed to send data. When

Re: need instruction on how the Flink metric works

Thanks, When I started jconsole, it listed com.apache.flink.runtime.jobmanager..:[port] as one of the Local Process, i was able to connect to it with insecure connection, but i was not able to locate the Counter metrics, I only saw some system metrics. On Sun, Sep 17, 2017 at 7:39 PM, Michael Fong