We use Flink to process transactional events. A job was created to aggregate
information about the clients, day of week and hour of day and thus creating a
profile as shown in the attached code.
val stream = env.addSource(consumer)
val result = stream
.map(openTransaction => {
val transac
Stephan Ewen created FLINK-9292:
---
Summary: Remove TypeInfoParser
Key: FLINK-9292
URL: https://issues.apache.org/jira/browse/FLINK-9292
Project: Flink
Issue Type: Task
Components: Core
shuai.xu created FLINK-9293:
---
Summary: SlotPool should check slot id when accepting a slot offer
with existing allocation id
Key: FLINK-9293
URL: https://issues.apache.org/jira/browse/FLINK-9293
Project: Fl
Rong Rong created FLINK-9294:
Summary: Improve type inference for UDFs with composite parameter
or result type
Key: FLINK-9294
URL: https://issues.apache.org/jira/browse/FLINK-9294
Project: Flink
Christopher Ng created FLINK-9295:
-
Summary: FlinkKafkaProducer011 sometimes throws
ProducerFencedExceptions when in EXACTLY_ONCE mode
Key: FLINK-9295
URL: https://issues.apache.org/jira/browse/FLINK-9295
Currently in the MemCheckpointStreamFactory.checkSize() method, it can throw an
IOException with:
throw new IOException(
"Size of the state is larger than the
maximum permitted memory-backed state. Size="
Agree it is definitely not intuitive trying to figure out what to do based
on this message.
I think the message should be changed to "please consider increasing
maximum permitted memory size, increase task manager parallelism, or using
a non-memory-based state backend".
Could you please open a JIR
Hi Gabriel,
Yes, using RocksDB state backend can relieve your RAM usage. I see a few
issues with your job: 1) it's keeping track of 672 windows (28x24), that's
lots of data, so try to reduce number of windows 2) use reduce functions to
incrementally aggregate state, rather than buffering data inte
Agree with Bowen on this note: you should probably use some more efficient
way of handling the data in sliding window, since data will be "assigned"
to each sliding window through a window assigner and thus costs extra
memory usage.
BTW: since we are on this topic, I was wondering if there's any w
