Hi Ovidiu,
the parameters to configure the amount of managed memory
(taskmanager.memory.size,
taskmanager.memory.fraction) are valid for on and off-heap memory.
Have you tried these parameters and didn't they work as expected?
Best, Fabian
2016-03-16 11:43 GMT+01:00 Ovidiu-Cristian MARCU <
ovi
Hi,
Is it possible to add a parameter off-heap.size for the task manager off-heap
memory [1]?
It is not possible to limit the off-heap memory size, at least I found nothing
in the documentation.
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#managed-memory
Hi Radu,
the mapping which StreamOperator is executed by which StreamTask happens
first in the StreamGraph.addOperator method. However, there is a second
step in the StreamingJobGraphGenerator.createChain where chainable
operators are chained and then executed by a single StreamTask. The
construct
Hi Max,
thanks for the tips. What we did has been running kinit on each node with
the same user that then went on running the start-cluster.sh script. Right
now the LDAP groups are backed by the OS ones and the user that ran the
launch script is part of the flink group, that is on every node of th
Hi Stefano,
You have probably seen
https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#kerberos
?
Currently, all nodes need to be authenticated with the Kerberos before
Flink is started (not just the JobManager). Could it be that the
start-cluster.sh script actually is n
Sorry for joining this discussion late. Maybe this is also interesting for
you:
http://www.confluent.io/blog/bottled-water-real-time-integration-of-postgresql-and-kafka/
On Wed, Mar 9, 2016 at 1:47 PM, Prez Cannady
wrote:
> Thanks. Need to dive in a bit better, but I did clarify some things in
Hi Mengqi,
I did not completely understand your use case.
If you would like to use a composite key (a key with multiple fields) there
are two alternatives:
- use a tuple as key type. This only works if all records have the same
number of key fields. Tuple serialization and comparisons are very
ef
Hi Ovidiu,
putting the CompactingHashTable aside, all data structures and algorithms
that use managed memory can spill to disk if data exceeds memory capacity.
It was a conscious choice to not let the CompactingHashTable spill. Once
the solution set hash table is spilled, (parts of) the hash tabl
Hi,
Regarding the solution set going out of memory, I would like an issue to be
filled against it.
Looking into code for CompactingHashTable I see
The hash table is internally divided into two parts: The hash index, and the
partition buffers that store the actual records. When records are inse
