Hi,
If you didn't configure your program to use RocksDB then you're already not
using RocksDB. I think the main issue, as others have pointed out, is that by
keying on a constant key you're essentially turning your program into a
parallelism-of-1 program, thereby wasting almost all cluster reso
Is there a way to not go between RocksDB? For this test application it
seems not necessary as we don't expect fault tolerance and this is an
streaming case.
Thanks,
Supun..
On Thu, Mar 1, 2018 at 11:55 AM, Stephan Ewen wrote:
> Few quick checks:
>
> - Do you properly set the parallelism?
>
Few quick checks:
- Do you properly set the parallelism?
- If you start 640 tasks (parallelism), and you use the same key for
everything, that behaves like parallelism 1 (Piotr mentioned this)
- Do you use the RocksDB state backend? If yes, try the FsStateBackend.
It looks like your state d
Yes, the program runs fine, I can see it on the UI. Sorry, didn't include
the part where the execute is called.
Thanks,
Supun..
On Thu, Mar 1, 2018 at 10:27 AM, Fabian Hueske wrote:
> Are you sure the program is doing anything at all?
> Do you call execute() on the StreamExecutionEnvironment?
>
Are you sure the program is doing anything at all?
Do you call execute() on the StreamExecutionEnvironment?
2018-03-01 15:55 GMT+01:00 Supun Kamburugamuve :
> Thanks Piotrek,
>
> I did it this way on purpose to see how Flink performs. With 128000
> messages it takes an un-reasonable amount of tim
Thanks Piotrek,
I did it this way on purpose to see how Flink performs. With 128000
messages it takes an un-reasonable amount of time for Flink to complete the
operation. With another framework the same operation completes in about 70
seconds for 1000 messages of size 128000, while Flink takes hou
Hi,
First of all learn about what’s going with your job: check the status of the
machines, cpu/network usage on the cluster. If CPU is not ~100%, analyse what
is preventing the machines to work faster (network bottleneck, locking,
blocking operations etc). If CPU is ~100%, profile the TaskManag
Hi,
I'm trying to run a simple benchmark on Flink streaming reduce. It seems it
is very slow. Could you let me know if I'm doing something wrong.
Here is the program. I'm running this on 32 nodes with 20 tasks in each
node. So the parallelism is at 640.
public class StreamingReduce {
int size;
