Re: Running multiple streaming jobs in same cluster

2017-02-12 Thread Tzu-Li (Gordon) Tai
Hi Ozan, From your description, it seems like your original huge job can be broken down into smaller disconnected graphs, with only some of the graphs requiring checkpointing / snapshots. In general, it would be a good practice to split disconnected graphs of the execution graph into multiple j

Running multiple streaming jobs in same cluster

2017-02-10 Thread Ozan DENÄ°Z
Hi everyone, We have a huge execution graph for one streaming job. To update this execution graph, we take the snapshot of the job and start the job with snapshot. However this can take too much time. One option is splitting this huge streaming job into smaller ones. We can cancel or run new