Re: [DISCUSS] Semantic and implementation of per-job mode

2019-11-04 Thread tison
> >>> Besides, by adding an compiling option, we need to consider more things >>> when submitting a job like "Is my program including multiple job?" or "Does >>> the program need to be initialized before submitting to a remote cluster?", >>> which l

Re: [DISCUSS] Semantic and implementation of per-job mode

2019-10-31 Thread tison
l. We heavily use multiple >>> job graphs to get isolation around retry logic and resource allocation >>> across the job graphs. Putting all these parallel flows into a single graph >>> would mean sharing of TaskManagers across what was meant to be truly >>> independent. >>> >>

Re: [DISCUSS] Semantic and implementation of per-job mode

2019-10-31 Thread Peter Huang
tioned in the mailing list or maybe we need to > reconsider a better concept and definition which is easy to understand. > > > > Best, > > Jiayi Liao > > Original Message > *Sender:* Rong Rong > *Recipient:* Regina" > *Cc:* Theo Diefenthal; > u...@flink.apache.org &

Re: [DISCUSS] Semantic and implementation of per-job mode

2019-10-31 Thread bupt_ljy
and we have found flink (as os 1.3/1.6.4) cannot handle graphs of that size. We’re currently testing 1.9.1 but have not retested the large graph scenario. Billy From: Paul Lam [mailto:paullin3...@gmail.com] Sent: Wednesday, October 30, 2019 8:41 AM To: SHI Xiaogang Cc: tison; dev; user

Re: [DISCUSS] Semantic and implementation of per-job mode

2019-10-31 Thread Flavio Pompermaier
s in it, we do a job graph per table rather than a >>> single job graph that does all tables instead. A single job graph can be in >>> the tens of thousands of nodes in our largest cases and we have found flink >>> (as os 1.3/1.6.4) cannot handle graphs of that size. We’r

Re: [DISCUSS] Semantic and implementation of per-job mode

2019-10-31 Thread Yang Wang
y >> >> >> >> >> >> *From:* Paul Lam [mailto:paullin3...@gmail.com] >> *Sent:* Wednesday, October 30, 2019 8:41 AM >> *To:* SHI Xiaogang >> *Cc:* tison; dev; user >> *Subject:* Re: [DISCUSS] Semantic and implementation of per-job mode >> >&g

Re: [DISCUSS] Semantic and implementation of per-job mode

2019-10-31 Thread Gyula Fóra
d flink >> (as os 1.3/1.6.4) cannot handle graphs of that size. We’re currently >> testing 1.9.1 but have not retested the large graph scenario. >> >> >> >> Billy >> >> >> >> >> >> *From:* Paul Lam [mailto:paullin3...@gmail.com] >> *Sent:*

Re: [DISCUSS] Semantic and implementation of per-job mode

2019-10-30 Thread tison
gang > *Cc:* tison; dev; user > *Subject:* Re: [DISCUSS] Semantic and implementation of per-job mode > > > > Hi, > > > > Thanks for starting the discussion. > > > > WRT the per-job semantic, it looks natural to me that per-job means > per-job-graph, > >

Re: [DISCUSS] Semantic and implementation of per-job mode

2019-10-30 Thread Paul Lam
Hi, Thanks for starting the discussion. WRT the per-job semantic, it looks natural to me that per-job means per-job-graph, because in my understanding JobGraph is the representation of a job. Could you share some use case in which a user program should contain multiple job graphs? WRT the per-

Re: [DISCUSS] Semantic and implementation of per-job mode

2019-10-30 Thread SHI Xiaogang
Hi Thanks for bringing this. The design looks very nice to me in that 1. In the new per-job mode, we don't need to compile user programs in the client and can directly run user programs with user jars. That way, it's easier for resource isolation in multi-tenant platforms and is much safer. 2. Th