[jira] [Created] (FLINK-33174) enabling tablesample bernoulli in flink
xiaogang zhou created FLINK-33174: - Summary: enabling tablesample bernoulli in flink Key: FLINK-33174 URL: https://issues.apache.org/jira/browse/FLINK-33174 Project: Flink Issue Type: Improvement Components: Table SQL / Planner Affects Versions: 1.17.1 Reporter: xiaogang zhou I'd like to introduce a table sample function to enable fast sampling to streamings. this is enlighted by https://issues.apache.org/jira/browse/CALCITE-5971 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] FLIP-370 : Support Balanced Tasks Scheduling
Hi Yangze, Thanks for your feedback! > 1. Is it possible for the SlotPool to get the slot allocation results > from the SlotManager in advance instead of waiting for the actual > physical slots to be registered, and perform pre-allocation? The > benefit of doing this is to make the task deployment process smoother, > especially when there are a large number of tasks in the job. Could you elaborate on that? I didn't understand what's the benefit and smoother. > 2. If user enable the cluster.evenly-spread-out-slots, the issue in > example 2 of section 2.2.3 can be resolved. Do I understand it > correctly? The example assigned result is the final allocation result when flink user enables the cluster.evenly-spread-out-slots. We think the assigned result is expected, so I think your understanding is right. Best, Rui On Thu, Sep 28, 2023 at 1:10 PM Shammon FY wrote: > Thanks Yuepeng for initiating this discussion. > > +1 in general too, in fact we have implemented a similar mechanism > internally to ensure a balanced allocation of tasks to slots, it works > well. > > Some comments about the mechanism > > 1. This mechanism will be only supported in `SlotPool` or both `SlotPool` > and `DeclarativeSlotPool`? Currently the two slot pools are used in > different schedulers. I think this will also bring value to > `DeclarativeSlotPool`, but currently FLIP content seems to be based on > `SlotPool`, right? > > 2. In fine-grained resource management, we can set different resource > requirements for different nodes, which means that the resources of each > slot are different. What should be done when the slot selected by the > round-robin strategy cannot meet the resource requirements? Will this lead > to the failure of the balance strategy? > > 3. Is the assignment of tasks to slots balanced based on region or job > level? When multiple TMs fail over, will it cause the balancing strategy to > fail or even worse? What is the current processing strategy? > > For Zhuzhu and Rui: > > IIUC, the overall balance is divided into two parts: slot to TM and task to > slot. > 1. Slot to TM is guaranteed by SlotManager in ResourceManager > 2. Task to slot is guaranteed by the slot pool in JM > > These two are completely independent, what are the benefits of unifying > these two into one option? Also, do we want to share the same > option between SlotPool in JM and SlotManager in RM? This sounds a bit > strange. > > Best, > Shammon FY > > > > On Thu, Sep 28, 2023 at 12:08 PM Rui Fan <1996fan...@gmail.com> wrote: > > > Hi Zhu Zhu, > > > > Thanks for your feedback here! > > > > You are right, user needs to set 2 options: > > - cluster.evenly-spread-out-slots=true > > - slot.sharing-strategy=TASK_BALANCED_PREFERRED > > > > Update it to one option is useful at user side, so > > `taskmanager.load-balance.mode` sounds good to me. > > I want to check some points and behaviors about this option: > > > > 1. The default value is None, right? > > 2. When it's set to Tasks, how to assign slots to TM? > > - Option1: It's just check task number > > - Option2: It''s check the slot number first, then check the > > task number when the slot number is the same. > > > > Giving an example to explain what's the difference between them: > > > > - A session cluster has 2 flink jobs, they are jobA and jobB > > - Each TM has 4 slots. > > - The task number of one slot of jobA is 3 > > - The task number of one slot of jobB is 1 > > - We have 2 TaskManagers: > > - tm1 runs 3 slots of jobB, so tm1 runs 3 tasks > > - tm2 runs 1 slot of jobA, and 1 slot of jobB, so tm2 runs 4 tasks. > > > > Now, we need to run a new slot, which tm should offer it? > > - Option1: If we just check the task number, the tm1 is better. > > - Option2: If we check the slot number first, and then check task, the > tm2 > > is better > > > > The original FLIP selected option2, that's why we didn't add the > > third option. The option2 didn't break the semantics when > > `cluster.evenly-spread-out-slots` is true, and it just improve the > > behavior without the semantics is changed. > > > > In the other hands, if we choose option2, when user set > > `taskmanager.load-balance.mode` is Tasks. It also can achieve > > the goal when it's Slots. > > > > So I think the `Slots` enum isn't needed if we choose option2. > > Of course, If we choose the option1, the enum is needed. > > > > Looking forward to your feedback, thanks~ > > > > Best, > > Rui > > > > On Wed, Sep 27, 2023 at 9:11 PM Zhu Zhu wrote: > > > > > Thanks Yuepeng and Rui for creating this FLIP. > > > > > > +1 in general > > > The idea is straight forward: best-effort gather all the slot requests > > > and offered slots to form an overview before assigning slots, trying to > > > balance the loads of task managers when assigning slots. > > > > > > I have one comment regarding the configuration for ease of use: > > > > > > IIUC, this FLIP uses an existing config > 'cluster.evenly-spread-out-slots' > > > as the main switch of t
Re: [DISCUSS] FLIP-370 : Support Balanced Tasks Scheduling
Hi Shammon, Thanks for your feedback as well! > IIUC, the overall balance is divided into two parts: slot to TM and task to slot. > 1. Slot to TM is guaranteed by SlotManager in ResourceManager > 2. Task to slot is guaranteed by the slot pool in JM > > These two are completely independent, what are the benefits of unifying > these two into one option? Also, do we want to share the same > option between SlotPool in JM and SlotManager in RM? This sounds a bit > strange. Your understanding is totally right, the balance needs 2 parts: slot to TM and task to slot. As I understand, the following are benefits of unifying them into one option: - Flink users don't care about these principles inside of flink, they don't know these 2 parts. - If flink provides 2 options, flink users need to set 2 options for their job. - If one option is missed, the final result may not be good. (Users may have questions when using) - If flink just provides 1 option, enabling one option is enough. (Reduce the probability of misconfiguration) Also, Flink’s options are user-oriented. Each option represents a switch or parameter of a feature. A feature may be composed of multiple components inside Flink. It might be better to keep only one switch per feature. Actually, the cluster.evenly-spread-out-slots option is used between SlotPool in JM and SlotManager in RM. 2 components to ensure this feature works well. Please correct me if my understanding is wrong, and looking forward to your feedback, thanks! Best, Rui On Sun, Oct 1, 2023 at 5:52 PM Rui Fan <1996fan...@gmail.com> wrote: > Hi Yangze, > > Thanks for your feedback! > > > 1. Is it possible for the SlotPool to get the slot allocation results > > from the SlotManager in advance instead of waiting for the actual > > physical slots to be registered, and perform pre-allocation? The > > benefit of doing this is to make the task deployment process smoother, > > especially when there are a large number of tasks in the job. > > Could you elaborate on that? I didn't understand what's the benefit and > smoother. > > > 2. If user enable the cluster.evenly-spread-out-slots, the issue in > > example 2 of section 2.2.3 can be resolved. Do I understand it > > correctly? > > The example assigned result is the final allocation result when flink > user enables the cluster.evenly-spread-out-slots. We think the > assigned result is expected, so I think your understanding is right. > > Best, > Rui > > On Thu, Sep 28, 2023 at 1:10 PM Shammon FY wrote: > >> Thanks Yuepeng for initiating this discussion. >> >> +1 in general too, in fact we have implemented a similar mechanism >> internally to ensure a balanced allocation of tasks to slots, it works >> well. >> >> Some comments about the mechanism >> >> 1. This mechanism will be only supported in `SlotPool` or both `SlotPool` >> and `DeclarativeSlotPool`? Currently the two slot pools are used in >> different schedulers. I think this will also bring value to >> `DeclarativeSlotPool`, but currently FLIP content seems to be based on >> `SlotPool`, right? >> >> 2. In fine-grained resource management, we can set different resource >> requirements for different nodes, which means that the resources of each >> slot are different. What should be done when the slot selected by the >> round-robin strategy cannot meet the resource requirements? Will this lead >> to the failure of the balance strategy? >> >> 3. Is the assignment of tasks to slots balanced based on region or job >> level? When multiple TMs fail over, will it cause the balancing strategy >> to >> fail or even worse? What is the current processing strategy? >> >> For Zhuzhu and Rui: >> >> IIUC, the overall balance is divided into two parts: slot to TM and task >> to >> slot. >> 1. Slot to TM is guaranteed by SlotManager in ResourceManager >> 2. Task to slot is guaranteed by the slot pool in JM >> >> These two are completely independent, what are the benefits of unifying >> these two into one option? Also, do we want to share the same >> option between SlotPool in JM and SlotManager in RM? This sounds a bit >> strange. >> >> Best, >> Shammon FY >> >> >> >> On Thu, Sep 28, 2023 at 12:08 PM Rui Fan <1996fan...@gmail.com> wrote: >> >> > Hi Zhu Zhu, >> > >> > Thanks for your feedback here! >> > >> > You are right, user needs to set 2 options: >> > - cluster.evenly-spread-out-slots=true >> > - slot.sharing-strategy=TASK_BALANCED_PREFERRED >> > >> > Update it to one option is useful at user side, so >> > `taskmanager.load-balance.mode` sounds good to me. >> > I want to check some points and behaviors about this option: >> > >> > 1. The default value is None, right? >> > 2. When it's set to Tasks, how to assign slots to TM? >> > - Option1: It's just check task number >> > - Option2: It''s check the slot number first, then check the >> > task number when the slot number is the same. >> > >> > Giving an example to explain what's the difference between them: >> > >> > - A session cluster has 2 flink jo
Re: [DISCUSS] FLIP-370 : Support Balanced Tasks Scheduling
Hi, Rui, 1. With the current mechanism, when physical slots are offered from TM, the JobMaster will start deploying tasks and synchronizing their states. With the addition of the waiting mechanism, IIUC, the JobMaster will deploy and synchronize the states of all tasks only after all resources are available. The task deployment and state synchronization both occupy the JobMaster's RPC main thread. In complex jobs with a lot of tasks, this waiting mechanism may increase the pressure on the JobMaster and increase the end-to-end job deployment time. 2. From my understanding, if user enable the cluster.evenly-spread-out-slots, LeastUtilizationResourceMatchingStrategy will be used to determine the slot distribution and the slot allocation in the three TM will be (taskmanager.numberOfTaskSlots=3): TM1: 3 slot TM2: 2 slot TM3: 2 slot Best, Yangze Guo On Sun, Oct 1, 2023 at 6:14 PM Rui Fan <1996fan...@gmail.com> wrote: > > Hi Shammon, > > Thanks for your feedback as well! > > > IIUC, the overall balance is divided into two parts: slot to TM and task > to slot. > > 1. Slot to TM is guaranteed by SlotManager in ResourceManager > > 2. Task to slot is guaranteed by the slot pool in JM > > > > These two are completely independent, what are the benefits of unifying > > these two into one option? Also, do we want to share the same > > option between SlotPool in JM and SlotManager in RM? This sounds a bit > > strange. > > Your understanding is totally right, the balance needs 2 parts: slot to TM > and task to slot. > > As I understand, the following are benefits of unifying them into one > option: > > - Flink users don't care about these principles inside of flink, they don't > know these 2 parts. > - If flink provides 2 options, flink users need to set 2 options for their > job. > - If one option is missed, the final result may not be good. (Users may > have questions when using) > - If flink just provides 1 option, enabling one option is enough. (Reduce > the probability of misconfiguration) > > Also, Flink’s options are user-oriented. Each option represents a switch or > parameter of a feature. > A feature may be composed of multiple components inside Flink. > It might be better to keep only one switch per feature. > > Actually, the cluster.evenly-spread-out-slots option is used between > SlotPool in JM and SlotManager in RM. 2 components to ensure > this feature works well. > > Please correct me if my understanding is wrong, > and looking forward to your feedback, thanks! > > Best, > Rui > > On Sun, Oct 1, 2023 at 5:52 PM Rui Fan <1996fan...@gmail.com> wrote: > > > Hi Yangze, > > > > Thanks for your feedback! > > > > > 1. Is it possible for the SlotPool to get the slot allocation results > > > from the SlotManager in advance instead of waiting for the actual > > > physical slots to be registered, and perform pre-allocation? The > > > benefit of doing this is to make the task deployment process smoother, > > > especially when there are a large number of tasks in the job. > > > > Could you elaborate on that? I didn't understand what's the benefit and > > smoother. > > > > > 2. If user enable the cluster.evenly-spread-out-slots, the issue in > > > example 2 of section 2.2.3 can be resolved. Do I understand it > > > correctly? > > > > The example assigned result is the final allocation result when flink > > user enables the cluster.evenly-spread-out-slots. We think the > > assigned result is expected, so I think your understanding is right. > > > > Best, > > Rui > > > > On Thu, Sep 28, 2023 at 1:10 PM Shammon FY wrote: > > > >> Thanks Yuepeng for initiating this discussion. > >> > >> +1 in general too, in fact we have implemented a similar mechanism > >> internally to ensure a balanced allocation of tasks to slots, it works > >> well. > >> > >> Some comments about the mechanism > >> > >> 1. This mechanism will be only supported in `SlotPool` or both `SlotPool` > >> and `DeclarativeSlotPool`? Currently the two slot pools are used in > >> different schedulers. I think this will also bring value to > >> `DeclarativeSlotPool`, but currently FLIP content seems to be based on > >> `SlotPool`, right? > >> > >> 2. In fine-grained resource management, we can set different resource > >> requirements for different nodes, which means that the resources of each > >> slot are different. What should be done when the slot selected by the > >> round-robin strategy cannot meet the resource requirements? Will this lead > >> to the failure of the balance strategy? > >> > >> 3. Is the assignment of tasks to slots balanced based on region or job > >> level? When multiple TMs fail over, will it cause the balancing strategy > >> to > >> fail or even worse? What is the current processing strategy? > >> > >> For Zhuzhu and Rui: > >> > >> IIUC, the overall balance is divided into two parts: slot to TM and task > >> to > >> slot. > >> 1. Slot to TM is guaranteed by SlotManager in ResourceManager > >> 2. Task to slot is guaranteed by the slot po
[jira] [Created] (FLINK-33175) Nightly builds from S3 are not available for download, breaking all connector tests
Martijn Visser created FLINK-33175:
--
Summary: Nightly builds from S3 are not available for download,
breaking all connector tests
Key: FLINK-33175
URL: https://issues.apache.org/jira/browse/FLINK-33175
Project: Flink
Issue Type: Bug
Components: Connectors / Common
Reporter: Martijn Visser
All downloads of Flink binaries fail with:
{code:java}
Run wget -q -c
https://s3.amazonaws.com/flink-nightly/flink-1.18-SNAPSHOT-bin-scala_2.12.tgz
-O - | tar -xz
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
Error: Process completed with exit code 2.
{code}
This goes for 1.18, but also 1.17 and 1.16
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
