Hi All, Since it has turned out that DTM can't be added as member of JobMaster <https://github.com/gaborgsomogyi/flink/blob/8ab75e46013f159778ccfce52463e7bc63e395a9/flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java#L176> I've came up with a better proposal. David, thanks for pinpointing this out, you've caught a bug in the early phase!
Namely ResourceManager <https://github.com/apache/flink/blob/674bc96662285b25e395fd3dddf9291a602fc183/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java#L124> is a single instance class where DTM can be added as member variable. It has a list of all already registered TMs and new TM registration is also happening here. The following can be added from logic perspective to be more specific: * Create new DTM instance in ResourceManager <https://github.com/apache/flink/blob/674bc96662285b25e395fd3dddf9291a602fc183/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java#L124> and start it (re-occurring thread to obtain new tokens) * Add a new function named "updateDelegationTokens" to TaskExecutorGateway <https://github.com/apache/flink/blob/674bc96662285b25e395fd3dddf9291a602fc183/flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutorGateway.java#L54> * Call "updateDelegationTokens" on all registered TMs to propagate new DTs * In case of new TM registration call "updateDelegationTokens" before registration succeeds to setup new TM properly This way: * only a single DTM would live within a cluster which is the expected behavior * DTM is going to be added to a central place where all deployment target can make use of it * DTs are going to be pushed to TMs which would generate less network traffic than pull based approach (please see my previous mail where I've described both approaches) * HA scenario is going to be consistent because such <https://github.com/apache/flink/blob/674bc96662285b25e395fd3dddf9291a602fc183/flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java#L1069> a solution can be added to "updateDelegationTokens" @David or all others plz share whether you agree on this or you have better idea/suggestion. BR, G On Tue, Jan 25, 2022 at 11:00 AM Gabor Somogyi <gabor.g.somo...@gmail.com> wrote: > First of all thanks for investing your time and helping me out. As I see > you have pretty solid knowledge in the RPC area. > I would like to rely on your knowledge since I'm learning this part. > > > - Do we need to introduce a new RPC method or can we for example > piggyback > on heartbeats? > > I'm fine with either solution but one thing is important conceptually. > There are fundamentally 2 ways how tokens can be updated: > - Push way: When there are new DTs then JM JVM pushes DTs to TM JVMs. This > is the preferred one since tiny amount of control logic needed. > - Pull way: Each time a TM would like to poll JM whether there are new > tokens and each TM wants to decide alone whether DTs needs to be updated or > not. > As you've mentioned here some ID needs to be generated, it would generated > quite some additional network traffic which can be definitely avoided. > As a final thought in Spark we've had this way of DT propagation logic and > we've had major issues with it. > > So all in all DTM needs to obtain new tokens and there must a way to send > this data to all TMs from JM. > > > - What delivery semantics are we looking for? (what if we're only able to > update subset of TMs / what happens if we exhaust retries / should we even > have the retry mechanism whatsoever) - I have a feeling that somehow > leveraging the existing heartbeat mechanism could help to answer these > questions > > Let's go through these questions one by one. > > What delivery semantics are we looking for? > > DTM must receive an exception when at least one TM was not able to get DTs. > > > what if we're only able to update subset of TMs? > > Such case DTM will reschedule token obtain after > "security.kerberos.tokens.retry-wait" time. > > > what happens if we exhaust retries? > > There is no number of retries. In default configuration tokens needs to be > re-obtained after one day. > DTM tries to obtain new tokens after 1day * 0.75 > (security.kerberos.tokens.renewal-ratio) = 18 hours. > When fails it retries after "security.kerberos.tokens.retry-wait" which is > 1 hour by default. > If it never succeeds then authentication error is going to happen on the > TM side and the workload is > going to stop. > > > should we even have the retry mechanism whatsoever? > > Yes, because there are always temporary cluster issues. > > > What does it mean for the running application (how does this look like > from > the user perspective)? As far as I remember the logs are only collected > ("aggregated") after the container is stopped, is that correct? > > With default config it works like that but it can be forced to aggregate > at specific intervals. > A useful feature is forcing YARN to aggregate logs while the job is still > running. > For long-running jobs such as streaming jobs, this is invaluable. To do > this, > yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds must be > set to a non-negative value. > When this is set, a timer will be set for the given duration, and whenever > that timer goes off, > log aggregation will run on new files. > > > I think > this topic should get its own section in the FLIP (having some cross > reference to YARN ticket would be really useful, but I'm not sure if there > are any). > > I think this is important knowledge but this FLIP is not touching the > already existing behavior. > DTs are set on the AM container which is renewed by YARN until it's not > possible anymore. > Any kind of new code is not going to change this limitation. BTW, there is > no jira for this. > If you think it worth to write this down then I think the good place is > the official security doc > area as caveat. > > > If we split the FLIP into two parts / sections that I've suggested, I > don't > really think that you need to explicitly test for each deployment scenario > / cluster framework, because the DTM part is completely independent of the > deployment target. Basically this is what I'm aiming for with "making it > work with the standalone" (as simple as starting a new java process) Flink > first (which is also how most people deploy streaming application on k8s > and the direction we're pushing forward with the auto-scaling / reactive > mode initiatives). > > I see your point and agree the main direction. k8s is the megatrend which > most of the peoples > will use sooner or later. Not 100% sure what kind of split you suggest but > in my view > the main target is to add this feature and I'm open to any logical work > ordering. > Please share the specific details and we work it out... > > G > > > On Mon, Jan 24, 2022 at 3:04 PM David Morávek <d...@apache.org> wrote: > >> > >> > Could you point to a code where you think it could be added exactly? A >> > helping hand is welcome here 🙂 >> > >> >> I think you can take a look at _ResourceManagerPartitionTracker_ [1] which >> seems to have somewhat similar properties to the DTM. >> >> One topic that needs to be addressed there is how the RPC with the >> _TaskExecutorGateway_ should look like. >> - Do we need to introduce a new RPC method or can we for example piggyback >> on heartbeats? >> - What delivery semantics are we looking for? (what if we're only able to >> update subset of TMs / what happens if we exhaust retries / should we even >> have the retry mechanism whatsoever) - I have a feeling that somehow >> leveraging the existing heartbeat mechanism could help to answer these >> questions >> >> In short, after DT reaches it's max lifetime then log aggregation stops >> > >> >> What does it mean for the running application (how does this look like >> from >> the user perspective)? As far as I remember the logs are only collected >> ("aggregated") after the container is stopped, is that correct? I think >> this topic should get its own section in the FLIP (having some cross >> reference to YARN ticket would be really useful, but I'm not sure if there >> are any). >> >> All deployment modes (per-job, per-app, ...) are planned to be tested and >> > expect to work with the initial implementation however not all >> deployment >> > targets (k8s, local, ... >> > >> >> If we split the FLIP into two parts / sections that I've suggested, I >> don't >> really think that you need to explicitly test for each deployment scenario >> / cluster framework, because the DTM part is completely independent of the >> deployment target. Basically this is what I'm aiming for with "making it >> work with the standalone" (as simple as starting a new java process) Flink >> first (which is also how most people deploy streaming application on k8s >> and the direction we're pushing forward with the auto-scaling / reactive >> mode initiatives). >> >> The whole integration with YARN (let's forget about log aggregation for a >> moment) / k8s-native only boils down to how do we make the keytab file >> local to the JobManager so the DTM can read it, so it's basically built on >> top of that. The only special thing that needs to be tested there is the >> "keytab distribution" code path. >> >> [1] >> >> https://github.com/apache/flink/blob/release-1.14.3/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResourceManagerPartitionTracker.java >> >> Best, >> D. >> >> On Mon, Jan 24, 2022 at 12:35 PM Gabor Somogyi <gabor.g.somo...@gmail.com >> > >> wrote: >> >> > > There is a separate JobMaster for each job >> > within a Flink cluster and each JobMaster only has a partial view of the >> > task managers >> > >> > Good point! I've had a deeper look and you're right. We definitely need >> to >> > find another place. >> > >> > > Related per-cluster or per-job keytab: >> > >> > In the current code per-cluster keytab is implemented and I'm intended >> to >> > keep it like this within this FLIP. The reason is simple: tokens on TM >> side >> > can be stored within the UserGroupInformation (UGI) structure which is >> > global. I'm not telling it's impossible to change that but I think that >> > this is such a complexity which the initial implementation is not >> required >> > to contain. Additionally we've not seen such need from user side. If the >> > need may rise later on then another FLIP with this topic can be created >> and >> > discussed. Proper multi-UGI handling within a single JVM is a topic >> where >> > several round of deep-dive with the Hadoop/YARN guys are required. >> > >> > > single DTM instance embedded with >> > the ResourceManager (the Flink component) >> > >> > Could you point to a code where you think it could be added exactly? A >> > helping hand is welcome here🙂 >> > >> > > Then the single (initial) implementation should work with all the >> > deployments modes out of the box (which is not what the FLIP suggests). >> Is >> > that correct? >> > >> > All deployment modes (per-job, per-app, ...) are planned to be tested >> and >> > expect to work with the initial implementation however not all >> deployment >> > targets (k8s, local, ...) are not intended to be tested. Per deployment >> > target new jira needs to be created where I expect small number of codes >> > needs to be added and relatively expensive testing effort is required. >> > >> > > I've taken a look into the prototype and in the >> "YarnClusterDescriptor" >> > you're injecting a delegation token into the AM [1] (that's obtained >> using >> > the provided keytab). If I understand this correctly from previous >> > discussion / FLIP, this is to support log aggregation and DT has a >> limited >> > validity. How is this DT going to be renewed? >> > >> > You're clever and touched a limitation which Spark has too. In short, >> after >> > DT reaches it's max lifetime then log aggregation stops. I've had >> several >> > deep-dive rounds with the YARN guys at Spark years because wanted to >> fill >> > this gap. They can't provide us any way to re-inject the newly obtained >> DT >> > so at the end I gave up this. >> > >> > BR, >> > G >> > >> > >> > On Mon, 24 Jan 2022, 11:00 David Morávek, <d...@apache.org> wrote: >> > >> > > Hi Gabor, >> > > >> > > There is actually a huge difference between JobManager (process) and >> > > JobMaster (job coordinator). The naming is unfortunately bit >> misleading >> > > here from historical reasons. There is a separate JobMaster for each >> job >> > > within a Flink cluster and each JobMaster only has a partial view of >> the >> > > task managers (depends on where the slots for a particular job are >> > > allocated). This means that you'll end up with N >> > "DelegationTokenManagers" >> > > competing with each other (N = number of running jobs in the cluster). >> > > >> > > This makes me think we're mixing two abstraction levels here: >> > > >> > > a) Per-cluster delegation tokens >> > > - Simpler approach, it would involve a single DTM instance embedded >> with >> > > the ResourceManager (the Flink component) >> > > b) Per-job delegation tokens >> > > - More complex approach, but could be more flexible from the user >> side of >> > > things. >> > > - Multiple DTM instances, that are bound with the JobMaster lifecycle. >> > > Delegation tokens are attached with a particular slots that are >> executing >> > > the job tasks instead of the whole task manager (TM could be executing >> > > multiple jobs with different tokens). >> > > - The question is which keytab should be used for the clustering >> > framework, >> > > to support log aggregation on YARN (an extra keytab, keytab that comes >> > with >> > > the first job?) >> > > >> > > I think these are the things that need to be clarified in the FLIP >> before >> > > proceeding. >> > > >> > > A follow-up question for getting a better understanding where this >> should >> > > be headed: Are there any use cases where user may want to use >> different >> > > keytabs with each job, or are we fine with using a cluster-wide >> keytab? >> > If >> > > we go with per-cluster keytabs, is it OK that all jobs submitted into >> > this >> > > cluster can access it (even the future ones)? Should this be a >> security >> > > concern? >> > > >> > > Presume you though I would implement a new class with JobManager name. >> > The >> > > > plan is not that. >> > > > >> > > >> > > I've never suggested such thing. >> > > >> > > >> > > > No. That said earlier DT handling is planned to be done completely >> in >> > > > Flink. DTM has a renewal thread which re-obtains tokens in the >> proper >> > > time >> > > > when needed. >> > > > >> > > >> > > Then the single (initial) implementation should work with all the >> > > deployments modes out of the box (which is not what the FLIP >> suggests). >> > Is >> > > that correct? >> > > >> > > If the cluster framework, also requires delegation token for their >> inner >> > > working (this is IMO only applies to YARN), it might need an extra >> step >> > > (injecting the token into application master container). >> > > >> > > Separating the individual layers (actual Flink cluster - basically >> making >> > > this work with a standalone deployment / "cluster framework" - >> support >> > for >> > > YARN log aggregation) in the FLIP would be useful. >> > > >> > > Reading the linked Spark readme could be useful. >> > > > >> > > >> > > I've read that, but please be patient with the questions, Kerberos is >> not >> > > an easy topic to get into and I've had a very little contact with it >> in >> > the >> > > past. >> > > >> > > >> > > >> > >> https://github.com/gaborgsomogyi/flink/blob/8ab75e46013f159778ccfce52463e7bc63e395a9/flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java#L176 >> > > > >> > > >> > > I've taken a look into the prototype and in the >> "YarnClusterDescriptor" >> > > you're injecting a delegation token into the AM [1] (that's obtained >> > using >> > > the provided keytab). If I understand this correctly from previous >> > > discussion / FLIP, this is to support log aggregation and DT has a >> > limited >> > > validity. How is this DT going to be renewed? >> > > >> > > [1] >> > > >> > > >> > >> https://github.com/gaborgsomogyi/flink/commit/8ab75e46013f159778ccfce52463e7bc63e395a9#diff-02416e2d6ca99e1456f9c3949f3d7c2ac523d3fe25378620c09632e4aac34e4eR1261 >> > > >> > > Best, >> > > D. >> > > >> > > On Fri, Jan 21, 2022 at 9:35 PM Gabor Somogyi < >> gabor.g.somo...@gmail.com >> > > >> > > wrote: >> > > >> > > > Here is the exact class, I'm from mobile so not had a look at the >> exact >> > > > class name: >> > > > >> > > > >> > > >> > >> https://github.com/gaborgsomogyi/flink/blob/8ab75e46013f159778ccfce52463e7bc63e395a9/flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java#L176 >> > > > That keeps track of TMs where the tokens can be sent to. >> > > > >> > > > > My feeling would be that we shouldn't really introduce a new >> > component >> > > > with >> > > > a custom lifecycle, but rather we should try to incorporate this >> into >> > > > existing ones. >> > > > >> > > > Can you be more specific? Presume you though I would implement a new >> > > class >> > > > with JobManager name. The plan is not that. >> > > > >> > > > > If I understand this correctly, this means that we then push the >> > token >> > > > renewal logic to YARN. >> > > > >> > > > No. That said earlier DT handling is planned to be done completely >> in >> > > > Flink. DTM has a renewal thread which re-obtains tokens in the >> proper >> > > time >> > > > when needed. YARN log aggregation is a totally different feature, >> where >> > > > YARN does the renewal. Log aggregation was an example why the code >> > can't >> > > be >> > > > 100% reusable for all resource managers. Reading the linked Spark >> > readme >> > > > could be useful. >> > > > >> > > > G >> > > > >> > > > On Fri, 21 Jan 2022, 21:05 David Morávek, <d...@apache.org> wrote: >> > > > >> > > > > > >> > > > > > JobManager is the Flink class. >> > > > > >> > > > > >> > > > > There is no such class in Flink. The closest thing to the >> JobManager >> > > is a >> > > > > ClusterEntrypoint. The cluster entrypoint spawns new RM Runner & >> > > > Dispatcher >> > > > > Runner that start participating in the leader election. Once they >> > gain >> > > > > leadership they spawn the actual underlying instances of these two >> > > "main >> > > > > components". >> > > > > >> > > > > My feeling would be that we shouldn't really introduce a new >> > component >> > > > with >> > > > > a custom lifecycle, but rather we should try to incorporate this >> into >> > > > > existing ones. >> > > > > >> > > > > My biggest concerns would be: >> > > > > >> > > > > - How would the lifecycle of the new component look like with >> regards >> > > to >> > > > HA >> > > > > setups. If we really try to decide to introduce a completely new >> > > > component, >> > > > > how should this work in case of multiple JobManager instances? >> > > > > - Which components does it talk to / how? For example how does the >> > > > > broadcast of new token to task managers (TaskManagerGateway) look >> > like? >> > > > Do >> > > > > we simply introduce a new RPC on the ResourceManagerGateway that >> > > > broadcasts >> > > > > it or does the new component need to do some kind of bookkeeping >> of >> > > task >> > > > > managers that it needs to notify? >> > > > > >> > > > > YARN based HDFS log aggregation would not work by dropping that >> code. >> > > > Just >> > > > > > to be crystal clear, the actual implementation contains this fir >> > > > exactly >> > > > > > this reason. >> > > > > > >> > > > > >> > > > > This is the missing part +1. If I understand this correctly, this >> > means >> > > > > that we then push the token renewal logic to YARN. How do you >> plan to >> > > > > implement the renewal logic on k8s? >> > > > > >> > > > > D. >> > > > > >> > > > > On Fri, Jan 21, 2022 at 8:37 PM Gabor Somogyi < >> > > gabor.g.somo...@gmail.com >> > > > > >> > > > > wrote: >> > > > > >> > > > > > > I think we might both mean something different by the RM. >> > > > > > >> > > > > > You feel it well, I've not specified these terms well in the >> > > > explanation. >> > > > > > RM I meant resource management framework. JobManager is the >> Flink >> > > > class. >> > > > > > This means that inside JM instance there will be a DTM >> instance, so >> > > > they >> > > > > > would have the same lifecycle. Hope I've answered the question. >> > > > > > >> > > > > > > If we have tokens available on the client side, why do we >> need to >> > > set >> > > > > > them >> > > > > > into the AM (yarn specific concept) launch context? >> > > > > > >> > > > > > YARN based HDFS log aggregation would not work by dropping that >> > code. >> > > > > Just >> > > > > > to be crystal clear, the actual implementation contains this fir >> > > > exactly >> > > > > > this reason. >> > > > > > >> > > > > > G >> > > > > > >> > > > > > On Fri, 21 Jan 2022, 20:12 David Morávek, <d...@apache.org> >> wrote: >> > > > > > >> > > > > > > Hi Gabor, >> > > > > > > >> > > > > > > 1. One thing is important, token management is planned to be >> done >> > > > > > > > generically within Flink and not scattered in RM specific >> code. >> > > > > > > JobManager >> > > > > > > > has a DelegationTokenManager which obtains tokens >> time-to-time >> > > (if >> > > > > > > > configured properly). JM knows which TaskManagers are in >> place >> > so >> > > > it >> > > > > > can >> > > > > > > > distribute it to all TMs. That's it basically. >> > > > > > > >> > > > > > > >> > > > > > > I think we might both mean something different by the RM. >> > > JobManager >> > > > is >> > > > > > > basically just a process encapsulating multiple components, >> one >> > of >> > > > > which >> > > > > > is >> > > > > > > a ResourceManager, which is the component that manages task >> > manager >> > > > > > > registrations [1]. There is more or less a single >> implementation >> > of >> > > > the >> > > > > > RM >> > > > > > > with plugable drivers for the active integrations (yarn, k8s). >> > > > > > > >> > > > > > > It would be great if you could share more details of how >> exactly >> > > the >> > > > > DTM >> > > > > > is >> > > > > > > going to fit in the current JM architecture. >> > > > > > > >> > > > > > > 2. 99.9% of the code is generic but each RM handles tokens >> > > > > differently. A >> > > > > > > > good example is YARN obtains tokens on client side and then >> > sets >> > > > them >> > > > > > on >> > > > > > > > the newly created AM container launch context. This is >> purely >> > > YARN >> > > > > > > specific >> > > > > > > > and cant't be spared. With my actual plans standalone can be >> > > > changed >> > > > > to >> > > > > > > use >> > > > > > > > the framework. By using it I mean no RM specific DTM or >> > > whatsoever >> > > > is >> > > > > > > > needed. >> > > > > > > > >> > > > > > > >> > > > > > > If we have tokens available on the client side, why do we >> need to >> > > set >> > > > > > them >> > > > > > > into the AM (yarn specific concept) launch context? Why can't >> we >> > > > simply >> > > > > > > send them to the JM, eg. as a parameter of the job submission >> / >> > via >> > > > > > > separate RPC call? There might be something I'm missing due to >> > > > limited >> > > > > > > knowledge, but handling the token on the "cluster framework" >> > level >> > > > > > doesn't >> > > > > > > seem necessary. >> > > > > > > >> > > > > > > [1] >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/concepts/flink-architecture/#jobmanager >> > > > > > > >> > > > > > > Best, >> > > > > > > D. >> > > > > > > >> > > > > > > On Fri, Jan 21, 2022 at 7:48 PM Gabor Somogyi < >> > > > > gabor.g.somo...@gmail.com >> > > > > > > >> > > > > > > wrote: >> > > > > > > >> > > > > > > > Oh and one more thing. I'm planning to add this feature in >> > small >> > > > > chunk >> > > > > > of >> > > > > > > > PRs because security is super hairy area. That way reviewers >> > can >> > > be >> > > > > > more >> > > > > > > > easily obtains the concept. >> > > > > > > > >> > > > > > > > On Fri, 21 Jan 2022, 18:03 David Morávek, <d...@apache.org> >> > > wrote: >> > > > > > > > >> > > > > > > > > Hi Gabor, >> > > > > > > > > >> > > > > > > > > thanks for drafting the FLIP, I think having a solid >> Kerberos >> > > > > support >> > > > > > > is >> > > > > > > > > crucial for many enterprise deployments. >> > > > > > > > > >> > > > > > > > > I have multiple questions regarding the implementation >> (note >> > > > that I >> > > > > > > have >> > > > > > > > > very limited knowledge of Kerberos): >> > > > > > > > > >> > > > > > > > > 1) If I understand it correctly, we'll only obtain tokens >> in >> > > the >> > > > > job >> > > > > > > > > manager and then we'll distribute them via RPC (needs to >> be >> > > > > secured). >> > > > > > > > > >> > > > > > > > > Can you please outline how the communication will look >> like? >> > Is >> > > > the >> > > > > > > > > DelegationTokenManager going to be a part of the >> > > ResourceManager? >> > > > > Can >> > > > > > > you >> > > > > > > > > outline it's lifecycle / how it's going to be integrated >> > there? >> > > > > > > > > >> > > > > > > > > 2) Do we really need a YARN / k8s specific >> implementations? >> > Is >> > > it >> > > > > > > > possible >> > > > > > > > > to obtain / renew a token in a generic way? Maybe to >> rephrase >> > > > that, >> > > > > > is >> > > > > > > it >> > > > > > > > > possible to implement DelegationTokenManager for the >> > standalone >> > > > > > Flink? >> > > > > > > If >> > > > > > > > > we're able to solve this point, it could be possible to >> > target >> > > > all >> > > > > > > > > deployment scenarios with a single implementation. >> > > > > > > > > >> > > > > > > > > Best, >> > > > > > > > > D. >> > > > > > > > > >> > > > > > > > > On Fri, Jan 14, 2022 at 3:47 AM Junfan Zhang < >> > > > > > zuston.sha...@gmail.com> >> > > > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > Hi G >> > > > > > > > > > >> > > > > > > > > > Thanks for your explain in detail. I have gotten your >> > > thoughts, >> > > > > and >> > > > > > > any >> > > > > > > > > > way this proposal >> > > > > > > > > > is a great improvement. >> > > > > > > > > > >> > > > > > > > > > Looking forward to your implementation and i will keep >> > focus >> > > on >> > > > > it. >> > > > > > > > > > Thanks again. >> > > > > > > > > > >> > > > > > > > > > Best >> > > > > > > > > > JunFan. >> > > > > > > > > > On Jan 13, 2022, 9:20 PM +0800, Gabor Somogyi < >> > > > > > > > gabor.g.somo...@gmail.com >> > > > > > > > > >, >> > > > > > > > > > wrote: >> > > > > > > > > > > Just to confirm keeping >> > > > > > "security.kerberos.fetch.delegation-token" >> > > > > > > is >> > > > > > > > > > added >> > > > > > > > > > > to the doc. >> > > > > > > > > > > >> > > > > > > > > > > BR, >> > > > > > > > > > > G >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > On Thu, Jan 13, 2022 at 1:34 PM Gabor Somogyi < >> > > > > > > > > gabor.g.somo...@gmail.com >> > > > > > > > > > > >> > > > > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > Hi JunFan, >> > > > > > > > > > > > >> > > > > > > > > > > > > By the way, maybe this should be added in the >> > migration >> > > > > plan >> > > > > > or >> > > > > > > > > > > > intergation section in the FLIP-211. >> > > > > > > > > > > > >> > > > > > > > > > > > Going to add this soon. >> > > > > > > > > > > > >> > > > > > > > > > > > > Besides, I have a question that the KDC will >> collapse >> > > > when >> > > > > > the >> > > > > > > > > > cluster >> > > > > > > > > > > > reached 200 nodes you described >> > > > > > > > > > > > in the google doc. Do you have any attachment or >> > > reference >> > > > to >> > > > > > > prove >> > > > > > > > > it? >> > > > > > > > > > > > >> > > > > > > > > > > > "KDC *may* collapse under some circumstances" is the >> > > proper >> > > > > > > > wording. >> > > > > > > > > > > > >> > > > > > > > > > > > We have several customers who are executing >> workloads >> > on >> > > > > > > > Spark/Flink. >> > > > > > > > > > Most >> > > > > > > > > > > > of the time I'm facing their >> > > > > > > > > > > > daily issues which is heavily environment and >> use-case >> > > > > > dependent. >> > > > > > > > > I've >> > > > > > > > > > > > seen various cases: >> > > > > > > > > > > > * where the mentioned ~1k nodes were working fine >> > > > > > > > > > > > * where KDC thought the number of requests are >> coming >> > > from >> > > > > DDOS >> > > > > > > > > attack >> > > > > > > > > > so >> > > > > > > > > > > > discontinued authentication >> > > > > > > > > > > > * where KDC was simply not responding because of the >> > load >> > > > > > > > > > > > * where KDC was intermittently had some outage (this >> > was >> > > > the >> > > > > > most >> > > > > > > > > nasty >> > > > > > > > > > > > thing) >> > > > > > > > > > > > >> > > > > > > > > > > > Since you're managing relatively big cluster then >> you >> > > know >> > > > > that >> > > > > > > KDC >> > > > > > > > > is >> > > > > > > > > > not >> > > > > > > > > > > > only used by Spark/Flink workloads >> > > > > > > > > > > > but the whole company IT infrastructure is bombing >> it >> > so >> > > it >> > > > > > > really >> > > > > > > > > > depends >> > > > > > > > > > > > on other factors too whether KDC is reaching >> > > > > > > > > > > > it's limit or not. Not sure what kind of evidence >> are >> > you >> > > > > > looking >> > > > > > > > for >> > > > > > > > > > but >> > > > > > > > > > > > I'm not authorized to share any information about >> > > > > > > > > > > > our clients data. >> > > > > > > > > > > > >> > > > > > > > > > > > One thing is for sure. The more external system >> types >> > are >> > > > > used >> > > > > > in >> > > > > > > > > > > > workloads (for ex. HDFS, HBase, Hive, Kafka) which >> > > > > > > > > > > > are authenticating through KDC the more possibility >> to >> > > > reach >> > > > > > this >> > > > > > > > > > > > threshold when the cluster is big enough. >> > > > > > > > > > > > >> > > > > > > > > > > > All in all this feature is here to help all users >> never >> > > > reach >> > > > > > > this >> > > > > > > > > > > > limitation. >> > > > > > > > > > > > >> > > > > > > > > > > > BR, >> > > > > > > > > > > > G >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > On Thu, Jan 13, 2022 at 1:00 PM 张俊帆 < >> > > > zuston.sha...@gmail.com >> > > > > > >> > > > > > > > wrote: >> > > > > > > > > > > > >> > > > > > > > > > > > > Hi G >> > > > > > > > > > > > > >> > > > > > > > > > > > > Thanks for your quick reply. I think reserving the >> > > config >> > > > > of >> > > > > > > > > > > > > *security.kerberos.fetch.delegation-token* >> > > > > > > > > > > > > and simplifying disable the token fetching is a >> good >> > > > > idea.By >> > > > > > > the >> > > > > > > > > way, >> > > > > > > > > > > > > maybe this should be added >> > > > > > > > > > > > > in the migration plan or intergation section in >> the >> > > > > FLIP-211. >> > > > > > > > > > > > > >> > > > > > > > > > > > > Besides, I have a question that the KDC will >> collapse >> > > > when >> > > > > > the >> > > > > > > > > > cluster >> > > > > > > > > > > > > reached 200 nodes you described >> > > > > > > > > > > > > in the google doc. Do you have any attachment or >> > > > reference >> > > > > to >> > > > > > > > prove >> > > > > > > > > > it? >> > > > > > > > > > > > > Because in our internal per-cluster, >> > > > > > > > > > > > > the nodes reaches > 1000 and KDC looks good. Do i >> > > missed >> > > > or >> > > > > > > > > > misunderstood >> > > > > > > > > > > > > something? Please correct me. >> > > > > > > > > > > > > >> > > > > > > > > > > > > Best >> > > > > > > > > > > > > JunFan. >> > > > > > > > > > > > > On Jan 13, 2022, 5:26 PM +0800, >> dev@flink.apache.org >> > , >> > > > > wrote: >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://docs.google.com/document/d/1JzMbQ1pCJsLVz8yHrCxroYMRP2GwGwvacLrGyaIx5Yc/edit?fbclid=IwAR0vfeJvAbEUSzHQAAJfnWTaX46L6o7LyXhMfBUCcPrNi-uXNgoOaI8PMDQ >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> >
