Hi everyone, sorry for joining late and thanks for the insightful discussion.
In general, I'd personally prefer not to increase the surface area of Apache Flink unless there is a good reason. It seems we all agree that authx is not part of the core value proposition of Apache Flink, so if we can delegate this problem to a more specialized tool, I am in favor of that. Apache Flink is already huge and a lot of work goes into maintenance, so I personally have become more sensitive to this aspect over time. If we add support for Basic Auth and Kerberos now, users will sooner or later ask for OIDC, LDAP, SAML,... I acknowledge that Kerberos is widely used in the corporate, on-premises context, but isn't the focus moving more towards more web-friendly standards like OIDC/OAuth 2.0? If we only want to support a single protocol, there is an argument to be made that it should be OIDC and Dex [1,2] as a bridge to everything else. Have OIDC or OAuth2 been considered instead of Kerberos? How do you see the market moving? But as I said before, in my opinion we can generate more value by investing into other areas of Apache Flink. Authorization also has the potential to become more fine-grained and complex over time: you already mentioned restricting the actions that a specific user can do in a cluster. Cheers, Konstantin [1] https://github.com/dexidp/dex [2] https://github.com/dexidp/dex/issues/1903 On Wed, Jun 16, 2021 at 11:44 AM Gabor Somogyi <gabor.g.somo...@gmail.com> wrote: > Hi Till, > > Did you have the chance to take a look at the doc? Not yet seen any update. > > BR, > G > > > On Wed, Jun 9, 2021 at 1:43 PM Till Rohrmann <trohrm...@apache.org> wrote: > > > Thanks for the update Gabor. I'll take a look and respond in the > document. > > > > Cheers, > > Till > > > > On Wed, Jun 9, 2021 at 12:59 PM Gabor Somogyi <gabor.g.somo...@gmail.com > > > > wrote: > > > >> Hi Till, > >> > >> Your proxy suggestion has been considered in-depth and updated the FLIP > >> accordingly. > >> We've considered 2 proxy implementation (Nginx and Squid) but according > >> to our analysis and testing it's not suitable for the mentioned > use-cases. > >> Please take a look at the rejected alternatives for detailed > explanation. > >> > >> Thanks for your time in advance! > >> > >> BR, > >> G > >> > >> > >> On Fri, Jun 4, 2021 at 3:31 PM Till Rohrmann <trohrm...@apache.org> > >> wrote: > >> > >>> As I've said I am not a security expert and that's why I have to ask > for > >>> clarification, Gabor. You are saying that if we configure a truststore > for > >>> the REST endpoint with a single trusted certificate which has been > >>> generated by the operator of the Flink cluster, then the attacker can > >>> generate a new certificate, sign it and then talk to the Flink cluster > if > >>> he has access to the node on which the REST endpoint runs? My > understanding > >>> was that you need the corresponding private key which in my proposed > setup > >>> would be under the control of the operator as well (e.g. stored in a > >>> keystore on the same machine but guarded by some secret). That way (if > I am > >>> not mistaken), only the entity which has access to the keystore is > able to > >>> talk to the Flink cluster. > >>> > >>> Maybe we are also getting our wires crossed here and are talking about > >>> different things. > >>> > >>> Thanks for listing the pros and cons of Kerberos. Concerning what other > >>> authentication mechanisms are used in the industry, I am not 100% sure. > >>> > >>> Cheers, > >>> Till > >>> > >>> On Fri, Jun 4, 2021 at 11:09 AM Gabor Somogyi < > gabor.g.somo...@gmail.com> > >>> wrote: > >>> > >>>> > I did not mean for the user to sign its own certificates but for the > >>>> operator of the cluster. Once the user request hits the proxy, it > should no > >>>> longer be under his control. I think I do not fully understand yet > why this > >>>> would not work. > >>>> I said it's not solving the authentication problem over any proxy. > Even > >>>> if the operator is signing the certificate one can have access to an > >>>> internal node. > >>>> Such case anybody can craft certificates which is accepted by the > >>>> server. When it's accepted a bad guy can cancel jobs causing huge > impacts. > >>>> > >>>> > Also, I am missing a bit the comparison of Kerberos to other > >>>> authentication mechanisms and why they were rejected in favour of > Kerberos. > >>>> PROS: > >>>> * Since it's not depending on cloud provider and/or k8s or bare-metal > >>>> etc. deployment it's the biggest plus > >>>> * Centralized with tools and no need to write tons of tools around > >>>> * There are clients/tools on almost all OS-es and several languages > >>>> * Super huge users are using it for years in production w/o huge > issues > >>>> * Provides cross-realm trust possibility amongst other features > >>>> * Several open source components using it which could increase > >>>> compatibility > >>>> > >>>> CONS: > >>>> * Not everybody using kerberos > >>>> * It would increase the code footprint but this is true for many > >>>> features (as a side note I'm here to maintain it) > >>>> > >>>> Feel free to add your points because it only represents a single > >>>> viewpoint. > >>>> Also if you have any better option for strong authentication please > >>>> share it and we can consider the pros/cons here. > >>>> > >>>> BR, > >>>> G > >>>> > >>>> > >>>> On Fri, Jun 4, 2021 at 10:32 AM Till Rohrmann <trohrm...@apache.org> > >>>> wrote: > >>>> > >>>>> I did not mean for the user to sign its own certificates but for the > >>>>> operator of the cluster. Once the user request hits the proxy, it > should no > >>>>> longer be under his control. I think I do not fully understand yet > why this > >>>>> would not work. > >>>>> > >>>>> What I would like to avoid is to add more complexity into Flink if > >>>>> there is an easy solution which fulfills the requirements. That's > why I > >>>>> would like to exercise thoroughly through the different > alternatives. Also, > >>>>> I am missing a bit the comparison of Kerberos to other authentication > >>>>> mechanisms and why they were rejected in favour of Kerberos. > >>>>> > >>>>> Cheers, > >>>>> Till > >>>>> > >>>>> On Fri, Jun 4, 2021 at 10:26 AM Gyula Fóra <gyf...@apache.org> > wrote: > >>>>> > >>>>>> Hi! > >>>>>> > >>>>>> I think there might be possible alternatives but it seems Kerberos > on > >>>>>> the rest endpoint ticks all the right boxes and provides a super > clean and > >>>>>> simple solution for strong authentication. > >>>>>> > >>>>>> I wouldn’t even consider sidecar proxies etc if we can solve it in > >>>>>> such a simple way as proposed by G. > >>>>>> > >>>>>> Cheers > >>>>>> Gyula > >>>>>> > >>>>>> On Fri, 4 Jun 2021 at 10:03, Till Rohrmann <trohrm...@apache.org> > >>>>>> wrote: > >>>>>> > >>>>>>> I am not saying that we shouldn't add a strong authentication > >>>>>>> mechanism if there are good reasons for it. I primarily would like > to > >>>>>>> understand the context a bit better in order to give qualified > feedback and > >>>>>>> come to a good decision. In order to do this, I have the feeling > that we > >>>>>>> haven't fully considered all available options which are on the > table, tbh. > >>>>>>> > >>>>>>> Does the problem of certificate expiry also apply for self-signed > >>>>>>> certificates? If yes, then this should then also be a problem for > the > >>>>>>> internal encryption of Flink's communication. If not, then one > could use > >>>>>>> self-signed certificates with a longer validity to solve the > mentioned > >>>>>>> issue. > >>>>>>> > >>>>>>> I think you can set up Flink in such a way that you don't have to > >>>>>>> handle all the different certificates. For example, you could > deploy Flink > >>>>>>> with a "sidecar proxy" which is responsible for the authentication > using an > >>>>>>> arbitrary method (e.g. Kerberos) and then bind the REST endpoint > to a local > >>>>>>> network interface. That way, the REST endpoint would only be > available > >>>>>>> through the sidecar proxy. Additionally, one could enable SSL for > this > >>>>>>> communication. Would this be a solution for the problem? > >>>>>>> > >>>>>>> Cheers, > >>>>>>> Till > >>>>>>> > >>>>>>> On Thu, Jun 3, 2021 at 10:46 PM Márton Balassi < > >>>>>>> balassi.mar...@gmail.com> wrote: > >>>>>>> > >>>>>>>> That is an interesting idea, Till. > >>>>>>>> > >>>>>>>> The main issue with it is that TLS certificates have an expiration > >>>>>>>> time, usually they get approved for a couple years. Forcing our > users to > >>>>>>>> restart jobs to reprovision TLS certificates would be weird when > we could > >>>>>>>> just implement a single proper strong authentication mechanism > instead in a > >>>>>>>> couple hundred lines of code. :-) > >>>>>>>> > >>>>>>>> In many cases it is also impractical to go the TLS mutual route, > >>>>>>>> because the Flink Dashboard can end up on any node in the > k8s/Yarn cluster > >>>>>>>> which means that we need a certificate per node (due to the > mutual auth), > >>>>>>>> but if we also want to protect the private key of these from users > >>>>>>>> accidentally or intentionally leaking them then we need this per > user. As > >>>>>>>> in we end up managing user*machine number certificates and having > to renew > >>>>>>>> them periodically, which albeit automatable is unfortunately not > yet > >>>>>>>> automated in all large organizations. > >>>>>>>> > >>>>>>>> I fully agree that TLS certificate mutual authentication has its > >>>>>>>> nice properties, especially at very large (multiple thousand > node) clusters > >>>>>>>> - but it has its own challenges too. Thanks for bringing it up. > >>>>>>>> > >>>>>>>> Happy to have this added to the rejected alternative list so that > >>>>>>>> we have the full picture documented. > >>>>>>>> > >>>>>>>> On Thu, Jun 3, 2021 at 5:52 PM Till Rohrmann < > trohrm...@apache.org> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> I guess the idea would then be to let the proxy do the > >>>>>>>>> authentication job and only forward the request via an SSL > mutually > >>>>>>>>> encrypted connection to the Flink cluster. Would this be > possible? The > >>>>>>>>> beauty of this setup is in my opinion that this setup should > work with all > >>>>>>>>> kinds of authentication mechanisms. > >>>>>>>>> > >>>>>>>>> Cheers, > >>>>>>>>> Till > >>>>>>>>> > >>>>>>>>> On Thu, Jun 3, 2021 at 3:12 PM Gabor Somogyi < > >>>>>>>>> gabor.g.somo...@gmail.com> wrote: > >>>>>>>>> > >>>>>>>>>> Thanks for giving options to fulfil the need. > >>>>>>>>>> > >>>>>>>>>> Users are looking for a solution where users can be identified > on > >>>>>>>>>> the whole cluster and restrict access to resources/actions. > >>>>>>>>>> A good example for such an action is cancelling other users > >>>>>>>>>> running jobs. > >>>>>>>>>> > >>>>>>>>>> * SSL does provide mutual authentication but when authentication > >>>>>>>>>> passed there is no user based on restrictions can be made. > >>>>>>>>>> * The less problematic part is that generating/maintaining short > >>>>>>>>>> time valid certificates would be a hard (that's the reason KDC > like servers > >>>>>>>>>> exist). > >>>>>>>>>> Having long time valid certificates would widen the attack > >>>>>>>>>> surface but since the first concern is there this is just a > cosmetic issue. > >>>>>>>>>> > >>>>>>>>>> All in all using TLS certificates is not sufficient in these > >>>>>>>>>> environments unfortunately. > >>>>>>>>>> > >>>>>>>>>> BR, > >>>>>>>>>> G > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Thu, Jun 3, 2021 at 12:49 PM Till Rohrmann < > >>>>>>>>>> trohrm...@apache.org> wrote: > >>>>>>>>>> > >>>>>>>>>>> Thanks for the information Gabor. If it is about securing the > >>>>>>>>>>> communication between the REST client and the REST server, > then Flink > >>>>>>>>>>> already supports enabling mutual SSL authentication [1]. Would > this be > >>>>>>>>>>> enough to secure the communication and to pass an audit? > >>>>>>>>>>> > >>>>>>>>>>> [1] > >>>>>>>>>>> > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/security/security-ssl/#external--rest-connectivity > >>>>>>>>>>> > >>>>>>>>>>> Cheers, > >>>>>>>>>>> Till > >>>>>>>>>>> > >>>>>>>>>>> On Thu, Jun 3, 2021 at 10:33 AM Gabor Somogyi < > >>>>>>>>>>> gabor.g.somo...@gmail.com> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi Till, > >>>>>>>>>>>> > >>>>>>>>>>>> Since I'm working in security area 10+ years let me share my > >>>>>>>>>>>> thought. > >>>>>>>>>>>> I would like to emphasise there are experts better than me but > >>>>>>>>>>>> I have some > >>>>>>>>>>>> basics. > >>>>>>>>>>>> The discussion is open and not trying to tell alone things... > >>>>>>>>>>>> > >>>>>>>>>>>> > I mean if an attacker can get access to one of the machines, > >>>>>>>>>>>> then it > >>>>>>>>>>>> should also be possible to obtain the right Kerberos token. > >>>>>>>>>>>> Not necessarily. For example if one gets access to a specific > >>>>>>>>>>>> user's > >>>>>>>>>>>> credentials then it's not possible to compromise other user's > >>>>>>>>>>>> jobs, data, > >>>>>>>>>>>> etc... > >>>>>>>>>>>> Security is like an onion, the more layers has been added the > >>>>>>>>>>>> more time an > >>>>>>>>>>>> attacker needs to proceed. > >>>>>>>>>>>> At the end of the day if one is in, then most probably can > find > >>>>>>>>>>>> the way but > >>>>>>>>>>>> this time is normally enough to sysadmins or security experts > to > >>>>>>>>>>>> close down the system and minimize the damage. > >>>>>>>>>>>> > >>>>>>>>>>>> The other thing is that all tokens has a timeout and if the > >>>>>>>>>>>> token is > >>>>>>>>>>>> invalid then the attacker can't proceed further. > >>>>>>>>>>>> > >>>>>>>>>>>> > Is Kerberos also the standard authentication protocol for > >>>>>>>>>>>> Kubernetes > >>>>>>>>>>>> deployments? > >>>>>>>>>>>> Kerberos is an industry standard which is cloud/deployment > >>>>>>>>>>>> agnostic and it > >>>>>>>>>>>> can be used in any deployments including k8s. > >>>>>>>>>>>> The main intention is to use kerberos in k8s deployments too > >>>>>>>>>>>> since we're > >>>>>>>>>>>> going this direction as well. > >>>>>>>>>>>> Please see how Spark does this: > >>>>>>>>>>>> > >>>>>>>>>>>> > https://spark.apache.org/docs/latest/security.html#secure-interaction-with-kubernetes > >>>>>>>>>>>> > >>>>>>>>>>>> Last but not least the most important reason to add at least > >>>>>>>>>>>> one strong > >>>>>>>>>>>> authentication is that we have users who has > >>>>>>>>>>>> hard requirements on this. They're doing security audits and > if > >>>>>>>>>>>> they fail > >>>>>>>>>>>> then it's deal breaking. > >>>>>>>>>>>> That is why we have added kerberos at the first place. > >>>>>>>>>>>> Unfortunately we > >>>>>>>>>>>> can't name them in this public list, however > >>>>>>>>>>>> the customers who specifically asked for this were mainly in > >>>>>>>>>>>> the banking > >>>>>>>>>>>> and telco sector. > >>>>>>>>>>>> > >>>>>>>>>>>> BR, > >>>>>>>>>>>> G > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Thu, Jun 3, 2021 at 9:20 AM Till Rohrmann < > >>>>>>>>>>>> trohrm...@apache.org> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> > Thanks for updating the document Márton. Why is it that > banks > >>>>>>>>>>>> will > >>>>>>>>>>>> > consider it more secure if Flink comes with Kerberos > >>>>>>>>>>>> authentication > >>>>>>>>>>>> > (assuming a properly secured setup)? I mean if an attacker > >>>>>>>>>>>> can get access > >>>>>>>>>>>> > to one of the machines, then it should also be possible to > >>>>>>>>>>>> obtain the right > >>>>>>>>>>>> > Kerberos token. > >>>>>>>>>>>> > > >>>>>>>>>>>> > I am not an authentication expert and that's why I wanted to > >>>>>>>>>>>> ask what are > >>>>>>>>>>>> > other authentication protocols other than Kerberos? Why did > >>>>>>>>>>>> we select > >>>>>>>>>>>> > Kerberos and not any other authentication protocol? Maybe > you > >>>>>>>>>>>> can list the > >>>>>>>>>>>> > pros and cons for the different protocols. Is Kerberos also > >>>>>>>>>>>> the standard > >>>>>>>>>>>> > authentication protocol for Kubernetes deployments? If not, > >>>>>>>>>>>> what would be > >>>>>>>>>>>> > the answer when deploying on K8s? > >>>>>>>>>>>> > > >>>>>>>>>>>> > Cheers, > >>>>>>>>>>>> > Till > >>>>>>>>>>>> > > >>>>>>>>>>>> > On Wed, Jun 2, 2021 at 12:07 PM Gabor Somogyi < > >>>>>>>>>>>> gabor.g.somo...@gmail.com> > >>>>>>>>>>>> > wrote: > >>>>>>>>>>>> > > >>>>>>>>>>>> >> Hi team, > >>>>>>>>>>>> >> > >>>>>>>>>>>> >> Happy to be here and hope I can provide quality additions > in > >>>>>>>>>>>> the future. > >>>>>>>>>>>> >> > >>>>>>>>>>>> >> Thank you all for helpful the suggestions! > >>>>>>>>>>>> >> Considering them the FLIP has been modified and the work > >>>>>>>>>>>> continues on the > >>>>>>>>>>>> >> already existing Jira. > >>>>>>>>>>>> >> > >>>>>>>>>>>> >> BR, > >>>>>>>>>>>> >> G > >>>>>>>>>>>> >> > >>>>>>>>>>>> >> > >>>>>>>>>>>> >> On Wed, Jun 2, 2021 at 11:23 AM Márton Balassi < > >>>>>>>>>>>> balassi.mar...@gmail.com> > >>>>>>>>>>>> >> wrote: > >>>>>>>>>>>> >> > >>>>>>>>>>>> >>> Thanks, Chesney - I totally missed that. Answered on the > >>>>>>>>>>>> ticket too, let > >>>>>>>>>>>> >>> us continue there then. > >>>>>>>>>>>> >>> > >>>>>>>>>>>> >>> Till, I agree that we should keep this codepath as slim as > >>>>>>>>>>>> possible. It > >>>>>>>>>>>> >>> is an important design decision that we aim to keep the > >>>>>>>>>>>> list of > >>>>>>>>>>>> >>> authentication protocols to a minimum. We believe that > this > >>>>>>>>>>>> should not be a > >>>>>>>>>>>> >>> primary concern of Flink and a trusted proxy service (for > >>>>>>>>>>>> example Apache > >>>>>>>>>>>> >>> Knox) should be used to enable a multitude of enduser > >>>>>>>>>>>> authentication > >>>>>>>>>>>> >>> mechanisms. The bare minimum of authentication mechanisms > >>>>>>>>>>>> to support > >>>>>>>>>>>> >>> consequently consist of a single strong authentication > >>>>>>>>>>>> protocol for which > >>>>>>>>>>>> >>> Kerberos is the enterprise solution and HTTP Basic primary > >>>>>>>>>>>> for development > >>>>>>>>>>>> >>> and light-weight scenarios. > >>>>>>>>>>>> >>> > >>>>>>>>>>>> >>> Added the above wording to G's doc. > >>>>>>>>>>>> >>> > >>>>>>>>>>>> >>> > >>>>>>>>>>>> > https://docs.google.com/document/d/1NMPeJ9H0G49TGy3AzTVVJVKmYC0okwOtqLTSPnGqzHw/edit > >>>>>>>>>>>> >>> > >>>>>>>>>>>> >>> > >>>>>>>>>>>> >>> > >>>>>>>>>>>> >>> On Tue, Jun 1, 2021 at 11:47 AM Chesnay Schepler < > >>>>>>>>>>>> ches...@apache.org> > >>>>>>>>>>>> >>> wrote: > >>>>>>>>>>>> >>> > >>>>>>>>>>>> >>>> There's a related effort: > >>>>>>>>>>>> >>>> https://issues.apache.org/jira/browse/FLINK-21108 > >>>>>>>>>>>> >>>> > >>>>>>>>>>>> >>>> On 6/1/2021 10:14 AM, Till Rohrmann wrote: > >>>>>>>>>>>> >>>> > Hi Gabor, welcome to the Flink community! > >>>>>>>>>>>> >>>> > > >>>>>>>>>>>> >>>> > Thanks for sharing this proposal with the community > >>>>>>>>>>>> Márton. In > >>>>>>>>>>>> >>>> general, I > >>>>>>>>>>>> >>>> > agree that authentication is missing and that this is > >>>>>>>>>>>> required for > >>>>>>>>>>>> >>>> using > >>>>>>>>>>>> >>>> > Flink within an enterprise. The thing I am wondering is > >>>>>>>>>>>> whether this > >>>>>>>>>>>> >>>> > feature strictly needs to be implemented inside of > Flink > >>>>>>>>>>>> or whether a > >>>>>>>>>>>> >>>> proxy > >>>>>>>>>>>> >>>> > setup could do the job? Have you considered this > option? > >>>>>>>>>>>> If yes, then > >>>>>>>>>>>> >>>> it > >>>>>>>>>>>> >>>> > would be good to list it under the point of rejected > >>>>>>>>>>>> alternatives. > >>>>>>>>>>>> >>>> > > >>>>>>>>>>>> >>>> > I do see the benefit of implementing this feature > inside > >>>>>>>>>>>> of Flink if > >>>>>>>>>>>> >>>> many > >>>>>>>>>>>> >>>> > users need it. If not, then it might be easier for the > >>>>>>>>>>>> project to not > >>>>>>>>>>>> >>>> > increase the surface area since it makes the overall > >>>>>>>>>>>> maintenance > >>>>>>>>>>>> >>>> harder. > >>>>>>>>>>>> >>>> > > >>>>>>>>>>>> >>>> > Cheers, > >>>>>>>>>>>> >>>> > Till > >>>>>>>>>>>> >>>> > > >>>>>>>>>>>> >>>> > On Mon, May 31, 2021 at 4:57 PM Márton Balassi < > >>>>>>>>>>>> mbala...@apache.org> > >>>>>>>>>>>> >>>> wrote: > >>>>>>>>>>>> >>>> > > >>>>>>>>>>>> >>>> >> Hi team, > >>>>>>>>>>>> >>>> >> > >>>>>>>>>>>> >>>> >> Firstly I would like to introduce Gabor or G [1] for > >>>>>>>>>>>> short to the > >>>>>>>>>>>> >>>> >> community, he is a Spark committer who has recently > >>>>>>>>>>>> transitioned to > >>>>>>>>>>>> >>>> the > >>>>>>>>>>>> >>>> >> Flink Engineering team at Cloudera and is looking > >>>>>>>>>>>> forward to > >>>>>>>>>>>> >>>> contributing > >>>>>>>>>>>> >>>> >> to Apache Flink. Previously G primarily focused on > >>>>>>>>>>>> Spark Streaming > >>>>>>>>>>>> >>>> and > >>>>>>>>>>>> >>>> >> security. > >>>>>>>>>>>> >>>> >> > >>>>>>>>>>>> >>>> >> Based on requests from our customers G has implemented > >>>>>>>>>>>> Kerberos and > >>>>>>>>>>>> >>>> HTTP > >>>>>>>>>>>> >>>> >> Basic Authentication for the Flink Dashboard and > >>>>>>>>>>>> HistoryServer. > >>>>>>>>>>>> >>>> Previously > >>>>>>>>>>>> >>>> >> lacked an authentication story. > >>>>>>>>>>>> >>>> >> > >>>>>>>>>>>> >>>> >> We are looking to contribute this functionality back > to > >>>>>>>>>>>> the > >>>>>>>>>>>> >>>> community, we > >>>>>>>>>>>> >>>> >> believe that given Flink's maturity there should be a > >>>>>>>>>>>> common code > >>>>>>>>>>>> >>>> solution > >>>>>>>>>>>> >>>> >> for this general pattern. > >>>>>>>>>>>> >>>> >> > >>>>>>>>>>>> >>>> >> We are looking forward to your feedback on G's design. > >>>>>>>>>>>> [2] > >>>>>>>>>>>> >>>> >> > >>>>>>>>>>>> >>>> >> [1] http://gaborsomogyi.com/ > >>>>>>>>>>>> >>>> >> [2] > >>>>>>>>>>>> >>>> >> > >>>>>>>>>>>> >>>> >> > >>>>>>>>>>>> >>>> > >>>>>>>>>>>> > https://docs.google.com/document/d/1NMPeJ9H0G49TGy3AzTVVJVKmYC0okwOtqLTSPnGqzHw/edit > >>>>>>>>>>>> >>>> >> > >>>>>>>>>>>> >>>> > >>>>>>>>>>>> >>>> > >>>>>>>>>>>> > >>>>>>>>>>> > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
