Upleveling my high level feedback from the doc in case others have thoughts:
I'm a little skeptical about baking specific auth logic mostly needed for Kafka into the core worker logic, I wonder if we could make this easier without going this far - one option would be to provide a templated dockerfile example users can easily use to build their container. Or we could try to make something more generic to allow users to override envs (doing this well seems hard though). We could also build a function users could call as part of JVMInitializer which takes care of the messy details there (I am leaning in this direction since it is opt-in, pretty easy to use, and fits into our existing flows). In general, I'm +1 to making this space easier, but I think it is not a universal enough problem to justify a lot of worker setup logic for all users, or for maintaining separate worker infrastructure. I am, however, curious if others have run into issues in this space or if folks have thoughts on how to solve this more generically. Lastly, noting that this is loosely related to https://lists.apache.org/thread/8r8mq601ztlwo37qvm2ycx8q7z6cv2op; there may be a trend of needing more control over the environment without wanting to go through the whole custom container flow. We also may need something similar for x-lang containers. Thanks, Danny On Mon, Dec 16, 2024 at 7:28 AM Radek Stankiewicz via dev < dev@beam.apache.org> wrote: > Hi everyone, > > Kerberizing workers is not an easy thing for many users, especially for > Python users where some of the transforms are implemented via expansion > service. > I've drafted some ideas on how we could simplify this process - please > read this design doc > <https://docs.google.com/document/d/1T3Py6VZhP-FNQMjiURj38ddZyhWQRa_vDEUEc4f1P5A/edit?tab=t.0> > . > > Looking for your feedback - maybe you have a better way of configuring > kerberos or do you see some scenario where this design won't work with your > IOs you have used. > > Thank you, > Radek > > > > >
