My thoughts are, External shuffle service is not a blocker for spark on k8s to be a production ready.
Others may think otherwise, but there are other ways to enable auto scaling. External shuffle service feature will be useful for all the ways of deployments be it yarn, standalone, k8s and not just k8s. About the GA, I have not yet seen a very large deployment working yet. Others can share, how they are using spark on k8s, that can give us more confidence on moving towards GA. Thanks, On Thu, Aug 6, 2020 at 9:18 AM Holden Karau <hol...@pigscanfly.ca> wrote: > Sounds good. I think we can make a slightly stronger statement than that > one (left a comment, but it's my own thoughts so others should chime in if > they have a different opinion). > > On Wed, Aug 5, 2020 at 7:32 PM Takeshi Yamamuro <linguin....@gmail.com> > wrote: > >> Thanks for the info, all. okay, I understood that we need more time to >> announce GA officially. >> But, I'm still worried that users hesitate a bit to use this feature by >> referring to the statement in the doc, >> so how about updating it according to the current situation? Please check >> my suggestion in https://github.com/apache/spark/pull/29368. >> >> Anyway, many thanks! >> >> >> On Tue, Aug 4, 2020 at 12:26 AM Holden Karau <hol...@pigscanfly.ca> >> wrote: >> >>> There was discussion around removing the statement and declaring it GA >>> but I believe it was decided to leave it in until an external shuffle >>> service is supported on K8s. >>> >>> On Mon, Aug 3, 2020 at 2:45 AM JackyLee <qcsd2...@163.com> wrote: >>> >>>> +1. It has been worked well in our company and we has used it to support >>>> online services since March in this year. >>>> >>>> >>>> >>>> -- >>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>> >>>> -- >>> Twitter: https://twitter.com/holdenkarau >>> Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> >> >> >> -- >> --- >> Takeshi Yamamuro >> > > > -- > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau >
