Hi, From https://spark.apache.org/docs/3.1.1/cloud-integration.html#recommended-settings-for-writing-to-object-stores :
> For object stores whose consistency model means that rename-based commits are safe use the FileOutputCommitter v2 algorithm for performance; v1 for safety. These are "safe" and "safety" meanings. Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski "The Internals Of" Online Books <https://books.japila.pl/> Follow me on https://twitter.com/jaceklaskowski <https://twitter.com/jaceklaskowski> On Sat, Apr 3, 2021 at 7:49 PM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Hi Jacek, > > Can you please clarify your question? > > with regard to your point: > > "... I think I'm missing some fundamentals on how cloud object stores work > (GCS in particular) and hence the question." > > The end result is the safe storage of data in object storage in GCP right? > > HTH > > > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Sat, 3 Apr 2021 at 17:13, Jacek Laskowski <ja...@japila.pl> wrote: > >> Hi, >> >> I've just posted a question on StackOverflow [1] about the safety of the >> v2 algorithm while writing out to Google Cloud Storage. I think I'm missing >> some fundamentals on how cloud object stores work (GCS in particular) and >> hence the question. >> >> Is this all about File.rename and how many HTTP calls are there under the >> covers? How to know it for GCS? >> >> Thank you for any help you can provide. Merci beaucoup mes amis :) >> >> [1] https://stackoverflow.com/q/66933229/1305344 >> >> Pozdrawiam, >> Jacek Laskowski >> ---- >> https://about.me/JacekLaskowski >> "The Internals Of" Online Books <https://books.japila.pl/> >> Follow me on https://twitter.com/jaceklaskowski >> >> <https://twitter.com/jaceklaskowski> >> >
