Which shuffle operations trigger AQE and which don't?

I thought that AQE is triggered after every kind of shuffle operation. But it seems that it isn't. Is there a list of operations that trigger and don't trigger AQE? For example I noticed that repartition(partitionsNumber) does not trigger AQE.

When exactly does AQE coalesce small partitions or break up large partitions?

When exactly does AQE coalesce small partitions or break up large partitions? Say, I do df.repartition(2), and the data in this df is huge, will spark first repartition into 2 partitions (which may cause an OOM) and then break them up? Or will it make the necessary number of partitions according to

Re: startTimestamp doesn't work when using rate-micro-batch format

t; Dad | Technologist | Solutions Architect | Engineer > London > United Kingdom > > >view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* Use it

startTimestamp doesn't work when using rate-micro-batch format

I described the issue here: https://stackoverflow.com/questions/77893939/how-does-starttimestamp-option-work-for-the-rate-micro-batch-format Could someone please respond? The rate-micro-batch format doesn't seem to respect the startTimestamp option. Thanks.

How exactly does dropDuplicatesWithinWatermark work?

Hello, I have trouble understanding how dropDuplicatesWithinWatermark works. And I posted this stackoverflow question: https://stackoverflow.com/questions/77512507/how-exactly-does-dropduplicateswithinwatermark-work Could somebody answer it please? Best Regards, Pavel.