Hi Max, it is not possible to deactivate spilling to disk at the moment. It might be possible to implement, but this would require a few more changes to make it feasible. For instance, we would need to add more fine-grained control about how memory is distributed among operators. This is currently statically decided by the optimizer and there is no parameter to influence it.
Cheers, Fabian 2016-05-04 13:46 GMT+02:00 Maximilian Bode <maximilian.b...@tngtech.com>: > Hi everyone, > > is there a way to prevent operators from spilling to disk? If not, would > it be conceivable to make this configurable either per job or operator? > > The use case is a batch application with the formal requirement not to > persist in-flight data to disk (even temporarily) so it would be preferable > to see the job fail and then be able to grant sufficient memory and run it > again. > > Cheers, > Max > > — > Maximilian Bode * Software Consultant * maximilian.b...@tngtech.com > TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke > Sitz: Unterföhring * Amtsgericht München * HRB 135082 > >
