andygrove commented on issue #44: URL: https://github.com/apache/datafusion-ray/issues/44#issuecomment-2480748656
I tried running locally rather than in k8s using `ray.init()` to create the cluster. The issue is that we are using too much object store memory. For TPC-H q2 @ 100GB, it consumes all the memory on my workstation (128 GB) and then crashed. I tried limiting object store memory with `ray.init(num_cpus=concurrency, object_store_memory=512 * 1024 * 1024)` and it ran longer, but is spilling huge amounts of data to disk and is taking an unreasonable amount of time. Here is an example where it is spilling a huge amount of data. ``` (raylet) Spilled 35419 MiB, 1062 objects, write throughput 1534 MiB/s. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
