Re: Running Spark Rapids on GPU-Powered Spark Cluster

2021-07-31 Thread Gourav Sengupta
Hi Artemis, please do not insult people here, and give your personal opinions as well. Your comments are insulting to all big corporations which pay salaries and provide platforms for a lot of people here. Best of luck with your endeavors. Regards, Gourav Sengupta

Re: Running Spark Rapids on GPU-Powered Spark Cluster

2021-07-30 Thread Mich Talebzadeh
Hi, If I may say, from time to time we have had some disagreements in the forum myself included. However, we have all been here long enough to look for collaboration as opposed to going in tangent (no pun intended). So I repeat what our friend Artemis User requested originally for anyone with e

Re: Running Spark Rapids on GPU-Powered Spark Cluster

2021-07-30 Thread Artemis User
Gourav, with all respect, I really don't want to start a conversation about your political correctness.  I don't think my comments offend anyone in this group (including you) except big corporations.  Again, I am looking for concrete answers to my questions that can help me to get my project st

Re: Running Spark Rapids on GPU-Powered Spark Cluster

2021-07-30 Thread Gourav Sengupta
Hi Artemis, no one, and I repeat no one, is monopolising the data science market, in fact almost all algorithms and code and papers are available for free with largest open source contributions coming in from Amazon, Google, and Azure, who you are saying are trying to monopolise the market. I thi

Re: Running Spark Rapids on GPU-Powered Spark Cluster

2021-07-30 Thread Artemis User
Thanks Gourav for the info.  Actually I am looking for concrete experiences and detailed best practices from people who have build their own GPU-powered environment instead of relying on big cloud providers who are dominating and trying to monopolize the data science market -- ND On 7/30/

Re: Running Spark Rapids on GPU-Powered Spark Cluster

2021-07-30 Thread Gourav Sengupta
Hi, there are no cons of using SPARK with GPU's you just have to be careful about the GPU memory and a few other details. I have seen sometimes 10x improvement over general SPARK 3.x performance and sometimes around 30x. Not all the queries will be performant with GPU's and it is up to you to te