submitting tasks failed in Spark standalone mode due to missing failureaccess jar file

2023-10-23 Thread eab...@163.com
Hi Team. I use spark 3.5.0 to start Spark cluster with start-master.sh and start-worker.sh, when I use ./bin/spark-shell --master spark://LAPTOP-TC4A0SCV.:7077 and get error logs: ``` 23/10/24 12:00:46 ERROR TaskSchedulerImpl: Lost an executor 1 (already removed): Command exited with code

Re: Hive's performance for querying the Iceberg table is very poor.

2023-10-23 Thread Ayush Saxena
Hi Lisoda, The iceberg jar for hive 3.1.3 doesn't have a lot of changes, We did a bunch of improvements on the 4.x line for Hive-Iceberg. You can give iceberg a try on the 4.0.0-beta-1 release mentioned here [1], we have a bunch of improvements like vecotrization and stuff like that. If you wanna

Re: Hive's performance for querying the Iceberg table is very poor.

2023-10-23 Thread Albert Wong
Too bad. Tencent Games used StarRocks with Apache Iceberg to power their analytics. https://medium.com/starrocks-engineering/tencent-games-inside-scoop-the-road-to-cloud-native-with-starrocks-d7dcb2438e25. On Mon, Oct 23, 2023 at 10:55 AM lisoda wrote: > We are not going to use starrocks. > m

Re: Hive's performance for querying the Iceberg table is very poor.

2023-10-23 Thread lisoda
We are not going to use starrocks. mpp architecture databases have natural limitations, and starrocks does not necessarily perform better than hive llap. Replied Message | From | Albert Wong | | Date | 10/24/2023 01:39 | | To | user@hive.apache.org | | Cc | | | Subject | Re: Hive's pe

Re: Hive's performance for querying the Iceberg table is very poor.

2023-10-23 Thread Albert Wong
I would try http://starrocks.io. StarRocks is an MPP OLAP database that can query Apache Iceberg and we can cache the data for faster performance. We also have additional features like building materialized views that span across Apache Iceberg, Apache Hudi and Apache Hive. Here is a video of c

Hive's performance for querying the Iceberg table is very poor.

2023-10-23 Thread lisoda
Hi Team. I recently was testing Hive query Iceberg table , I found that Hive query Iceberg table performance is very very poor . Almost impossible to use in the production environment . And Join conditions can not be pushed down to the Iceberg partition. I'm using the 1.3.1 Hive Runt