Spark on kubernete, configmap add log4j2.properties data

2025-06-23 Thread melin li
hello The spark on streaming task is running for a long time and has the need to dynamically adjust the log level (monitorInterval=30). It is more convenient to modify the log4j configuration through configmap.

Re: Spark 3.0 yarn does not support cdh5

2019-10-21 Thread melin li
Many clusters still use cdh5, and want to continue to support cdh5,cdh5 based on hadoop 2.6 melin li 于2019年10月21日周一 下午3:02写道: > 很多集群还是使用cdh5,希望继续支持cdh5,cdh5是基于hadoop 2.6 > > dev/make-distribution.sh --tgz -Pkubernetes -Pyarn -Phive-thriftserver > -Phive -Dhadoop.version=2.6

Spark 3.0 yarn does not support cdh5

2019-10-21 Thread melin li
很多集群还是使用cdh5,希望继续支持cdh5,cdh5是基于hadoop 2.6 dev/make-distribution.sh --tgz -Pkubernetes -Pyarn -Phive-thriftserver -Phive -Dhadoop.version=2.6.0-cdh5.15.0 -DskipTest ``` [INFO] Compiling 25 Scala sources to /Users/libinsong/Documents/codes/tongdun/spark-3.0/resource-managers/yarn/target/scala-2.12/

Orc upgrade to 1.6 to support zstd

2019-10-18 Thread melin li
Orc upgrade to 1.6 to support zstd https://issues.apache.org/jira/browse/ORC-363