dujl commented on PR #12188: URL: https://github.com/apache/doris/pull/12188#issuecomment-1239010649
> Some ideas to supoort the multi-version spark engine in spark load: > > 1. The spark load module is relatively independent and has the characteristics of low update frequency. So maybe we can just update and maintain the fat jar package. > 2. For each future version, e.g. 1.x.x, we only maintain and update the latest jar package. > 3. For the elder version(such as spark 2.x), we just package it into a jar and upload it to the official website. > 4. Based on 1 and 2, the spark-dpp build phase can skipped by default when building fe. Instead, the corresponding spark-dpp jar package can be download by enviroment variable url in `build.sh` and users can choose not to upgrade the spark-dpp jar package when upgrading fe. The spark runtime environment is very complex. Different user maybe has different version. So It would be great if we could provide flexible version dependencie. For case 1&2, There may be multiple spark versions in for a foreseeable period of timeļ¼Such as current mainstream version of spark includes spark2.x and spark3.x. So it is very great to provide spark 2.x and spark3.x dependencie with different compile profile. Like hudi/iceberg. For case 3, some user maybe need compile the spark-dpp with their own version which major version same as our the elder version(such as spark 2.x). It is very inconvenient if we can only provide binary packages on website. For case 4, agree with u to move spark-dpp out of the doris' core. But some compile env can not down package from the extranet My idea is: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
