This is an automated email from the ASF dual-hosted git repository.
xxyu pushed a change to branch kylin-on-parquet-v2
in repository https://gitbox.apache.org/repos/asf/kylin.git.
omit 655c129 [maven-release-plugin] prepare for next development iteration
omit 3405ab2 KYLIN-4719 Refine kylin-defaults.properties for parquet
Storage
omit 63f8c84 KYLIN-4705 Monitor Spark's AppMaster for QueryServer - Part II
omit 276f906 KYLIN-4747 Use the first dimension column as sort column
within a partiton
omit 0da41e2 KYLIN-4746 Improve build performance by reducing the count of
calling 'count()' function
omit 1147553 KYLIN-4660 Cleanup
omit 582969d KYLIN-4744 Add tracking URL for build spark job on yarn
omit 762e7c8 KYLIN-4742 NullPointerException when auto merge segments if
exist discard jobs
omit be0aedd KYLIN-4449 The spark process is not actually killed after the
job be paused
omit e62ca6e KYLIN-4725 NSparkCubingStep returns error state when pause
build job
omit 24fabdc HOTFIX Fix test cases for merging segments
omit 5eb9b37 KYLIN-4732 The cube size is wrong after disabling/enabling
the cube
omit f48a032 KYLIN-4733 The cube size is inconsistent with the size of all
segments
omit e573ed3 KYLIN-4705 add rest api for sparder-context url monitor
omit 1b63cec KYLIN-4730 Add scan bytes metric to the query results
omit 07fd1b4 KYLIN-4724 Support to download Apache Spark 2.4.6 binary
package in download-spark.sh
omit 698af2c KYLIN-4723 Set the configurations about shard by to cube level
omit 2d6d8b5 KYLIN-4695 Rewrite application id to file when restart
sparder application
omit ea267e2 KYLIN-4704 Add 'PERCENTILE' measure for sample metadata
omit 2d998f1 KYLIN-4695 Automatically start sparder (for query)
application when start kylin instance
omit 621868c KYLIN-4660 remove deprecated config
'spark.yarn.executor.memoryOverhead'
omit ff338de KYLIN-2971 Fix the wrong "Realization Names" and missing
"Cuboid Ids" in logQuery when hit cache
omit cfdcb9f KYLIN-4699 Delete job_tmp path after build/merge successfully
omit 56cf4e6 KYLIN-4698 Delete segment storage path after merging segment,
deleting segment and droping cube
omit 7a336fa KYLIN-4660 Remove duplicated/misleading code or comment
omit d43aca4 KYLIN-4721 The default source source type should be CSV not
Hive with the local debug mode
omit 65ad984 KYLIN-4722 Add more statistics to the query results
omit 3f0b2a8 KYLIN-4713 Support use diff spark schedule pool for diff query
omit ccdff32 KYLIN-4660 Add comparison with prepared data set
omit 656d9f3 KYLIN-4660 Cleanup for IT and logger
omit 35308f7 KYLIN-4701 Front end change for Parquet Storage
omit 14b5750 KYLIN-4705 add rest api for sparder-context url monitor
omit c9370ea KYLIN-4704 Upgrade sample metadata
omit 1b4e4c2 KYLIN-4694 Fix 'NoClassDefFoundError:
Lcom/esotericsoftware/kryo/io/Output' when query with sparder on yarn
omit 47dd4a2 KYLIN-4680 Avoid annoying log messages of query module and
unit test
omit 8e6c90a KYLIN-4516 Fix 'next on empty iterator' error
omit 15305ea KYLIN-4662 Migrate from third-party Spark to offical Apache
Spark
omit a2e9bfa KYLIN-4516 support System cube
omit 3d14e97 KYLIN-4680 Avoid annoying log messages of unit test and
integration test
omit dc1a80f KYLIN-4660 fix package.sh
omit 5764e47 KYLIN-4660 Normalize variable and file naming
omit 4bfaf99 KYLIN-4646 Fix merge error
omit 82045c2 KYLIN-4660 Fix UT and compile error
omit db866b7 KYLIN-4660 remove some unsupported features
omit db1eba8 KYLIN-4660 remove some useless modules
omit a8de6d4 KYLIN-4660 Create two module
new 97db36c KYLIN-4660 Remove some module from maven project
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (655c129)
\
N -- N -- N refs/heads/kylin-on-parquet-v2 (97db36c)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.gitignore | 3 +-
build-engine/pom.xml | 2 +-
build/bin/check-migration-acl.sh | 2 +-
build/bin/download-spark.sh | 16 +-
build/bin/kill-process-tree.sh | 56 -
build/bin/kylin.sh | 58 +-
build/conf/kylin-parquet-log4j.properties | 2 -
build/conf/kylin-server-log4j.properties | 18 +-
build/deploy/context.xml | 2 +-
build/{bin => script}/download-spark.sh | 16 +-
build/script/prepare-libs.sh | 8 +-
cache/pom.xml | 5 +-
core-common/pom.xml | 2 +-
.../org/apache/kylin/common/JobProcessContext.java | 10 -
.../java/org/apache/kylin/common/KylinConfig.java | 21 +-
.../org/apache/kylin/common/KylinConfigBase.java | 31 +-
.../java/org/apache/kylin/common/QueryContext.java | 48 +-
.../apache/kylin/common/debug/BackdoorToggles.java | 6 -
.../kylin/common/persistence/ResourceTool.java | 2 +-
.../src/main/resources/kylin-defaults.properties | 202 +-
.../kylin/common/util/AbstractKylinTestCase.java | 5 +
.../kylin/common/util/TempMetadataBuilder.java | 19 +-
core-cube/pom.xml | 2 +-
.../java/org/apache/kylin/cube/CubeManager.java | 2 +-
.../java/org/apache/kylin/cube/CubeSegment.java | 2 +-
.../java/org/apache/kylin/cube/model/CubeDesc.java | 23 +-
.../model/validation/rule/FunctionRuleTest.java | 17 +
core-dictionary/pom.xml | 2 +-
core-job/pom.xml | 2 +-
.../kylin/job/execution/AbstractExecutable.java | 13 +-
.../kylin/job/execution/ExecutableManager.java | 38 +-
.../org/apache/kylin/job/execution/Idempotent.java | 2 +-
...CardinalityExecutable.java => JobTypeEnum.java} | 16 +-
core-metadata/pom.xml | 2 +-
.../apache/kylin/measure/MeasureTypeFactory.java | 8 +-
.../apache/kylin/metadata/MetadataConstants.java | 11 +-
.../metadata/realization/RealizationType.java | 2 +-
core-metrics/pom.xml | 2 +-
core-storage/pom.xml | 2 +-
datasource-sdk/pom.xml | 6 +-
.../kylin/engine/spark/SparkUHCDictionary.java | 2 +-
examples/sample_cube/create_sample_tables.sql | 1 -
examples/sample_cube/data/DEFAULT.KYLIN_SALES.csv | 20000 +++++++++----------
.../template/cube_desc/kylin_sales_cube.json | 36 +-
.../template/model_desc/kylin_sales_model.json | 3 +-
.../table/DEFAULT.KYLIN_SALES--learn_kylin.json | 114 +-
.../test_kylin_cube_with_slr_left_join_desc.json | 4 +-
.../parquet_test/cube_desc/ci_left_join_cube.json | 21 +-
.../sample_local/table/DEFAULT.KYLIN_ACCOUNT.json | 2 +-
.../sample_local/table/DEFAULT.KYLIN_CAL_DT.json | 2 +-
.../table/DEFAULT.KYLIN_CATEGORY_GROUPINGS.json | 2 +-
.../sample_local/table/DEFAULT.KYLIN_COUNTRY.json | 2 +-
.../sample_local/table/DEFAULT.KYLIN_SALES.json | 2 +-
examples/test_case_data/sandbox/kylin.properties | 4 +-
jdbc/pom.xml | 2 +-
.../resources/query/sql_h2_uncapable/query03.sql | 5 +-
.../resources/query/sql_h2_uncapable/query04.sql | 6 +-
.../kylin-spark-classloader/pom.xml | 8 +-
...lassLoader.java => DebugTomcatClassLoader.java} | 8 +-
...catClassLoader.java => KylinItClassLoader.java} | 109 +-
...assLoader.java => KylinItSparkClassLoader.java} | 118 +-
.../kylin/spark/classloader/SparkClassLoader.java | 22 +-
.../kylin/spark/classloader/TomcatClassLoader.java | 84 +-
kylin-spark-project/kylin-spark-common/pom.xml | 8 +-
...nDateTimeUtils.scala => KapDateTimeUtils.scala} | 2 +-
.../engine/spark/metadata/cube/PathManager.java | 52 +-
.../org/apache/spark/dict/NBucketDictionary.java | 2 +-
.../spark/dict/NGlobalDictBuilderAssist.scala | 4 +-
...balDictionary.java => NGlobalDictionaryV2.java} | 8 +-
.../apache/spark/sql/KylinDataFrameManager.scala | 3 +-
.../org/apache/spark/sql/KylinFunctions.scala | 34 +-
.../sql/catalyst/expressions/DictEncodeImpl.scala | 4 +-
...linExpresssions.scala => KapExpresssions.scala} | 32 +-
.../catalyst/expressions/TimestampAddImpl.scala | 2 +-
.../catalyst/expressions/TimestampDiffImpl.scala | 10 +-
.../sql/execution/KylinFileSourceScanExec.scala | 5 +-
.../sql/execution/datasource/FilePruner.scala | 6 +-
.../datasource/ResetShufflePartition.scala | 9 +-
.../spark/sql/hive/utils/QueryMetricUtils.scala | 26 +-
.../main/scala/org/apache/spark/utils/LogEx.scala | 2 +-
kylin-spark-project/kylin-spark-engine/pom.xml | 26 +-
.../engine/spark/application/SparkApplication.java | 5 +-
.../engine/spark/builder/NBuildSourceInfo.java | 8 +
.../engine/spark/job}/CubingExecutableUtil.java | 2 +-
.../apache/kylin/engine/spark/job}/CubingJob.java | 54 +-
.../kylin/engine/spark/job/JobStepFactory.java | 12 +
.../apache/kylin/engine/spark/job/JobStepType.java | 4 +-
.../kylin/engine/spark/job/NSparkCubingJob.java | 11 +-
.../kylin/engine/spark/job/NSparkCubingStep.java | 25 -
.../kylin/engine/spark/job/NSparkCubingUtil.java | 18 +-
.../kylin/engine/spark/job/NSparkExecutable.java | 36 +-
.../kylin/engine/spark/job/NSparkMergingJob.java | 12 +-
.../NSparkUpdateMetaAndCleanupAfterMergeStep.java | 29 +-
.../merger/AfterMergeOrRefreshResourceMerger.java | 8 +-
.../kylin/engine/spark/merger/MetadataMerger.java | 2 +-
.../kylin/engine/spark/source/CsvSource.java | 6 +-
.../engine/spark/builder/CreateFlatTable.scala | 8 +-
...beBuilderHelper.scala => DFBuilderHelper.scala} | 4 +-
...naryBuilder.scala => DFDictionaryBuilder.scala} | 14 +-
...beMergeAssist.java => DFLayoutMergeAssist.java} | 4 +-
...apshotBuilder.scala => DFSnapshotBuilder.scala} | 2 +-
...CubeTableEncoder.scala => DFTableEncoder.scala} | 10 +-
.../kylin/engine/spark/builder/DictHelper.scala | 4 +-
.../spark/builder/DictionaryBuilderHelper.java | 4 +-
.../kylin/engine/spark/job/CubeBuildJob.java | 46 +-
.../kylin/engine/spark/job/CubeMergeJob.java | 44 +-
.../kylin/engine/spark/job/CuboidAggregator.scala | 2 +-
.../engine/spark/job/ParentSourceChooser.scala | 4 +-
.../spark/job/ResourceDetectBeforeMergingJob.java | 8 +-
.../kylin/engine/spark/utils/BuildUtils.scala | 5 +-
.../kylin/engine/spark/utils/JobMetricsUtils.scala | 11 +-
.../kylin/engine/spark/utils/Repartitioner.java | 26 +-
.../kylin/query/runtime/ExpressionConverter.scala | 4 +-
.../apache/spark/sql/udf/TimestampAddImpl.scala | 2 +-
.../apache/spark/sql/udf/TimestampDiffImpl.scala | 10 +-
.../engine/spark/LocalWithSparkSessionTest.java | 62 +-
...onaryTest.java => NGlobalDictionaryV2Test.java} | 16 +-
.../kylin/engine/spark/job/JobStepFactoryTest.java | 11 +-
.../kylin/engine/spark/job/SparkCubingJobTest.java | 3 +-
.../kylin/engine/spark/source/CsvSourceTest.java | 2 +-
.../src/test/resources/log4j.properties | 35 -
.../engine/spark/builder/TestCreateFlatTable.scala | 2 +-
.../engine/spark/builder/TestGlobalDictBuild.scala | 6 +-
.../engine/spark/builder/TestSnapshotBuilder.scala | 6 +-
.../kylin/engine/spark/job/TestCubeBuildJob.scala | 6 +-
.../kylin/engine/spark/job/TestUdfManager.scala | 2 +-
kylin-spark-project/kylin-spark-metadata/pom.xml | 6 +-
kylin-spark-project/kylin-spark-query/pom.xml | 13 +-
.../kylin/query/pushdown/SparkSqlClient.scala | 12 +-
.../kylin/query/runtime/DerivedProcess.scala | 1 +
.../kylin/query/runtime/ExpressionConverter.scala | 4 +-
.../apache/kylin/query/runtime/RuntimeHelper.scala | 1 +
.../kylin/query/runtime/SparderRexVisitor.scala | 4 +-
.../apache/kylin/query/runtime/SparkEngine.java | 12 +-
.../kylin/query/runtime/plans/ProjectPlan.scala | 4 +-
.../kylin/query/runtime/plans/ResultPlan.scala | 37 +-
.../org/apache/spark/sql/SparderContext.scala | 36 +-
.../pushdown/PushDownRunnerSparkImplTest.java | 10 +-
kylin-spark-project/kylin-spark-test/pom.xml | 113 +-
.../apache/kylin/engine/spark2/NExecAndComp.java | 73 +-
.../spark2/NManualBuildAndQueryCuboidTest.java | 4 +-
...ueryTest.java => NManualBuildAndQueryTest.java} | 108 +-
.../spark2/file_pruning/NFilePruningTest.java | 4 +-
.../kylin/engine/spark2/utils/QueryUtil.java | 26 +-
kylin-spark-project/pom.xml | 6 +-
metastore-hbase/pom.xml | 7 +-
metrics-reporter-hive/pom.xml | 7 +-
.../lib/impl}/hive/HiveMetaStoreClientFactory.java | 2 +-
.../kylin/metrics/lib/impl/hive/HiveProducer.java | 1 -
metrics-reporter-kafka/pom.xml | 2 +-
parquet-assembly/pom.xml | 6 +-
pom.xml | 20 +-
query/pom.xml | 2 +-
.../query/relnode/OLAPToEnumerableConverter.java | 21 +-
.../org/apache/kylin/query/routing/Candidate.java | 1 +
.../apache/kylin/query/routing/RoutingRule.java | 2 +-
query/src/test/resources/log4j.properties | 2 +-
server-base/pom.xml | 11 +-
.../kylin/rest/controller/AdminController.java | 9 -
.../kylin/rest/controller/CubeController.java | 45 +-
.../kylin/rest/init/InitialSparderContext.java | 45 -
.../apache/kylin/rest/job/KylinHealthCheckJob.java | 4 +-
.../apache/kylin/rest/job/MetadataCleanupJob.java | 2 +-
.../{StorageResponse.java => HBaseResponse.java} | 4 +-
.../apache/kylin/rest/response/SQLResponse.java | 40 -
.../kylin/rest/service/BadQueryDetector.java | 2 +-
.../org/apache/kylin/rest/service/CubeService.java | 59 +-
.../apache/kylin/rest/service/HBaseInfoUtil.java | 6 +-
.../org/apache/kylin/rest/service/JobService.java | 4 +-
.../apache/kylin/rest/service/QueryService.java | 46 +-
.../kylin/rest/job/MetadataCleanupJobTest.java | 2 +-
.../kylin/rest/response/SQLResponseTest.java | 3 +-
server/pom.xml | 2 +-
server/src/main/resources/applicationContext.xml | 1 -
.../kylin/rest/service/AdminServiceTest.java | 4 +-
source-hive/pom.xml | 12 +-
.../apache/kylin/source/hive/HiveFlinkInput.java | 92 +
.../apache/kylin/source/hive/HiveInputBase.java | 57 +
.../org/apache/kylin/source/hive/HiveSource.java | 13 +-
.../apache/kylin/source/hive/HiveSparkInput.java | 18 +-
tool-assembly/pom.xml | 2 +-
tool/pom.xml | 2 +-
.../tool/metrics/systemcube/CubeDescCreator.java | 2 +-
webapp/app/js/controllers/admin.js | 16 +-
webapp/app/js/controllers/cube.js | 4 +-
webapp/app/js/controllers/cubes.js | 2 -
webapp/app/js/filters/filter.js | 4 +-
webapp/app/js/model/cubeConfig.js | 2 +-
webapp/app/js/services/admin.js | 3 +-
webapp/app/js/services/cubes.js | 2 +-
webapp/app/partials/admin/admin.html | 6 +-
webapp/app/partials/admin/config.html | 3 -
.../partials/cubeDesigner/advanced_settings.html | 722 +-
webapp/app/partials/cubes/cube_detail.html | 2 +-
webapp/app/partials/cubes/cubes.html | 2 +-
webapp/app/partials/tables/source_table_tree.html | 4 +-
196 files changed, 11913 insertions(+), 11966 deletions(-)
delete mode 100644 build/bin/kill-process-tree.sh
copy build/{bin => script}/download-spark.sh (76%)
copy
core-job/src/main/java/org/apache/kylin/job/execution/{CardinalityExecutable.java
=> JobTypeEnum.java} (85%)
copy
kylin-spark-project/kylin-spark-classloader/src/main/java/org/apache/kylin/spark/classloader/{TomcatClassLoader.java
=> DebugTomcatClassLoader.java} (94%)
copy
kylin-spark-project/kylin-spark-classloader/src/main/java/org/apache/kylin/spark/classloader/{TomcatClassLoader.java
=> KylinItClassLoader.java} (53%)
copy
kylin-spark-project/kylin-spark-classloader/src/main/java/org/apache/kylin/spark/classloader/{SparkClassLoader.java
=> KylinItSparkClassLoader.java} (58%)
rename
kylin-spark-project/kylin-spark-common/src/main/java/org/apache/kylin/engine/spark/common/util/{KylinDateTimeUtils.scala
=> KapDateTimeUtils.scala} (99%)
rename
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/dict/{NGlobalDictionary.java
=> NGlobalDictionaryV2.java} (94%)
rename
kylin-spark-project/kylin-spark-common/src/main/scala/org/apache/spark/sql/catalyst/expressions/{KylinExpresssions.scala
=> KapExpresssions.scala} (93%)
copy {build-engine/src/main/java/org/apache/kylin/engine/mr/steps =>
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/job}/CubingExecutableUtil.java
(99%)
copy {build-engine/src/main/java/org/apache/kylin/engine/mr =>
kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/job}/CubingJob.java
(87%)
rename
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/{CubeBuilderHelper.scala
=> DFBuilderHelper.scala} (97%)
rename
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/{CubeDictionaryBuilder.scala
=> DFDictionaryBuilder.scala} (88%)
rename
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/{CubeMergeAssist.java
=> DFLayoutMergeAssist.java} (97%)
rename
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/{CubeSnapshotBuilder.scala
=> DFSnapshotBuilder.scala} (99%)
rename
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/{CubeTableEncoder.scala
=> DFTableEncoder.scala} (87%)
rename
kylin-spark-project/kylin-spark-engine/src/test/java/org/apache/kylin/engine/spark/dict/{NGlobalDictionaryTest.java
=> NGlobalDictionaryV2Test.java} (89%)
delete mode 100644
kylin-spark-project/kylin-spark-engine/src/test/resources/log4j.properties
rename
kylin-spark-project/kylin-spark-test/src/test/java/org/apache/kylin/engine/spark2/{NBuildAndQueryTest.java
=> NManualBuildAndQueryTest.java} (77%)
rename {source-hive/src/main/java/org/apache/kylin/source =>
metrics-reporter-hive/src/main/java/org/apache/kylin/metrics/lib/impl}/hive/HiveMetaStoreClientFactory.java
(98%)
delete mode 100644
server-base/src/main/java/org/apache/kylin/rest/init/InitialSparderContext.java
rename
server-base/src/main/java/org/apache/kylin/rest/response/{StorageResponse.java
=> HBaseResponse.java} (97%)
create mode 100644
source-hive/src/main/java/org/apache/kylin/source/hive/HiveFlinkInput.java
copy
source-jdbc/src/main/java/org/apache/kylin/source/jdbc/JdbcHiveSparkInput.java
=> source-hive/src/main/java/org/apache/kylin/source/hive/HiveSparkInput.java
(71%)
