Author: lidong
Date: Mon Aug 30 10:05:19 2021
New Revision: 1892729
URL: http://svn.apache.org/viewvc?rev=1892729&view=rev
Log:
Release Kylin 4.0.0
Modified:
kylin/site/cn/docs40/howto/howto_cleanup_storage.html
kylin/site/cn/download/index.html
kylin/site/docs40/howto/howto_cleanup_storage.html
kylin/site/docs40/release_notes.html
kylin/site/download/index.html
kylin/site/feed.xml
Modified: kylin/site/cn/docs40/howto/howto_cleanup_storage.html
URL:
http://svn.apache.org/viewvc/kylin/site/cn/docs40/howto/howto_cleanup_storage.html?rev=1892729&r1=1892728&r2=1892729&view=diff
==============================================================================
--- kylin/site/cn/docs40/howto/howto_cleanup_storage.html (original)
+++ kylin/site/cn/docs40/howto/howto_cleanup_storage.html Mon Aug 30 10:05:19
2021
@@ -187,25 +187,7 @@ var _hmt = _hmt || [];
<article
class="post-content" >
- <p>Kylin å¨æ建 cube
æé´ä¼å¨ HDFS ä¸çæ临æ¶æ件ï¼é¤æ¤ä¹å¤ï¼å½æ¸
ç/å
é¤/å并 cube æ¶ï¼ä¸äº parquet
æ件å¯è½è¢«éçä½æ¯ä»¥ååä¹ä¸ä¼è¢«æ¥è¯¢ï¼è½ç¶ Kylin
å·²ç»å¼å§åèªå¨åçåå¾åæ¶ï¼ä½ä¸ä¸å®è½è¦çå°ææçæ
åµï¼ä½ å¯ä»¥å®æå离线çåå¨æ¸
çï¼</p>
-
-<p>å¯ä»¥è¢«å é¤çæ件å
æ¬ï¼<br />
-- 临æ¶çä»»å¡æ件<br />
-<code
class="highlighter-rouge">hdfs:///kylin/${metadata_url}/${project}/job_tmp</code><br
/>
-- ä¸ä¼å被ç¨å°çsegmentçcuboidæ件<br />
-<code
class="highlighter-rouge">hdfs:///kylin/${metadata_url}/${project}/${cube_name}/${non_used_segment}
</code></p>
-
-<p>æ¥éª¤ï¼<br />
-1. æ£æ¥åªäºèµæºå¯ä»¥æ¸
çï¼è¿ä¸æ¥ä¸ä¼å é¤ä»»ä½ä¸è¥¿ï¼</p>
-
-<div class="highlight"><pre><code class="language-groff"
data-lang="groff">export KYLIN_HOME=/path/to/kylin_home
-${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete
false</code></pre></div>
-
-<p>请å°è¿éç (version) æ¿æ¢ä¸ºä½ å®è£
ç Kylin jar çæ¬ã<br />
-2. ä½
å¯ä»¥æ½æ¥ä¸ä¸¤ä¸ªèµæºæ¥æ£æ¥å®ä»¬æ¯å¦å·²ç»æ²¡æ被å¼ç¨äºï¼ç¶åå
ä¸ââdelete trueâé项è¿è¡æ¸
çã</p>
-
-<div class="highlight"><pre><code class="language-groff"
data-lang="groff">${KYLIN_HOME}/bin/kylin.sh
org.apache.kylin.tool.StorageCleanupJob --delete true</code></pre></div>
-
+ <p>Wiki :
https://cwiki.apache.org/confluence/display/KYLIN/How+to+clean+up+storage+in+Kylin+4</p>
</article>
</div>
Modified: kylin/site/cn/download/index.html
URL:
http://svn.apache.org/viewvc/kylin/site/cn/download/index.html?rev=1892729&r1=1892728&r2=1892729&view=diff
==============================================================================
--- kylin/site/cn/download/index.html (original)
+++ kylin/site/cn/download/index.html Mon Aug 30 10:05:19 2021
@@ -188,14 +188,15 @@ var _hmt = _hmt || [];
</header>
<p>æ¨å¯ä»¥æç
§è¿äº<a
href="https://www.apache.org/info/verification.html";>æ¥éª¤</a>
并使ç¨è¿äº<a
href="https://www.apache.org/dist/kylin/KEYS";>KEYS</a>æ¥éªè¯ä¸è½½æ件çæææ§.</p>
-<h4 id="v400-beta">v4.0.0-beta</h4>
+<h4 id="v400">v4.0.0</h4>
<ul>
- <li>è¿æ¯ 4.0.0-alpha çæ¬åçä¸ä¸ªä¸»è¦çæ¬ï¼å
å«25个æ°åè½ä»¥åæ¹è¿å14个é®é¢çä¿®å¤ãå
³äºå
·ä½å
容请æ¥çåå¸è¯´æã</li>
+ <li>è¿æ¯ Kylin 4 ç第ä¸ä¸ªæ£å¼çæ¬ï¼å
å«32个æ°åè½ä»¥åæ¹è¿å10个é®é¢çä¿®å¤ãå
³äºå
·ä½å
容请æ¥çåå¸è¯´æã</li>
<li><a href="/docs/release_notes.html">åå¸è¯´æ</a>, <a
href="https://cwiki.apache.org/confluence/display/KYLIN/Installation+Guide";>å®è£
æå</a> and <a href="/docs40/howto/howto_upgrade.html">å级æå</a></li>
- <li>æºç ä¸è½½: <a
href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-source-release.zip";>apache-kylin-4.0.0-beta-source-release.zip</a>
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-source-release.zip.asc";>asc</a>]
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-source-release.zip.sha256";>sha256</a>]</li>
- <li>Hadoop 2 å Hadoop 3 äºè¿å¶å
(请为 Kylin 4.x
使ç¨æå®çæ¬ç Sparkï¼çæ¬ä¸º Apache Spark 2.4.6,
èä¸æ¯ç¯å¢èªå¸¦ç Spark):
+ <li>æºç ä¸è½½: <a
href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-source-release.zip";>apache-kylin-4.0.0-source-release.zip</a>
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-source-release.zip.asc";>asc</a>]
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-source-release.zip.sha256";>sha256</a>]</li>
+ <li>äºè¿å¶å
(éæ©äºè¿å¶å
å请æ£æ¥ææ¡£ <a
href="https://cwiki.apache.org/confluence/display/KYLIN/Support+Hadoop+Version+Matrix+of+Kylin+4";>Hadoop
æ¯æç©éµ</a>):
<ul>
- <li><a
href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-bin.tar.gz";>apache-kylin-4.0.0-beta-bin.tar.gz</a>
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-bin.tar.gz.asc";>asc</a>]
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-bin.tar.gz.sha256";>sha256</a>]
(å·²ç»å¨ CDH 5.7, CDH 6.2, AWS EMR 5.31, AWS EMR 6.0.0, HDP 2.4
ç¯å¢ä¸éªè¯, Hadoop3 å EMR ç¯å¢éåé¢å¤é
ç½®ï¼è¯·<a
href="https://cwiki.apache.org/confluence/display/KYLIN/Installation+Guide";>æ¥çå®è£
æå</a>)</li>
+ <li>for Apache Spark 2.4.7 <a
href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark2.tar.gz";>apache-kylin-4.0.0-bin-spark2.tar.gz</a>
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark2.tar.gz.asc";>asc</a>]
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark2.tar.gz.sha256";>sha256</a>]</li>
+ <li>for Apache Spark 3.1.1 <a
href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark3.tar.gz";>apache-kylin-4.0.0-bin-spark3.tar.gz</a>
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark3.tar.gz.asc";>asc</a>]
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark3.tar.gz.sha256";>sha256</a>]</li>
</ul>
</li>
</ul>
Modified: kylin/site/docs40/howto/howto_cleanup_storage.html
URL:
http://svn.apache.org/viewvc/kylin/site/docs40/howto/howto_cleanup_storage.html?rev=1892729&r1=1892728&r2=1892729&view=diff
==============================================================================
--- kylin/site/docs40/howto/howto_cleanup_storage.html (original)
+++ kylin/site/docs40/howto/howto_cleanup_storage.html Mon Aug 30 10:05:19 2021
@@ -187,27 +187,7 @@ var _hmt = _hmt || [];
<article
class="post-content" >
- <p>Kylin will generate
intermediate files in HDFS during the cube building; Besides, when
purge/drop/merge cubes, some Parquet file may be left and will no longer be
queried; Although Kylin has started to do some <br />
-automated garbage collection, it might not cover all cases; You can do an
offline storage cleanup periodically:<br />
-Which can be deleted:<br />
-- temp job files<br />
-<code
class="highlighter-rouge">hdfs:///kylin/${metadata_url}/${project}/job_tmp</code><br
/>
-- none used segment cuboid files<br />
-<code
class="highlighter-rouge">hdfs:///kylin/${metadata_url}/${project}/${cube_name}/${non_used_segment}</code></p>
-
-<p>Steps:<br />
-1. Check which resources can be cleanup, this will not remove anything:</p>
-
-<div class="highlight"><pre><code class="language-groff"
data-lang="groff">export KYLIN_HOME=/path/to/kylin_home
-${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete
false</code></pre></div>
-
-<p>Here please replace (version) with the specific Kylin jar version in your
installation;<br />
-2. You can pickup 1 or 2 resources to check whether theyâre no longer be
referred; Then add the ââdelete trueâ option to start the cleanup:</p>
-
-<div class="highlight"><pre><code class="language-groff"
data-lang="groff">${KYLIN_HOME}/bin/kylin.sh
org.apache.kylin.tool.StorageCleanupJob --delete true</code></pre></div>
-
-<p>On finish, the temp job files and none used segment cuboid files should be
dropped;</p>
-
+ <p>Wiki :
https://cwiki.apache.org/confluence/display/KYLIN/How+to+clean+up+storage+in+Kylin+4</p>
</article>
</div>
Modified: kylin/site/docs40/release_notes.html
URL:
http://svn.apache.org/viewvc/kylin/site/docs40/release_notes.html?rev=1892729&r1=1892728&r2=1892729&view=diff
==============================================================================
--- kylin/site/docs40/release_notes.html (original)
+++ kylin/site/docs40/release_notes.html Mon Aug 30 10:05:19 2021
@@ -187,7 +187,7 @@ var _hmt = _hmt || [];
<article
class="post-content" >
- <p>To download latest
release, please visit: <a
href="http://kylin.apache.org/download/";>http://kylin.apache.org/download/</a>,
<br />
+ <p>To download the
latest release, please visit: <a
href="http://kylin.apache.org/download/";>http://kylin.apache.org/download/</a>,
<br />
there are source code package, binary package and installation guide
avaliable.</p>
<p>Any problem or issue, please report to Apache Kylin JIRA project: <a
href="https://issues.apache.org/jira/browse/KYLIN";>https://issues.apache.org/jira/browse/KYLIN</a></p>
@@ -199,6 +199,65 @@ there are source code package, binary pa
<li>Development relative: <a
href="mailto:dev@kylin.apache.org">dev@kylin.apache.org</a></li>
</ul>
+<h2 id="v400---2021-08-30">v4.0.0 - 2021-08-30</h2>
+
+<p><strong>New Feature</strong></p>
+
+<ul>
+ <li>[KYLIN-4498] - CubePlaner for Kylin on Parquet</li>
+ <li>[KYLIN-4895] - change spark deploy mode of kylin4.0 engine from local to
cluster</li>
+ <li>[KYLIN-4905] - Support limit .. offset ⦠in spark query engine</li>
+ <li>[KYLIN-4925] - Use Spark 3 as build and query engine for Kylin 4</li>
+ <li>[KYLIN-4948] - Provide an API to allow users to adjust cuboids
manually</li>
+ <li>[KYLIN-4966] - Refresh the existing segment according to the new cuboid
list in kylin4</li>
+ <li>[KYLIN-5011] - Detect and scatter skewed data in dict encoding step</li>
+ <li>[KYLIN-5019] - Avoid building global dictionary from all data of fact
table each time</li>
+ <li>[KYLIN-5059] - Fix error when using different HDFS cluster in cube
building</li>
+</ul>
+
+<p><strong>Bug Fix</strong></p>
+
+<ul>
+ <li>[KYLIN-4729] - The hive table will be overwrited when add csv table with
the same name</li>
+ <li>[KYLIN-4879] - The function of sql to remove comments is not perfect. In
some cases, the sql query conditions used will be modified</li>
+ <li>[KYLIN-4887] - Segment pruner support string type partition col in spark
query engine</li>
+ <li>[KYLIN-4889] - Query error when spark engine in local mode</li>
+ <li>[KYLIN-4935] - Existing JobOutputâs extend info will be lost when it
is updated</li>
+ <li>[KYLIN-4967] - Forbid to set âspark.sql.adaptive.enabledâ to true
when building cube with Spark 2.X</li>
+ <li>[KYLIN-5013] - Write table_snapshot to wrong cluster in Kylin4.0</li>
+ <li>[KYLIN-5014] - Spark driver log is abnormal in yarn cluster mode</li>
+ <li>[KYLIN-5021] - FilePruner in Spark3 throws NPE when no partition columns
in cube</li>
+ <li>[KYLIN-5040] - Cuboid should not be exactly matched when there is no
group by time partition column and there are multiple segments in the query</li>
+</ul>
+
+<p><strong>Improvement</strong></p>
+
+<ul>
+ <li>[KYLIN-4554] - Validate âfilter conditionâ on model saving</li>
+ <li>[KYLIN-4888] - Performance optimization of union query with spark
engine</li>
+ <li>[KYLIN-4890] - Use numSlices = 1 to reduce task num when executing
sparder canary</li>
+ <li>[KYLIN-4892] - Reduce the times of fetching files status from HDFS in
FilePruner</li>
+ <li>[KYLIN-4893] - Optimize query performance when using shard by column</li>
+ <li>[KYLIN-4894] - Upgrade Apache Spark version to 2.4.7</li>
+ <li>[KYLIN-4897] - Add table snapshot and global dictionary cleaning in
StorageCleanupJob</li>
+ <li>[KYLIN-4898] - Add automated test cases</li>
+ <li>[KYLIN-4903] - cache parent datasource to accelerate next layerâs
cuboid building</li>
+ <li>[KYLIN-4906] - support query/job server dynamic register and discovery
in kylin4</li>
+ <li>[KYLIN-4908] - Segment pruner support integer partition col in spark
query engine</li>
+ <li>[KYLIN-4910] - Return hostname as Sparder URL address when spark master
is set to local</li>
+ <li>[KYLIN-4917] - Fix some problem of logger system in kylin4</li>
+ <li>[KYLIN-4923] - CubeMigration Tools support migrate meta from 2.x/3.x
cluster to 4.0 cluster</li>
+ <li>[KYLIN-4926] - Optimize Global Dict building: replace operation
âmapPartitions.count()â with âforeachPartitionsâ</li>
+ <li>[KYLIN-4927] - Forbid to use AE when building Global Dict</li>
+ <li>[KYLIN-4936] - Exactly aggregation canât transform to project</li>
+ <li>[KYLIN-4937] - Verify the uniqueness of the global dictionary after
building global dictionary</li>
+ <li>[KYLIN-4944] - Upgrade CentOS version, Hadoop version and Spark version
for Kylin Docker image</li>
+ <li>[KYLIN-4945] - Repartition encoded dataset to avoid data skew caused by
a single column</li>
+ <li>[KYLIN-4980] - Support prunning segments from complex filter
conditions</li>
+ <li>[KYLIN-5027] - Add the config of whether to build base cuboid in
kylin4</li>
+ <li>[KYLIN-5031] - The last_build_job_id of segment is null when the semgent
status is RUNNING or ERROR.</li>
+</ul>
+
<h2 id="v312---2021-04-26">v3.1.2 - 2021-04-26</h2>
<p><strong>New Feature</strong></p>
Modified: kylin/site/download/index.html
URL:
http://svn.apache.org/viewvc/kylin/site/download/index.html?rev=1892729&r1=1892728&r2=1892729&view=diff
==============================================================================
--- kylin/site/download/index.html (original)
+++ kylin/site/download/index.html Mon Aug 30 10:05:19 2021
@@ -193,14 +193,15 @@ var _hmt = _hmt || [];
</header>
<p>You can verify the download by following these <a
href="https://www.apache.org/info/verification.html";>procedures</a> and using
these <a href="https://www.apache.org/dist/kylin/KEYS";>KEYS</a>.</p>
-<h4 id="v400-beta">v4.0.0-beta</h4>
+<h4 id="v400">v4.0.0</h4>
<ul>
- <li>This is a major release after 4.0.0-alpha, with 25 new
features/improvements and 14 bug fixes. Check the release notes.</li>
+ <li>This is the first GA release for Kylin 4, with 32 new
features/improvements and 10 bug fixes. Check the release notes.</li>
<li><a href="/docs/release_notes.html">Release notes</a>, <a
href="https://cwiki.apache.org/confluence/display/KYLIN/Installation+Guide";>installation
guide</a> and <a
href="https://cwiki.apache.org/confluence/display/KYLIN/How+to+upgrade";>upgrade
guide</a></li>
- <li>Source download: <a
href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-source-release.zip";>apache-kylin-4.0.0-beta-source-release.zip</a>
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-source-release.zip.asc";>asc</a>]
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-source-release.zip.sha256";>sha256</a>]</li>
- <li>Binary for Apache Hadoop 2 and Hadoop 3 download (Please use the
specified version Spark for Kylin 4.X, the version should be Apache Spark
2.4.6, not the Spark that provided by the environment):
+ <li>Source download: <a
href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-source-release.zip";>apache-kylin-4.0.0-source-release.zip</a>
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-source-release.zip.asc";>asc</a>]
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-source-release.zip.sha256";>sha256</a>]</li>
+ <li>Binary for the download (check this to see which binary you should
choose <a
href="https://cwiki.apache.org/confluence/display/KYLIN/Support+Hadoop+Version+Matrix+of+Kylin+4";>Hadoop
Matrix supported</a>) :
<ul>
- <li><a
href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-bin.tar.gz";>apache-kylin-4.0.0-beta-bin.tar.gz</a>
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-bin.tar.gz.asc";>asc</a>]
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0-beta/apache-kylin-4.0.0-beta-bin.tar.gz.sha256";>sha256</a>]
(Verified on CDH 5.7, CDH 6.2, AWS EMR 5.31, AWS EMR 6.0.0, HDP 2.4, Hadoop 3
and EMR environments require additional configuration, please check the <a
href="https://cwiki.apache.org/confluence/display/KYLIN/Installation+Guide";>installation
guide</a>)</li>
+ <li>for Apache Spark 2.4.7 <a
href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark2.tar.gz";>apache-kylin-4.0.0-bin-spark2.tar.gz</a>
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark2.tar.gz.asc";>asc</a>]
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark2.tar.gz.sha256";>sha256</a>]</li>
+ <li>for Apache Spark 3.1.1 <a
href="https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark3.tar.gz";>apache-kylin-4.0.0-bin-spark3.tar.gz</a>
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark3.tar.gz.asc";>asc</a>]
[<a
href="https://www.apache.org/dist/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin-spark3.tar.gz.sha256";>sha256</a>]</li>
</ul>
</li>
</ul>
Modified: kylin/site/feed.xml
URL:
http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1892729&r1=1892728&r2=1892729&view=diff
==============================================================================
--- kylin/site/feed.xml (original)
+++ kylin/site/feed.xml Mon Aug 30 10:05:19 2021
@@ -19,8 +19,8 @@
<description>Apache Kylin Home</description>
<link>http://kylin.apache.org/</link>
<atom:link href="http://kylin.apache.org/feed.xml"; rel="self"
type="application/rss+xml"/>
- <pubDate>Mon, 23 Aug 2021 06:59:11 -0700</pubDate>
- <lastBuildDate>Mon, 23 Aug 2021 06:59:11 -0700</lastBuildDate>
+ <pubDate>Mon, 30 Aug 2021 02:47:11 -0700</pubDate>
+ <lastBuildDate>Mon, 30 Aug 2021 02:47:11 -0700</lastBuildDate>
<generator>Jekyll v2.5.3</generator>
<item>
