This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 66f1dda [BDBJE] Add a tool to view the data in BEBJE (#6209)
66f1dda is described below
commit 66f1ddaa7220f3dc53d6544d6b0a73e827a1cf98
Author: Mingyu Chen <[email protected]>
AuthorDate: Sat Jul 17 10:45:51 2021 +0800
[BDBJE] Add a tool to view the data in BEBJE (#6209)
When the config "enable_bdbje_debug_mode" of FE is set to true,
start FE and enter debug mode.
In this mode, only MySQL server and http server will be started.
After that, users can log in to Doris through the web front-end or MySQL
client,
and then use "show proc "/bdbje"" to view the data in bdbje.
Co-authored-by: chenmingyu <[email protected]>
---
.../operation/metadata-operation.md | 53 +++++
.../operation/metadata-operation.md | 54 +++++
.../src/main/java/org/apache/doris/PaloFe.java | 7 +
.../doris/common/proc/BDBJEDatabaseProcDir.java | 71 ++++++
.../common/proc/BDBJEJournalDataProcNode.java | 60 +++++
.../org/apache/doris/common/proc/BDBJEProcDir.java | 67 ++++++
.../org/apache/doris/common/proc/ProcService.java | 1 +
.../org/apache/doris/http/action/SystemAction.java | 9 +-
.../apache/doris/journal/bdbje/BDBDebugger.java | 253 +++++++++++++++++++++
.../org/apache/doris/persist/OperationType.java | 23 ++
10 files changed, 594 insertions(+), 4 deletions(-)
diff --git a/docs/en/administrator-guide/operation/metadata-operation.md
b/docs/en/administrator-guide/operation/metadata-operation.md
index f3cd74e..c5f6ced 100644
--- a/docs/en/administrator-guide/operation/metadata-operation.md
+++ b/docs/en/administrator-guide/operation/metadata-operation.md
@@ -277,6 +277,59 @@ curl -u $root_user:$password
http://$master_hostname:8030/dump
**Note: If the Image file is large, the entire process can take a long time,
so during this time, make sure Master FE does not generate a new image file via
checkpoint. When the image.ckpt file in the meta_dir/image directory on the
Master FE node is observed to be as large as the image.xxx file, the image.ckpt
file can be deleted directly.**
+### View data in BDBJE
+
+The metadata log of FE is stored in BDBJE in the form of Key-Value. In some
abnormal situations, FE may not be started due to metadata errors. In this
case, Doris provides a way to help users query the data stored in BDBJE to
facilitate troubleshooting.
+
+First, you need to add configuration in fe.conf:
`enable_bdbje_debug_mode=true`, and then start FE through `sh start_fe.sh
--daemon`.
+
+At this time, FE will enter the debug mode, only start the http server and
MySQL server, and open the BDBJE instance, but will not load any metadata and
other subsequent startup processes.
+
+This is, we can view the data stored in BDBJE by visiting the web page of FE,
or after connecting to Doris through the MySQL client, through `show proc
/bdbje;`.
+
+```
+mysql> show proc "/bdbje";
++----------+---------------+---------+
+| DbNames | JournalNumber | Comment |
++----------+---------------+---------+
+| 110589 | 4273 | |
+| epochDB | 4 | |
+| metricDB | 430694 | |
++----------+---------------+---------+
+```
+
+The first level directory will display all the database names in BDBJE and the
number of entries in each database.
+
+```
+mysql> show proc "/bdbje/110589";
++-----------+
+| JournalId |
++-----------+
+| 1 |
+| 2 |
+
+...
+| 114858 |
+| 114859 |
+| 114860 |
+| 114861 |
++-----------+
+4273 rows in set (0.06 sec)
+```
+
+Entering the second level, all the entry keys under the specified database
will be listed.
+
+```
+mysql> show proc "/bdbje/110589/114861";
++-----------+--------------+---------------------------------------------+
+| JournalId | OpType | Data |
++-----------+--------------+---------------------------------------------+
+| 114861 | OP_HEARTBEAT | org.apache.doris.persist.HbPackage@6583d5fb |
++-----------+--------------+---------------------------------------------+
+1 row in set (0.05 sec)
+```
+
+The third level can display the value information of the specified key.
## Best Practices
diff --git a/docs/zh-CN/administrator-guide/operation/metadata-operation.md
b/docs/zh-CN/administrator-guide/operation/metadata-operation.md
index 0279aa3..995f10f 100644
--- a/docs/zh-CN/administrator-guide/operation/metadata-operation.md
+++ b/docs/zh-CN/administrator-guide/operation/metadata-operation.md
@@ -278,6 +278,60 @@ curl -u $root_user:$password
http://$master_hostname:8030/dump
**注意：如果 Image 文件很大，整个操作过程耗时可能会很长，所以在此期间，要确保 Master FE 不会通过 checkpoint 生成新的
image 文件。
当观察到 Master FE 节点上 `meta_dir/image`目录下的 `image.ckpt` 文件快和 `image.xxx`
文件一样大时，可以直接删除掉`image.ckpt` 文件。**
+### 查看 BDBJE 中的数据
+
+FE 的元数据日志以 Key-Value 的方式存储在 BDBJE 中。某些异常情况下，可能因为元数据错误而无法启动 FE。在这种情况下，Doris
提供一种方式可以帮助用户查询 BDBJE 中存储的数据，以方便进行问题排查。
+
+首先需在 fe.conf 中增加配置：`enable_bdbje_debug_mode=true`，之后通过 `sh start_fe.sh
--daemon` 启动 FE。
+
+此时，FE 将进入 debug 模式，仅会启动 http server 和 MySQL server，并打开 BDBJE
实例，但不会进行任何元数据的加载及后续其他启动流程。
+
+这是，我们可以通过访问 FE 的 web 页面，或通过 MySQL 客户端连接到 Doris 后，通过 `show proc /bdbje;` 来查看
BDBJE 中存储的数据。
+
+```
+mysql> show proc "/bdbje";
++----------+---------------+---------+
+| DbNames | JournalNumber | Comment |
++----------+---------------+---------+
+| 110589 | 4273 | |
+| epochDB | 4 | |
+| metricDB | 430694 | |
++----------+---------------+---------+
+```
+
+第一级目录会展示 BDBJE 中所有的 database 名称，以及每个 database 中的 entry 数量。
+
+```
+mysql> show proc "/bdbje/110589";
++-----------+
+| JournalId |
++-----------+
+| 1 |
+| 2 |
+
+...
+| 114858 |
+| 114859 |
+| 114860 |
+| 114861 |
++-----------+
+4273 rows in set (0.06 sec)
+```
+
+进入第二级，则会罗列指定 database 下的所有 entry 的 key。
+
+```
+mysql> show proc "/bdbje/110589/114861";
++-----------+--------------+---------------------------------------------+
+| JournalId | OpType | Data |
++-----------+--------------+---------------------------------------------+
+| 114861 | OP_HEARTBEAT | org.apache.doris.persist.HbPackage@6583d5fb |
++-----------+--------------+---------------------------------------------+
+1 row in set (0.05 sec)
+```
+
+第三级则可以展示指定 key 的 value 信息。
+
## 最佳实践
FE 的部署推荐，在 [安装与部署文档](../../installing/install-deploy.md) 中有介绍，这里再做一些补充。
diff --git a/fe/fe-core/src/main/java/org/apache/doris/PaloFe.java
b/fe/fe-core/src/main/java/org/apache/doris/PaloFe.java
index 44fbef3..ab45c7a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/PaloFe.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/PaloFe.java
@@ -25,6 +25,7 @@ import org.apache.doris.common.ThreadPoolManager;
import org.apache.doris.common.Version;
import org.apache.doris.common.util.JdkUtils;
import org.apache.doris.http.HttpServer;
+import org.apache.doris.journal.bdbje.BDBDebugger;
import org.apache.doris.journal.bdbje.BDBTool;
import org.apache.doris.journal.bdbje.BDBToolOptions;
import org.apache.doris.qe.QeService;
@@ -104,6 +105,12 @@ public class PaloFe {
FrontendOptions.init();
+ if (Config.enable_bdbje_debug_mode) {
+ // Start in BDB Debug mode
+ BDBDebugger.get().startDebugMode(dorisHomeDir);
+ return;
+ }
+
// init catalog and wait it be ready
Catalog.getCurrentCatalog().initialize(args);
Catalog.getCurrentCatalog().waitForReady();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/proc/BDBJEDatabaseProcDir.java
b/fe/fe-core/src/main/java/org/apache/doris/common/proc/BDBJEDatabaseProcDir.java
new file mode 100644
index 0000000..36b568f
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/proc/BDBJEDatabaseProcDir.java
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.proc;
+
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
+import org.apache.doris.journal.bdbje.BDBDebugger;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+
+// SHOW PROC "/bdbje/dbname/"
+public class BDBJEDatabaseProcDir implements ProcDirInterface {
+ public static final ImmutableList<String> TITLE_NAMES = new
ImmutableList.Builder<String>()
+ .add("JournalId").build();
+
+ private String dbName;
+
+ public BDBJEDatabaseProcDir(String dbName){
+ this.dbName = dbName;
+ }
+
+ @Override
+ public boolean register(String name, ProcNodeInterface node) {
+ return false;
+ }
+
+ @Override
+ public ProcNodeInterface lookup(String journalId) throws AnalysisException
{
+ return new BDBJEJournalDataProcNode(dbName, Long.valueOf(journalId));
+ }
+
+ @Override
+ public ProcResult fetchResult() throws AnalysisException {
+ if (!Config.enable_bdbje_debug_mode) {
+ throw new AnalysisException("Not in bdbje debug mode");
+ }
+
+ BaseProcResult result = new BaseProcResult();
+ result.setNames(TITLE_NAMES);
+
+ BDBDebugger.BDBDebugEnv env = BDBDebugger.get().getEnv();
+ try {
+ List<Long> journalIds = env.getJournalIds(dbName);
+ for (Long journalId : journalIds) {
+ result.addRow(Lists.newArrayList(journalId.toString()));
+ }
+ } catch (BDBDebugger.BDBDebugException e) {
+ throw new AnalysisException(e.getMessage());
+ }
+
+ return result;
+ }
+}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/proc/BDBJEJournalDataProcNode.java
b/fe/fe-core/src/main/java/org/apache/doris/common/proc/BDBJEJournalDataProcNode.java
new file mode 100644
index 0000000..ab36a0f
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/proc/BDBJEJournalDataProcNode.java
@@ -0,0 +1,60 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.proc;
+
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
+import org.apache.doris.journal.bdbje.BDBDebugger;
+import org.apache.doris.persist.OperationType;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+
+// SHOW PROC "/bdbje/dbname/journalID"
+public class BDBJEJournalDataProcNode implements ProcNodeInterface {
+
+ public static final ImmutableList<String> TITLE_NAMES = new
ImmutableList.Builder<String>()
+ .add("JournalId").add("OpType").add("Data").build();
+
+ private String dbName;
+ private Long journalId;
+
+ public BDBJEJournalDataProcNode(String dbName, Long journalId) {
+ this.dbName = dbName;
+ this.journalId = journalId;
+ }
+
+ @Override
+ public ProcResult fetchResult() throws AnalysisException {
+ if (!Config.enable_bdbje_debug_mode) {
+ throw new AnalysisException("Not in bdbje debug mode");
+ }
+
+ BaseProcResult result = new BaseProcResult();
+ result.setNames(TITLE_NAMES);
+
+ BDBDebugger.BDBDebugEnv env = BDBDebugger.get().getEnv();
+ BDBDebugger.JournalEntityWrapper entity = env.getJournalEntity(dbName,
journalId);
+
+ short opCode = entity.entity == null ? -1 : entity.entity.getOpCode();
+ String data = entity.entity == null ? entity.errMsg :
entity.entity.getData().toString();
+ result.addRow(Lists.newArrayList(entity.journalId.toString(),
OperationType.getOpName(opCode), data));
+
+ return result;
+ }
+}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/proc/BDBJEProcDir.java
b/fe/fe-core/src/main/java/org/apache/doris/common/proc/BDBJEProcDir.java
new file mode 100644
index 0000000..84bf4a7
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/proc/BDBJEProcDir.java
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.proc;
+
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
+import org.apache.doris.journal.bdbje.BDBDebugger;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+
+// SHOW PROC "/bdbje"
+public class BDBJEProcDir implements ProcDirInterface {
+ public static final ImmutableList<String> TITLE_NAMES = new
ImmutableList.Builder<String>()
+ .add("DbNames").add("JournalNumber").add("Comment").build();
+
+ @Override
+ public boolean register(String name, ProcNodeInterface node) {
+ return false;
+ }
+
+ @Override
+ public ProcNodeInterface lookup(String dbName) throws AnalysisException {
+ return new BDBJEDatabaseProcDir(dbName);
+ }
+
+ @Override
+ public ProcResult fetchResult() throws AnalysisException {
+ if (!Config.enable_bdbje_debug_mode) {
+ throw new AnalysisException("Not in bdbje debug mode");
+ }
+
+ BaseProcResult result = new BaseProcResult();
+ result.setNames(TITLE_NAMES);
+
+ BDBDebugger.BDBDebugEnv debugEnv = BDBDebugger.get().getEnv();
+ List<String> dbNames = debugEnv.listDbNames();
+ TreeMap<String, Long> journalNumMap = new TreeMap<>();
+ for (String dbName : dbNames) {
+ journalNumMap.put(dbName, debugEnv.getJournalNumber(dbName));
+ }
+
+ for (Map.Entry<String, Long> entry : journalNumMap.entrySet()) {
+ result.addRow(Lists.newArrayList(entry.getKey(),
entry.getValue().toString(), ""));
+ }
+ return result;
+ }
+}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/proc/ProcService.java
b/fe/fe-core/src/main/java/org/apache/doris/common/proc/ProcService.java
index c93b137..1e23b0e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/proc/ProcService.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/proc/ProcService.java
@@ -51,6 +51,7 @@ public final class ProcService {
root.register("cluster_balance", new ClusterBalanceProcDir());
root.register("routine_loads", new RoutineLoadsProcDir());
root.register("colocation_group", new ColocationGroupProcDir());
+ root.register("bdbje", new BDBJEProcDir());
}
// 通过指定的路径获得对应的PROC Node
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/http/action/SystemAction.java
b/fe/fe-core/src/main/java/org/apache/doris/http/action/SystemAction.java
index 72b956c..3c941d9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/http/action/SystemAction.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/http/action/SystemAction.java
@@ -17,10 +17,12 @@
package org.apache.doris.http.action;
-import org.apache.commons.validator.routines.UrlValidator;
+import io.netty.handler.codec.http.HttpMethod;
+
import org.apache.doris.analysis.RedirectStatus;
import org.apache.doris.catalog.Catalog;
import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
import org.apache.doris.common.proc.ProcDirInterface;
import org.apache.doris.common.proc.ProcNodeInterface;
import org.apache.doris.common.proc.ProcResult;
@@ -36,14 +38,13 @@ import org.apache.doris.qe.ShowResultSet;
import com.google.common.base.Preconditions;
import com.google.common.base.Strings;
+import org.apache.commons.validator.routines.UrlValidator;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import java.util.List;
import java.util.stream.Collectors;
-import io.netty.handler.codec.http.HttpMethod;
-
public class SystemAction extends WebBaseAction {
private static final Logger LOG = LogManager.getLogger(SystemAction.class);
@@ -85,7 +86,7 @@ public class SystemAction extends WebBaseAction {
List<String> columnNames = null;
List<List<String>> rows = null;
- if (!Catalog.getCurrentCatalog().isMaster()) {
+ if (!Catalog.getCurrentCatalog().isMaster() &&
!Config.enable_bdbje_debug_mode) {
// forward to master
String showProcStmt = "SHOW PROC \"" + procPath + "\"";
MasterOpExecutor masterOpExecutor = new MasterOpExecutor(new
OriginStatement(showProcStmt, 0),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/journal/bdbje/BDBDebugger.java
b/fe/fe-core/src/main/java/org/apache/doris/journal/bdbje/BDBDebugger.java
new file mode 100644
index 0000000..1dc7e77
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/journal/bdbje/BDBDebugger.java
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.journal.bdbje;
+
+import org.apache.doris.common.Config;
+import org.apache.doris.common.FeMetaVersion;
+import org.apache.doris.common.ThreadPoolManager;
+import org.apache.doris.http.HttpServer;
+import org.apache.doris.http.IllegalArgException;
+import org.apache.doris.journal.JournalEntity;
+import org.apache.doris.meta.MetaContext;
+import org.apache.doris.qe.QeService;
+import org.apache.doris.service.ExecuteEnv;
+
+import com.clearspring.analytics.util.Lists;
+import com.google.common.base.Preconditions;
+import com.sleepycat.bind.tuple.TupleBinding;
+import com.sleepycat.je.Cursor;
+import com.sleepycat.je.Database;
+import com.sleepycat.je.DatabaseConfig;
+import com.sleepycat.je.DatabaseEntry;
+import com.sleepycat.je.DatabaseException;
+import com.sleepycat.je.Environment;
+import com.sleepycat.je.EnvironmentConfig;
+import com.sleepycat.je.LockMode;
+import com.sleepycat.je.OperationStatus;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+
+/**
+ * This class is mainly used to view the data in bdbje.
+ * When the config "enable_bdbje_debug_mode" of FE is set to true,
+ * start FE and enter debug mode.
+ * In this mode, only MySQL server and http server will be started.
+ * After that, users can log in to Palo through the web front-end or MySQL
client,
+ * and then use "show proc "/bdbje"" to view the data in bdbje.
+ */
+public class BDBDebugger {
+ private static final Logger LOG = LogManager.getLogger(BDBDebugger.class);
+
+ private static class SingletonHolder {
+ private static final BDBDebugger INSTANCE = new BDBDebugger();
+ }
+
+ public static BDBDebugger get() {
+ return SingletonHolder.INSTANCE;
+ }
+
+ private BDBDebugEnv debugEnv;
+
+ public void startDebugMode(String dorisHomeDir) {
+ try {
+ initDebugEnv();
+ startService(dorisHomeDir);
+ while (true) {
+ Thread.sleep(2000);
+ }
+ } catch (Throwable t) {
+ LOG.warn("BDB debug mode exception", t);
+ System.exit(-1);
+ }
+ }
+
+ // Only start MySQL and HttpServer
+ private void startService(String dorisHomeDir) throws IllegalArgException,
IOException {
+ // HTTP server
+ if (!Config.enable_http_server_v2) {
+ HttpServer httpServer = new HttpServer(
+ Config.http_port,
+ Config.http_max_line_length,
+ Config.http_max_header_size,
+ Config.http_max_chunk_size
+ );
+ httpServer.setup();
+ httpServer.start();
+ } else {
+ org.apache.doris.httpv2.HttpServer httpServer2 = new
org.apache.doris.httpv2.HttpServer();
+ httpServer2.setPort(Config.http_port);
+ httpServer2.start(dorisHomeDir);
+ }
+
+ // MySQl server
+ QeService qeService = new QeService(Config.query_port,
Config.mysql_service_nio_enabled, ExecuteEnv.getInstance().getScheduler());
+ qeService.start();
+
+ ThreadPoolManager.registerAllThreadPoolMetric();
+ }
+
+ private void initDebugEnv() throws BDBDebugException {
+ debugEnv = new BDBDebugEnv(Config.meta_dir + "/bdb/");
+ debugEnv.init();
+ }
+
+ /**
+ * A wrapper class of the BDBJE environment, used to obtain information in
bdbje
+ */
+ public static class BDBDebugEnv {
+ // the dir of bdbje data dir
+ private String metaPath;
+ private Environment env;
+
+ public BDBDebugEnv(String metaPath) {
+ this.metaPath = metaPath;
+ }
+
+ public void init() throws BDBDebugException {
+ EnvironmentConfig envConfig = new EnvironmentConfig();
+ envConfig.setAllowCreate(false);
+ envConfig.setReadOnly(true);
+ envConfig.setCachePercent(20);
+
+ try {
+ env = new Environment(new File(metaPath), envConfig);
+ } catch (DatabaseException e) {
+ throw new BDBDebugException("failed to init bdb env", e);
+ }
+ Preconditions.checkNotNull(env);
+ }
+
+ // list all database in bdbje
+ public List<String> listDbNames() {
+ return env.getDatabaseNames();
+ }
+
+ // get the number of journal in specified database
+ public Long getJournalNumber(String dbName) {
+ DatabaseConfig dbConfig = new DatabaseConfig();
+ dbConfig.setAllowCreate(false);
+ dbConfig.setReadOnly(true);
+ Database db = env.openDatabase(null, dbName, dbConfig);
+ return db.count();
+ }
+
+ // get list of journal id (key) in specified database
+ public List<Long> getJournalIds(String dbName) throws
BDBDebugException {
+ DatabaseConfig dbConfig = new DatabaseConfig();
+ dbConfig.setAllowCreate(false);
+ dbConfig.setReadOnly(true);
+ Database db = env.openDatabase(null, dbName, dbConfig);
+
+ List<Long> journalIds = Lists.newArrayList();
+
+ Cursor cursor = db.openCursor(null, null);
+ DatabaseEntry key = new DatabaseEntry();
+ DatabaseEntry value = new DatabaseEntry();
+ TupleBinding<Long> idBinding =
TupleBinding.getPrimitiveBinding(Long.class);
+ try {
+ while (true) {
+ OperationStatus status = cursor.getNext(key, value, null);
+ if (status == OperationStatus.NOTFOUND) {
+ break;
+ }
+
+ Long id = idBinding.entryToObject(key);
+ journalIds.add(id);
+ }
+ } catch (Exception e) {
+ LOG.warn("failed to get journal ids of {}", dbName, e);
+ throw new BDBDebugException("failed to get journal ids of
database " + dbName, e);
+ }
+ return journalIds;
+ }
+
+ // get the journal entity of the specified journal id.
+ public JournalEntityWrapper getJournalEntity(String dbName, Long
journalId) {
+ // meta version
+ // TODO(cmy): currently the journal data will be read with
VERSION_CURRENT
+ // So if we use a new version of Palo to read the meta of old
version,
+ // It may fail.
+ MetaContext metaContext = new MetaContext();
+ metaContext.setMetaVersion(FeMetaVersion.VERSION_CURRENT);
+ metaContext.setThreadLocalInfo();
+
+ DatabaseConfig dbConfig = new DatabaseConfig();
+ dbConfig.setAllowCreate(false);
+ dbConfig.setReadOnly(true);
+ Database db = env.openDatabase(null, dbName, dbConfig);
+
+ JournalEntityWrapper entityWrapper = new
JournalEntityWrapper(journalId);
+
+ DatabaseEntry key = new DatabaseEntry();
+ TupleBinding<Long> myBinding =
TupleBinding.getPrimitiveBinding(Long.class);
+ myBinding.objectToEntry(journalId, key);
+ DatabaseEntry value = new DatabaseEntry();
+
+ // get the journal
+ OperationStatus status = db.get(null, key, value,
LockMode.READ_COMMITTED);
+ if (status == OperationStatus.SUCCESS) {
+ byte[] retData = value.getData();
+ DataInputStream in = new DataInputStream(new
ByteArrayInputStream(retData));
+ JournalEntity entity = new JournalEntity();
+ try {
+ // read the journal data
+ entity.readFields(in);
+ entityWrapper.entity = entity;
+ } catch (Exception e) {
+ LOG.warn("failed to read entity", e);
+ entityWrapper.errMsg = e.getMessage();
+ }
+ } else if (status == OperationStatus.NOTFOUND) {
+ entityWrapper.errMsg = "Key not found";
+ }
+ MetaContext.remove();
+ return entityWrapper;
+ }
+ }
+
+ public static class JournalEntityWrapper {
+ public Long journalId;
+ public JournalEntity entity;
+ public String errMsg;
+
+ public JournalEntityWrapper(long journalId) {
+ this.journalId = journalId;
+ }
+ }
+
+ public BDBDebugEnv getEnv() {
+ return debugEnv;
+ }
+
+ public static class BDBDebugException extends Exception {
+ public BDBDebugException(String msg) {
+ super(msg);
+ }
+
+ public BDBDebugException(String msg, Throwable t) {
+ super(msg, t);
+ }
+ }
+}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/persist/OperationType.java
b/fe/fe-core/src/main/java/org/apache/doris/persist/OperationType.java
index 785cd1e..e32e287 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/persist/OperationType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/persist/OperationType.java
@@ -17,6 +17,9 @@
package org.apache.doris.persist;
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+
public class OperationType {
public static final short OP_INVALID = -1;
public static final short OP_SAVE_NEXTID = 0;
@@ -196,4 +199,24 @@ public class OperationType {
// alter external table
public static final short OP_ALTER_EXTERNAL_TABLE_SCHEMA = 280;
+
+ // get opcode name by op codeStri
+ public static String getOpName(short opCode) {
+ try {
+ Field[] fields = OperationType.class.getDeclaredFields();
+ for (Field field : fields) {
+ if (!Modifier.isStatic(field.getModifiers())) {
+ continue;
+ }
+ short s = field.getShort(null);
+ if (s != opCode) {
+ continue;
+ }
+ return field.getName();
+ }
+ } catch (Exception e) {
+ return "Not Found: " + e.getMessage();
+ }
+ return "Not Found";
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
