hemantk-12 commented on code in PR #8214:
URL: https://github.com/apache/ozone/pull/8214#discussion_r2026221033
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/RocksDBCheckpointDiffer.java:
##########
@@ -1382,6 +1393,62 @@ public void pruneSstFiles() {
}
}
+ /**
+ * Defines the task that removes OMKeyInfo from SST files from backup
directory to
+ * save disk space.
+ */
+ public void pruneSstFileValues() {
+ if (!shouldRun()) {
+ return;
+ }
+ Path sstBackupDirPath = Paths.get(sstBackupDir);
+ String kVSeparator = ",";
+
+ try (Stream<Path> files = Files.list(sstBackupDirPath)
+ .filter(file -> file.endsWith(ROCKSDB_SST_SUFFIX)) ) {
+
+ for (Path file : files.collect(Collectors.toList())) {
+ // Write the file.sst => file.sst.tmp
+ File sstFile = file.toFile();
+ File prunedSSTFile = Files.createFile(sstBackupDirPath
+ .resolve( sstFile.getName() + ".tmp")).toFile();
+
+ ManagedEnvOptions envOptions = new ManagedEnvOptions();
+ ManagedOptions managedOptions = new ManagedOptions();
+ ManagedSstFileWriter sstFileWriter = new
ManagedSstFileWriter(envOptions, managedOptions))
+ sstFileWriter.open(prunedSSTFile.getAbsolutePath());
+
+ ManagedRawSSTFileReader sstFileReader = new ManagedRawSSTFileReader<>(
Review Comment:
Can we reuse
[RDBSstFileWriter](https://github.com/apache/ozone/blob/0f52a34d0c45f5f17a568ea0d183880dfcdcd063/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBSstFileWriter.java#L34)?
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/RocksDBCheckpointDiffer.java:
##########
@@ -1382,6 +1393,62 @@ public void pruneSstFiles() {
}
}
+ /**
+ * Defines the task that removes OMKeyInfo from SST files from backup
directory to
+ * save disk space.
+ */
+ public void pruneSstFileValues() {
+ if (!shouldRun()) {
+ return;
+ }
+ Path sstBackupDirPath = Paths.get(sstBackupDir);
+ String kVSeparator = ",";
+
+ try (Stream<Path> files = Files.list(sstBackupDirPath)
Review Comment:
How are we differentiating if a file has been pruned or not?
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/RocksDBCheckpointDiffer.java:
##########
@@ -1382,6 +1393,62 @@ public void pruneSstFiles() {
}
}
+ /**
+ * Defines the task that removes OMKeyInfo from SST files from backup
directory to
+ * save disk space.
+ */
+ public void pruneSstFileValues() {
+ if (!shouldRun()) {
+ return;
+ }
+ Path sstBackupDirPath = Paths.get(sstBackupDir);
+ String kVSeparator = ",";
+
+ try (Stream<Path> files = Files.list(sstBackupDirPath)
+ .filter(file -> file.endsWith(ROCKSDB_SST_SUFFIX)) ) {
+
+ for (Path file : files.collect(Collectors.toList())) {
+ // Write the file.sst => file.sst.tmp
+ File sstFile = file.toFile();
+ File prunedSSTFile = Files.createFile(sstBackupDirPath
+ .resolve( sstFile.getName() + ".tmp")).toFile();
+
+ ManagedEnvOptions envOptions = new ManagedEnvOptions();
+ ManagedOptions managedOptions = new ManagedOptions();
Review Comment:
These can be outside the for loop and reused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org
For additional commands, e-mail: issues-h...@ozone.apache.org
