gavinchou commented on code in PR #34354:
URL: https://github.com/apache/doris/pull/34354#discussion_r1587034146
##########
be/src/io/fs/hdfs_file_writer.cpp:
##########
@@ -46,6 +49,52 @@ bvar::Adder<uint64_t>
hdfs_file_being_written("hdfs_file_writer_file_being_writt
static constexpr size_t MB = 1024 * 1024;
+// In practice, we've found that if the import frequency to HDFS is too fast,
+// it can cause an OutOfMemoryError (OOM) in the JVM started by the JNI.
+// For this, we should have a method to monitor how much JVM memory is
currently being used.
+// The HdfsWriteRateLimit class increments a recorded value during hdfsWrite
when writing to HDFS.
+// When hdfsCloseFile is called, all related memory in the JVM will be
invalidated, so the recorded value can be decreased at that time.
+// If the current usage exceeds the maximum set by the user, the current write
will sleep.
+// If the number of sleeps exceeds the number specified by the user, then the
current write is considered to have failed
+class HdfsWriteRateLimit {
+public:
+ HdfsWriteRateLimit()
+ : max_jvm_heap_size(JniUtil::get_max_jni_heap_memory_size()),
+ cur_memory_comsuption(0) {}
+ size_t max_usage() const {
+ return static_cast<size_t>(max_jvm_heap_size *
config::max_hdfs_jni_heap_usage_ratio);
+ }
+ Status do_rate_limit(size_t memory_size) {
Review Comment:
Make the name symmetric to `release_jni_memory()`,
and, may be the word `jni` is unnecessary.
say, `acquire_memory(size_t)` and `release_memory(size_t)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
