yiguolei commented on code in PR #69635:
URL: https://github.com/apache/doris/pull/69635#discussion_r4388051847
##########
be/src/exec/spill/spill_file_writer.h:
##########
@@ -44,29 +45,72 @@ class SpillFile;
///
/// Part rotation is fully internal. Each part file has its own footer with
/// block offset metadata. Parts are named 0, 1, 2, ... within the SpillFile's
-/// directory.
+/// directory. A part is closed synchronously when it is rotated out or when
close() is called.
+/// If the SpillFile is destroyed before the writer is closed, the unfinished
part is discarded.
+///
+/// Files are created on the file system pinned in the SpillFile, which is
either a local disk or
+/// the object storage of a cloud storage vault. For object storage the writer
additionally
+/// pauses before every append while the S3 upload queue holds at least
+/// spill_s3_max_inflight_upload_bytes of buffers, which slows down a spill
that
+/// produces faster than object storage takes it, and reports request
statistics derived from
+/// the closed part.
class SpillFileWriter {
public:
SpillFileWriter(const std::shared_ptr<SpillFile>& spill_file,
RuntimeState* state,
- RuntimeProfile* profile, SpillDataDir* data_dir, const
std::string& spill_dir);
+ RuntimeProfile* profile, SpillDataDir* data_dir,
io::FileSystemSPtr fs,
Review Comment:
你不如新写一个S3SpillFileWriter 了，现在代码里你加的这些都是S3 相关的；现在相当于把2者合并到一起了，后期代码越来越大，没必要再一起了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
