Attenders: Hui, Yiyang, Chung en, Sammi, Wei-Chiu
Key Achievements
The meeting focused on synchronizing the merge status of S3 Temporary Token
Service, discussing the LanceDB integration plan, and the design
details of Object
Versioning and Ranger permissions. Additionally, the team explored the
performance issue of slow data decommissioning in EC mode .
Progress Updates
- S3 Temporary Token Service: This feature has been merged into the
master branch, allowing users to exchange permanent secrets for short-term
tokens with a maximum duration of 12 hours, primarily used for automated
jobs or workers .
- Object Versioning:
- Sammi completed a design presentation within the team, which was
well-received.
- Yiyang added permission section in the design doc
- Yiyang has submitted 7 commits, Chen Xi is also advancing related
work, currently awaiting community review.
Decisions and Technical Solutions
- LanceDB Integration: Planning to use Ozone as the backend object
storage for LanceDB, making it appear in the official configuration options
.
-
Ranger Permission Extension:
- Drawing from the STS assume role model, introducing more fine-grained
permission control for Object Versioning .
- For security-related operations such as Locking, the plan is to add
specific permission categories in Ranger to support execution by
dedicated
security oversight roles, rather than only by owners .
- New permissions need to be synchronized with Ranger UI updates to
enable user assignment
Risks and Issues
- EC Performance Bottleneck: Wei-Chiu discovered that decommissioning is
extremely slow on single machines with over 100TB . Currently difficult to
reproduce on high-performance machines, suspected to be related to machine
performance or parameter configuration .
- Community Review Progress: In August, due to some community members
being on vacation, the review progress for submitted code has been slower .
Action Items
- Sammi: Assist in reviewing code submitted by yiyang to the community .
- Chung: Review comments left by Sammi in the design document and
consult Fabian on Ranger permission details .
- Wei-Chiu: Continue attempting to reproduce the slow EC decommission
issue and verify performance improvement solutions .
- Sammi: Attempt to write automation scripts to accelerate the release
verification process .
