@Devesh Thanks for the info. Good to know the issue is already under investigation.
@Tsz-wo Sze <szets...@apache.org> Yes, the basic idea we discussed is for every protobuf data structure, create a pure Java class for this, so deserialization only happens one time when the protobuf is received, and serialization before it's sent on wire or persist to DB. Given the protobuf data structure number, it's not a small task. But we can start from the most frequently used data structure first. On Thu, 7 Mar 2024 at 04:32, Tsz Wo Sze <szets...@apache.org> wrote: > > b. Observed that many objects in OM are serialized and deserialized > multiple times, which add the pressure to GC. > > Here is an idea to avoid multiple serialization/deserialization in OM: > https://issues.apache.org/jira/browse/HDDS-9290 > > Tsz-Wo > > > On Wed, Mar 6, 2024 at 12:30 AM Devesh Singh > <deveshsi...@cloudera.com.invalid> wrote: > > > Thank you Sammi for update. Regarding the below point: > > > > > > > When DN is dead or decommissioned, DN info is still held in SCM > > > memory. After SCM restarts, those DNs info will be gone. But Recon > still > > > shows these DNs on the UI. > > > > - Currently being tracked here and working on it: > > https://issues.apache.org/jira/browse/HDDS-10409 > > > > Regards > > Devesh > > > > > > > > > On 06-Mar-2024, at 1:55 PM, Sammi Chen <sammic...@apache.org> wrote: > > > > > > Attender:Hao, Mingyu, Yiyang, Xi, Bin, Jianghua, Hualong, Yuanben, > > Conway, > > > Chungen, Wei-Chiu, Sammi > > > 1. DiDi > > > a. Bad replicas are not timely detected and reported in the system > > so > > > that healthy replicas are deleted by RM. Some data is lost due to this. > > > Suggest turning on Disk Scanner by turning on > > > "hdds.container.scrub.enabled". > > > b. Two consecutive tasks are scheduled to replicate the same > > > container on the same DN. HDDS-9661 has fixed this issue. > > > c. Plan to use high density storage servers as Ozone DN. Wei-Chiu > > > shares a document about the last test experience of Ozone on high > density > > > storage. > > > 2. Shopee > > > a. When DN is dead or decommissioned, DN info is still held in SCM > > > memory. After SCM restarts, those DNs info will be gone. But Recon > still > > > shows these DNs on the UI. > > > b. Want to know the different type buckets and their > supportability > > > in each Ozone access interface(s3g, ofs, cli). > > > > > > https://blog.cloudera.com/apache-ozone-a-multi-protocol-aware-storage-system/ > > > 3. Qihoo > > > a. multipart key info serialization and deserialization overhead > is > > > very high for big objects(100G). Evaluate to add a new table for > > multipart > > > key info. Will create a JIRA for this later. > > > b. Observed that many objects in OM are serialized and > deserialized > > > multiple times, which add the pressure to GC. > > > c. Per user throughput throttle to avoid most of the throughput > > being > > > used by one user. FairCallQueue is suggested. > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@ozone.apache.org > > For additional commands, e-mail: dev-h...@ozone.apache.org > > > > >
