Nisha Moond <nisha.moond...@gmail.com> writes: > Thoughts? Looking forward to hearing others' opinions!
Had a productive conversation with Amit Kaplia today about time skew in distributed systems, and wanted to share some thoughts. Essentially, we're grappling with the classic distributed snapshot problem. In a multi-active environment, where multiple nodes can independently process transactions, it becomes crucial to determine the visibility of these transactions across the system. Time skew, where different machines have different timestamps make it a hard problem. How can we ensure consistent transaction ordering and visibility when time itself is unreliable? As you mentioned, there are several ways to tackle the time skew problem in distributed systems. These approaches generally fall into three main categories: 1. Centralized Timestamps (Timestamp Oracle) Mechanism: A dedicated server acts as a single source of truth for time, eliminating skew by providing timestamps to all nodes. Google Percolator and TiDB use this approach. Consistency level: Serializable Pros: Simple to implement. Cons: High latency for cross-geo transactions due to reliance on a central server. Can become a bottleneck. 2. Atomic Clocks (True Time) Mechanism: Utilizes highly accurate atomic clocks to provide a globally consistent view of time, as seen in Google Spanner. Consistency level: External Serializable Pros: Very high consistency level (externally consistent). Cons: Requires specialized and expensive hardware. Adds some latency to transactions, though less than centralized timestamps. 3. Hybrid Logical Clocks Mechanism: CombinesNTP for rough time synchronization with logical clocks for finer-grained ordering. Yugabyte and CockroachDB employ this strategy. Consistency level: Serializable Pros: Avoids the need for specialized hardware. Cons: Can introduce significant latency to transactions. 4 Local Clocks Mechanism: Just use logical clock Consistency level: Eventual Consistency Pros: Simple implementation Cons: The consistency level is very low Of the four implementations considered, only local clocks and the HLC approach offer a 'pure database' solution. Given PostgreSQL's practical use cases, I recommend starting with a local clock implementation. However, recognizing the increasing prevalence of distributed clock services, we should also implement a pluggable time access method. This allows users to integrate with different time services as needed. In the mid-term, implementing the HLC approach would provide highly consistent snapshot reads. This offers a significant advantage for many use cases. Long-term, we should consider integrating with a distributed time service like AWS Time Sync Service. This ensures high accuracy and scalability for demanding applications. Thanks, Shihao
