Re: [Result] [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

2018-12-13 Thread Vinod Kumar Vavilapalli
Agree, it isn't productive this way. I can't seem to find it, but was there a DISCUSS thread for this branch-merge? I usually recommend addressing issues on a DISCUSS thread instead of fighting things over a VOTE. +Vinod > On Dec 13, 2018, at 10:09 AM, Konstantin Shvachko > wrote: > > This

Re: [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

2018-12-13 Thread Konstantin Shvachko
Hi Yongjun, Good suggestion. This is essentially what HDFS-13873 is implementing to mitigate the concern. Thanks, --Konstantin On Wed, Dec 12, 2018 at 10:35 PM Yongjun Zhang wrote: > Hi Konstantin, > > Thanks for addressing my other question about failover. > > Some thought to share about the

[Result] [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

2018-12-13 Thread Konstantin Shvachko
This vote failed due to Daryn Sharp's veto. The concern is being addressed by HDFS-13873. I will start a new vote once this is committed. Note for Daryn. Your non-responsive handling of the veto makes a bad precedence and is a bad example of communication on the lists from a respected member of th

Re: [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

2018-12-12 Thread Yongjun Zhang
Hi Konstantin, Thanks for addressing my other question about failover. Some thought to share about the suggestion Daryn made. Seems we could try this: let ObserverNode throws an RetriableException back to client saying it has not reached the transaction ID to serve the client yet, maybe even inc

Re: [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

2018-12-07 Thread Konstantin Shvachko
Hi Daryn, Wanted to backup Chen's earlier response to your concerns about rotating calls in the call queue. Our design 1. targets directly the livelock problem by rejecting calls on the Observer that are not likely to be responded in timely matter: HDFS-13873. 2. The call queue rotation is only do

Re: [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

2018-12-06 Thread Chen Liang
Hi Daryn, This is an interesting and valid point to consider different implications for security. The purpose of the alignment context is to allow clients and servers sync on their global state, so that when clients switch between ANN/SBN or between SBNs, the reads are always consistent. One reas

Re: [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

2018-12-06 Thread Konstantin Shvachko
Hi Yongjun, Automatic failover sure needs to be fixed (see HDFS-14130 and HDFS-13182). Along with all other outstanding issues. We plan to continue this on trunk. The feature is usable now without this issues (see HDFS-14067). And we would like to get it in, so that people could have early access,

Re: [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

2018-12-06 Thread Anu Engineer
Hi Daryn, I have just started reading the patch. Hence my apologies if my question has a response somewhere hidden in the patch. Are you concerned that FSEditLock is taken in GlobalStateIdContext on Server side, and worried that a malicious or stupid client would cause this lock to be held up

Re: [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

2018-12-06 Thread Daryn Sharp
-1 pending additional info. After a cursory scan, I have serious concerns regarding the design. This seems like a feature that should have been purely implemented in hdfs w/o touching the common IPC layer. The biggest issue in the alignment context. It's purpose appears to be for allowing handl

Re: [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

2018-12-05 Thread Yongjun Zhang
Great work guys. Wonder if we can elaborate what's impact of not having #2 fixed, and why #2 is not needed for the feature to complete? 2. Need to fix automatic failover with ZKFC. Currently it does not doesn't know about ObserverNodes trying to convert them to SBNs. Thanks. --Yongjun On Wed, D

Re: [VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

2018-12-05 Thread Zhe Zhang
+1 (binding) Thanks Konstantin for leading the merge effort! I worked very closely with Chen, Konstantin, and Erik in the testing stage and I feel confident that the feature has now completed designed functionalities and has proven to be stable. Great team work with contributors from multiple co

[VOTE] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby

2018-12-05 Thread Konstantin Shvachko
Hi Hadoop developers, I would like to propose to merge to trunk the feature branch HDFS-12943 for Consistent Reads from Standby Node. The feature is intended to scale read RPC workloads. On large clusters reads comprise 95% of all RPCs to the NameNode. We should be able to accommodate higher overa