Re: Efficient and effective fsync during commit

2015-06-17 Thread Stefan Sperling
On Tue, Jun 16, 2015 at 11:01:36PM +0200, Stefan Fuhrmann wrote: > I feel this is something we need to talk about privately in Berlin. I'll make very sure indeed you two get to share a double bedroom at the hotel. Promise.

Re: Efficient and effective fsync during commit

2015-06-16 Thread Branko Čibej
On 15.06.2015 18:24, Mark Phippard wrote: > On Mon, Jun 15, 2015 at 12:15 PM, Branko Čibej > wrote: > > On 15.06.2015 17:36, Ivan Zhakov wrote: > > On 12 June 2015 at 15:11, Stefan Fuhrmann > > wrote: > >> To be c

Re: Efficient and effective fsync during commit

2015-06-16 Thread Stefan Fuhrmann
On Mon, Jun 15, 2015 at 5:36 PM, Ivan Zhakov wrote: > On 12 June 2015 at 15:11, Stefan Fuhrmann > wrote: > > On Fri, May 29, 2015 at 6:23 PM, Ivan Zhakov wrote: > >> > >> On 29 May 2015 at 18:55, Stefan Fuhrmann > >> wrote: > >> > If you assume / suspect that FlushFileBuffers() only > >> > ope

Re: Efficient and effective fsync during commit

2015-06-15 Thread Mark Phippard
On Mon, Jun 15, 2015 at 12:15 PM, Branko Čibej wrote: > On 15.06.2015 17:36, Ivan Zhakov wrote: > > On 12 June 2015 at 15:11, Stefan Fuhrmann > wrote: > >> To be clear: You are proposing that the code on Windows > >> is fundamentally broken (revision contents not being > >> committed) while I th

Re: Efficient and effective fsync during commit

2015-06-15 Thread Branko Čibej
On 15.06.2015 17:36, Ivan Zhakov wrote: > On 12 June 2015 at 15:11, Stefan Fuhrmann > wrote: >> To be clear: You are proposing that the code on Windows >> is fundamentally broken (revision contents not being >> committed) while I think we "only" have a persistence >> issue with renames. Since you

Re: Efficient and effective fsync during commit

2015-06-15 Thread Ivan Zhakov
On 12 June 2015 at 15:11, Stefan Fuhrmann wrote: > On Fri, May 29, 2015 at 6:23 PM, Ivan Zhakov wrote: >> >> On 29 May 2015 at 18:55, Stefan Fuhrmann >> wrote: >> > On Fri, May 29, 2015 at 4:14 PM, Ivan Zhakov wrote: >> >> On 28 May 2015 at 20:47, Stefan Fuhrmann >> >> wrote: >> >>> Hi all, >>

Re: Efficient and effective fsync during commit

2015-06-12 Thread Stefan Fuhrmann
On Fri, May 29, 2015 at 6:23 PM, Ivan Zhakov wrote: > On 29 May 2015 at 18:55, Stefan Fuhrmann > wrote: > > On Fri, May 29, 2015 at 4:14 PM, Ivan Zhakov wrote: > >> On 28 May 2015 at 20:47, Stefan Fuhrmann > wrote: > >>> Hi all, > >>> > >>> Most of us would agree that way we fsync FS changes >

Re: Efficient and effective fsync during commit

2015-06-04 Thread Daniel Klíma
2015-05-29 20:55 GMT+02:00 Branko Čibej : > On 29.05.2015 18:23, Ivan Zhakov wrote: > > On 29 May 2015 at 18:55, Stefan Fuhrmann > wrote: > >> You might be right. So, if you care about repository > >> integrity, you should use your MSDN subscription and > >> ask MS for clarification on FlushFileB

Re: Efficient and effective fsync during commit

2015-05-29 Thread Branko Čibej
On 29.05.2015 18:23, Ivan Zhakov wrote: > On 29 May 2015 at 18:55, Stefan Fuhrmann wrote: >> You might be right. So, if you care about repository >> integrity, you should use your MSDN subscription and >> ask MS for clarification on FlushFileBuffers() behaviour. > You also may request MSDN subscri

Re: Efficient and effective fsync during commit

2015-05-29 Thread Branko Čibej
On 29.05.2015 17:55, Stefan Fuhrmann wrote: > If you assume / suspect that FlushFileBuffers() only operates on the > open handle, i.e. only flushes those changes made through that thandle, >From my dabbling with the Windows I/O stack and filesystems way back, I'd say that flushing (and all other o

Re: Efficient and effective fsync during commit

2015-05-29 Thread Ivan Zhakov
On 29 May 2015 at 18:55, Stefan Fuhrmann wrote: > On Fri, May 29, 2015 at 4:14 PM, Ivan Zhakov wrote: >> On 28 May 2015 at 20:47, Stefan Fuhrmann >> wrote: >>> Hi all, >>> >>> Most of us would agree that way we fsync FS changes >>> in FSFS and FSX is slow (~10 commits / sec on a SSD, >>> YMMV)

Re: Efficient and effective fsync during commit

2015-05-29 Thread Stefan Fuhrmann
On Fri, May 29, 2015 at 4:14 PM, Ivan Zhakov wrote: > On 28 May 2015 at 20:47, Stefan Fuhrmann wrote: >> Hi all, >> >> Most of us would agree that way we fsync FS changes >> in FSFS and FSX is slow (~10 commits / sec on a SSD, >> YMMV) and not even all changes are fully fsync'ed >> (repo creation

Re: Efficient and effective fsync during commit

2015-05-29 Thread Ivan Zhakov
On 28 May 2015 at 20:47, Stefan Fuhrmann wrote: > Hi all, > > Most of us would agree that way we fsync FS changes > in FSFS and FSX is slow (~10 commits / sec on a SSD, > YMMV) and not even all changes are fully fsync'ed > (repo creation, upgrade). > The first question is it really a problem? I me

Re: Efficient and effective fsync during commit

2015-05-29 Thread Philip Martin
Stefan Fuhrmann writes: > On Thu, May 28, 2015 at 9:54 PM, Philip Martin > wrote: >> >> fsync() works on file descriptors rather than files, do we need to keep >> the original file descriptors open in order to fsync()? > > We could b/c there are at most 7 (4 files, 3 folders) of them for a > FSF

Re: Efficient and effective fsync during commit

2015-05-29 Thread Stefan Fuhrmann
On Thu, May 28, 2015 at 9:54 PM, Philip Martin wrote: > Stefan Fuhrmann writes: > >> In the future, we should implement step 1 as simple >> non-fsync'ing file operations. Then explicitly sync every >> file, and on POSIX the folders, once. Step 2 does not >> have any atomicity requirements. Finall

Re: Efficient and effective fsync during commit

2015-05-28 Thread Philip Martin
Stefan Fuhrmann writes: > In the future, we should implement step 1 as simple > non-fsync'ing file operations. Then explicitly sync every > file, and on POSIX the folders, once. Step 2 does not > have any atomicity requirements. Finally, do the 'current' > rename. This also only requires a single