> -Original Message-
> From: Neels J Hofmeyr [mailto:ne...@elego.de]
> Sent: woensdag 3 maart 2010 20:26
> To: Greg Stein
> Cc: Stefan Sperling; dev@subversion.apache.org
> Subject: Re: pristine store design
>
> Two separate things are getting mixed up her
Hello Stefan,
On Mittwoch, 3. März 2010, Stefan Sperling wrote:
> A block is "just another pristine".
> So a block can happen to also serve as a pristine for a different 10MB
> file which happens to have the same content as the block.
> We set the block size to something fixed, like 10MB.
> The la
Two separate things are getting mixed up here.
1) The fact that storing (only) md5 in the DB introduces the need for
locking where it would otherwise be lockless.
2) Some misty idea of us to split pristines into multiple chunks of a max
size each in order to be able to verify checksums of subsect
I didn't miss any part.
You're optimizing writes, when you should worry about reads.
The DB is always open, so reading and writing to it is "cheap".
I don't care about a scheme to seek to the end of a 16Gb chunk. You're
making stuff up again.
On Mar 3, 2010 9:45 AM, "Stefan Sperling" wrote:
O
On Wed, Mar 03, 2010 at 12:24:29PM -0500, Greg Stein wrote:
> You're talking about schemes to verify partial reads, yet stated you
> couldn't think of any cases.
I didn't say that. I'd very much like svn to verify data it
reads from the pristine store, on the fly, and point out corrupted
pristines
On Wed, 2010-03-03 at 11:10 -0500, Greg Stein wrote:
> On Wed, Mar 3, 2010 at 08:48, Neels J Hofmeyr wrote:
> >...
> >> If the caller creates the file, the caller chooses the file's name. We
> >> have to trust the caller to use the right kind of creation functions to
> >> give a safely unique name
You're talking about schemes to verify partial reads, yet stated you
couldn't think of any cases.
You're talking about splitting files for certain filesystems to help with
size limitations, yet the working file still has to be whole, and we have
no/few? reports of size problems.
IOW, you're just
On Wed, Mar 03, 2010 at 11:03:59AM -0500, Greg Stein wrote:
> We have one area for metadata, and that is the sqlite file named wc.db.
>
> Spreading metadata around is not very attractive. New custom file
> formats is also unattractive.
That's true. But it may be worth making an exception for this
On Wed, Mar 3, 2010 at 08:48, Neels J Hofmeyr wrote:
>...
>> If the caller creates the file, the caller chooses the file's name. We
>> have to trust the caller to use the right kind of creation functions to
>> give a safely unique name.
libsvn_wc. no big deal.
>> If the caller creates the file,
We have one area for metadata, and that is the sqlite file named wc.db.
Spreading metadata around is not very attractive. New custom file
formats is also unattractive.
You should not have any problems moving a pristine into place and
storing a record into the db. Sync is not the problem.
Funky h
Neels J Hofmeyr wrote:
> Julian Foad wrote:
[...]
> > I want us to be able to move on to bigger things like considering
> > whether to build a "pristine whole nodes" store on top of the "pristine
> > texts" store, because I think we should be able to reference a whole
> > node (especially a tree-co
On Wed, Mar 3, 2010 at 5:34 PM, Bert Huijben wrote:
>> > On Windows you get an access denied (bad) and a 15 second delay retrying
>> the
>> > move (worse).
>> >
>> > So we should try to avoid overwriting existing files here. (I would guess
>> > that tools like rsync and incremental backusp also li
> -Original Message-
> From: Ivan Zhakov [mailto:i...@visualsvn.com]
> Sent: woensdag 3 maart 2010 15:22
> To: Bert Huijben
> Cc: Stefan Sperling; Julian Foad; Neels J Hofmeyr; Greg Stein;
> dev@subversion.apache.org
> Subject: Re: pristine store design
>
>
On Wed, Mar 3, 2010 at 3:54 PM, Bert Huijben wrote:
> On posix, when using svn_io_rename_file() this would be true and this would
> be pretty safe.
>
> On Windows you get an access denied (bad) and a 15 second delay retrying the
> move (worse).
>
> So we should try to avoid overwriting existing fi
On Wed, Mar 03, 2010 at 01:54:28PM +0100, Bert Huijben wrote:
>
> > If two threads write to the same pristine, the content written
> > will be the same (except in case of a SHA1 checksum collision
> > which we choose to ignore). So, thread 1 writes to a tempfile, and
> > when it's done, it moves t
Julian Foad wrote:
> On Wed, 2010-03-03, Neels J Hofmeyr wrote:
>> Julian Foad wrote:
>>> On Thu, 2010-02-18, Neels J Hofmeyr wrote:
Great, moving forward fast on pristine design questions!
>>> Hi Neels.
>>>
>>> Did you start working the new knowledge into a document? Lots of stuff
>>> was sa
On Wed, 2010-03-03, Bert Huijben wrote:
>
> > -Original Message-
> > From: Stefan Sperling [mailto:s...@elego.de]
> > Sent: maandag 1 maart 2010 19:32
> > To: Julian Foad
> > Cc: Neels J Hofmeyr; Greg Stein; dev@subversion.apache.org; Bert Huijben
>
> -Original Message-
> From: Stefan Sperling [mailto:s...@elego.de]
> Sent: maandag 1 maart 2010 19:32
> To: Julian Foad
> Cc: Neels J Hofmeyr; Greg Stein; dev@subversion.apache.org; Bert Huijben
> Subject: Re: pristine store design
>
> On Mon, Mar 01, 2010 at
On Wed, 2010-03-03, Neels J Hofmeyr wrote:
> Julian Foad wrote:
> > On Thu, 2010-02-18, Neels J Hofmeyr wrote:
> >> Great, moving forward fast on pristine design questions!
> >
> > Hi Neels.
> >
> > Did you start working the new knowledge into a document? Lots of stuff
> > was said in this threa
Julian Foad wrote:
> On Thu, 2010-02-18, Neels J Hofmeyr wrote:
>> Great, moving forward fast on pristine design questions!
>
> Hi Neels.
>
> Did you start working the new knowledge into a document? Lots of stuff
> was said in this thread and it would be useful to see where we are at.
thanks fo
Stefan Sperling wrote:
> On Mon, Mar 01, 2010 at 06:14:33PM +, Julian Foad wrote:
> > On Thu, 2010-02-18, Neels J Hofmeyr wrote:
> > > The one thing left now is:
> > > > Can someone explain a motivation for even creating a database row before
> > > > the pristine file is moved into place in the
On Mon, Mar 01, 2010 at 06:14:33PM +, Julian Foad wrote:
> On Thu, 2010-02-18, Neels J Hofmeyr wrote:
> > The one thing left now is:
> > > Can someone explain a motivation for even creating a database row before
> > > the pristine file is moved into place in the pristine store? I currently
> >
On Thu, 2010-02-18, Neels J Hofmeyr wrote:
> Great, moving forward fast on pristine design questions!
Hi Neels.
Did you start working the new knowledge into a document? Lots of stuff
was said in this thread and it would be useful to see where we are at.
I have a couple of comments.
THE PRISTIN
Great, moving forward fast on pristine design questions!
The one thing left now is:
> Can someone explain a motivation for even creating a database row before
> the pristine file is moved into place in the pristine store? I currently
> don't see why it can't be way simpler.
>
> (And, if it is nece
On Wed, Feb 17, 2010 at 18:32, Neels J Hofmeyr wrote:
>...
> +1 -- I was hoping it'd make sense to go that way.
> - "always" internally checksum streams returned by pristine_read()
> (until proven that performance is affected significantly)
> - Allow a boolean flag to switch off this checksumming
Greg Stein wrote:
> On Wed, Feb 17, 2010 at 08:09, Bert Huijben wrote:
>> ...
>>> [ ] It is cheap to tee to a checksumming stream when reading a
>>> stored pristine. The I/O is much slower anyway.
>> When we are reading the complete stream: Yes.
>> I would skip verification if we only do a par
On Wed, Feb 17, 2010 at 08:09, Bert Huijben wrote:
>...
>> [ ] It is cheap to tee to a checksumming stream when reading a
>> stored pristine. The I/O is much slower anyway.
> When we are reading the complete stream: Yes.
> I would skip verification if we only do a partial read (if possible).
> -Original Message-
> From: Neels J Hofmeyr [mailto:ne...@elego.de]
> Sent: woensdag 17 februari 2010 13:29
> To: dev@subversion.apache.org
> Subject: Re: pristine store design
>
> Thanks for the numerous replies! I'd like the design document to soak them
> a
Thanks for the numerous replies! I'd like the design document to soak them
all up, and to that end have a few questions (misunderstandings?) to clarify.
Verification upon using a pristine.
Which of these are/should be true?
[ ] We want to make sure that there is no way that a corrupted pristine
On Mon, Feb 15, 2010 at 10:51, Julian Foad wrote:
>...
>> If you could be so kind to glance over it and straighten out my picture, if
>> necessary. Upon approval, I'll check it into notes/ so we can edit.
>>
>> Note, if my view is correct, this design text implies small changes to the
>> current s
On Mon, Feb 15, 2010 at 08:45, Neels J Hofmeyr wrote:
> Hi all,
>
> taking stock of the current state of the pristine store API and finding
> design docs missing, I have created a "design paper" to clarify.
>
> If you could be so kind to glance over it and straighten out my picture, if
> necessary
Neels J Hofmeyr writes:
> THE PRISTINE STORE
> ==
>
> The pristine store is a local cache of complete content of files that are
> known to be in the repository. It is hashed by a checksum of that content
> (SHA1).
I'm not sure whether you are planning one table per pristine store
Neels J Hofmeyr wrote:
> taking stock of the current state of the pristine store API and finding
> design docs missing, I have created a "design paper" to clarify.
Excellent. This is very useful.
BTW, I am joining the WC-NG effort so I'm glad to jump in here, as it
looks like one of the easier pl
33 matches
Mail list logo
