Re: [COMPRESS] Changeset design
Hi,
I made several changes.
- two files with the same name cannot added to the ChangeSet
- An "add change" command can have a replace = true or replace = false
flag. If true, the add will be added to the stream first. Equal
entries from the stream will be skipped. If false, the stream entrie
will be copied, and then it will be checked if the change can be
applied or not
- two entries with the same name in the stream are not possible any longer
- the ChangeSetPerformer.process method returns a Result object with
several lists of filenames. A user can see where the entries came from
- some tests for this
- added equals methods and hashCode to the ArchiveEntrys for the features above
This should address most problems in this thread - comments welcome!
Cheers,
Christian
On Thu, Apr 16, 2009 at 5:13 PM, Christian Grobmeier
wrote:
>>> I just added testDeletePlusAddSame() and found out that it works like
>>> expected (removes file A and adds file B under the same name as file
>>> A).
>>
>> However, it does not work exactly the same as Add then Delete, because
>> that removes all trace of the Add from the Set.
>
> Yes.
>
> If you make Add and Delete, the Add to a stream should never happen,
> cause you simply cannot delete it later.
>
> If Delete and Add, you have the Deletion from a Stream and an addition
> later. I don't think this order should remove the add, just because
> you called delete before.
>
> I imagine it like commandline.
>
> If you have:
>
>> add file
>> delete file
>
> you have nothing in the stream.
>
> If you make:
>
>> delete file
>> add file
>
> then you have the latest addition in the stream.
>
> Do you think this is bad behaviour?
>
>>> However, this could cause any more trouble if we decide to put the
>>> new file on the start of the archive. This is currently not supported.
>>>
>> Huh? AFAIK, that is what happens now - adds are processed first.
>
> Ah yes, of course, adds first. I had the wrong code in my head. Adds
> are processed first.
>
>>> Yes good idea. + hashCode, we have discussed it before.
>>> When is an ArchiveEntry equal? I think if the name (including any
>>> subfolders) and the size is the same.
>>
>> Not sure size is relevant. Most archivers will have problems coping
>> with multiple entries with the same name.
>
> don't know if it's important, but if one makes:
>
> if(entry.equals(other)) {
> out.putArchiveEntry(entry);
> } else {
> out.putArchiveEntry(other);
> }
>
> I think this is a very obscure usecase and cannot mind anything,
> so I think you are right and we should go on with same name, same entry.
>
>>> How can he know besides if he chooses what to do?
>>>
>>
>> Either an Exception is thrown if the archive was not in the expected
>> state, or the process() method needs to return a list of what was
>> actually processed.
>
> A returned list of actually processed files would surely be cool.
>
> Cheers
>
-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org
[math] MATH-224 - need a better idea
We should be able to find a clean way to do what this enhancement request is asking for. I am feeling stupid because even when I consider breaking compatibility / refactoring to use generics, I can't find a simple way to do it. Here is a description of the current API and some failed ideas that I have considered so far. As usual, I would like to minimize pain for current users in addressing this, but at this point I am starting to think that wholesale refactoring is necessary and I would appreciate ideas on the best way to do this. SummaryStatistics provides "storeless" computation of summary statistics - min, max, mean, variance, etc. Here "storeless" means that the class does not hold the stream of data in memory. It was designed to support pluggable implementations of the statistics that it computes. It does this in a way that looks smelly in the new world of type-safe Java (well, maybe it always smelled ;) The injectable implementation classes in SummaryStatistics are typed as "StorelessUnivariateStatistic" which is an interface that includes things like getResult() and increment(double). There is nothing preventing, for example, a variance implementation from being "plugged in" to implement the mean. The request in MATH-224 is to support aggregation in the following sense: SummaryStatistics instance 1 gets a stream of values and instance 2 gets another stream of values and we want to create a new instance or replace instance 1 with an instance that behaves as though it got all the data from both streams. The simplest way to do this would be to add an "aggregate" method to the StorelessUnivariateStatistic interface and then just implement aggregation in SummaryStatistics by delegation to the implementation instances. This is essentially what the patch attached to MATH-224 does. The problem with this approach is that supporting aggregation is a fairly strong requirement in general, stronger than just requiring that the statistic be computable without storing the data. Stronger still is the requirement that an implementation of a statistic be "aggregatable" with a possibly different implementation (since then it would have access only to the value of the other statistic). So the challenge is can we find a clean way to achieve the four objectives: 0) Maintain pluggability of statistics implementations 1) Support aggregation 2) Improve type safety 3) Minimize trauma for current users Dropping 0) makes things much simpler, but I would like to avoid that unless there is really no way to accomplish 1) and 2) without taking that step. Strictly speaking, 1) may be impossible as I know of no way to support this for the higher moments. I would be OK with aggregation forcing these to NaN (documented, of course). My first thought was to define a parameterized Aggregatable interface that requires the same types. Then two SummaryStatistics instances are aggregatable iff their implementation statistics match types. I am OK with these restrictions, but am having trouble actually making it work. Suggestions / patches welcome! Phil - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [math] MATH-224 - need a better idea
Phil Steitz a écrit : > We should be able to find a clean way to do what this enhancement > request is asking for. I am feeling stupid because even when I consider > breaking compatibility / refactoring to use generics, I can't find a > simple way to do it. Here is a description of the current API and some > failed ideas that I have considered so far. As usual, I would like to > minimize pain for current users in addressing this, but at this point I > am starting to think that wholesale refactoring is necessary and I would > appreciate ideas on the best way to do this. > > SummaryStatistics provides "storeless" computation of summary statistics > - min, max, mean, variance, etc. Here "storeless" means that the class > does not hold the stream of data in memory. It was designed to support > pluggable implementations of the statistics that it computes. It does > this in a way that looks smelly in the new world of type-safe Java > (well, maybe it always smelled ;) The injectable implementation classes > in SummaryStatistics are typed as "StorelessUnivariateStatistic" which > is an interface that includes things like getResult() and > increment(double). There is nothing preventing, for example, a variance > implementation from being "plugged in" to implement the mean. > > The request in MATH-224 is to support aggregation in the following > sense: SummaryStatistics instance 1 gets a stream of values and > instance 2 gets another stream of values and we want to create a new > instance or replace instance 1 with an instance that behaves as though > it got all the data from both streams. The simplest way to do this > would be to add an "aggregate" method to the > StorelessUnivariateStatistic interface and then just implement > aggregation in SummaryStatistics by delegation to the implementation > instances. This is essentially what the patch attached to MATH-224 > does. The problem with this approach is that supporting aggregation is > a fairly strong requirement in general, stronger than just requiring > that the statistic be computable without storing the data. Stronger > still is the requirement that an implementation of a statistic be > "aggregatable" with a possibly different implementation (since then it > would have access only to the value of the other statistic). > > So the challenge is can we find a clean way to achieve the four objectives: > > 0) Maintain pluggability of statistics implementations > 1) Support aggregation > 2) Improve type safety > 3) Minimize trauma for current users > > Dropping 0) makes things much simpler, but I would like to avoid that > unless there is really no way to accomplish 1) and 2) without taking > that step. Strictly speaking, 1) may be impossible as I know of no way > to support this for the higher moments. I would be OK with aggregation > forcing these to NaN (documented, of course). I think 1) has a high priority. It is the whole subject of the issue. I see several use cases for it, including for example parallel computation and later merge. I also think providing it for higher moments requires not only the final result but also intermediate values (typically sum(x^0), sum(x^1), sum(x^2) ...). So this implies these value are available in addition to final results. Perhaps one way to allow it would be to have different interfaces for the various statistics. Mean would provide sum(x^0) and sum(x^1) for example in addition to the final result which is the ratio. The cost for this is a more complex API, but I think it is worth trying. Luc > > My first thought was to define a parameterized Aggregatable interface > that requires the same types. Then two SummaryStatistics instances are > aggregatable iff their implementation statistics match types. I am OK > with these restrictions, but am having trouble actually making it work. > > Suggestions / patches welcome! > > Phil > > > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [math] MATH-224 - need a better idea
I'm looking at commons-math for the first time, but I don't think the feature can be implemented as requested in a manner that is suitably generic. On the other hand, I think the same objective could be achieved a different way without changing the base API at all. The key would be to generate the aggregate statistics at the same time as the per-partition ones, instead of aggregating them after the fact. That does require knowing beforehand that you're going to want the aggregate stats, but I think that's a fair tradeoff. This could be done without making client programs update two sets of statistics with each datum, by wrapping the each StorelessUnivariateStatistic with an implementation that forwards the data to two StorelessUnivariateStatistics -- the wrapped one and one for the aggregate. Almost all the work of setting that up can be automated. I'll see whether I can whip up a proof of concept for you to check out. John From: Phil Steitz To: Commons Developers List Sent: Sunday, April 19, 2009 11:34:24 AM Subject: [math] MATH-224 - need a better idea We should be able to find a clean way to do what this enhancement request is asking for. I am feeling stupid because even when I consider breaking compatibility / refactoring to use generics, I can't find a simple way to do it. Here is a description of the current API and some failed ideas that I have considered so far. As usual, I would like to minimize pain for current users in addressing this, but at this point I am starting to think that wholesale refactoring is necessary and I would appreciate ideas on the best way to do this. SummaryStatistics provides "storeless" computation of summary statistics - min, max, mean, variance, etc. Here "storeless" means that the class does not hold the stream of data in memory. It was designed to support pluggable implementations of the statistics that it computes. It does this in a way that looks smelly in the new world of type-safe Java (well, maybe it always smelled ;) The injectable implementation classes in SummaryStatistics are typed as "StorelessUnivariateStatistic" which is an interface that includes things like getResult() and increment(double). There is nothing preventing, for example, a variance implementation from being "plugged in" to implement the mean. The request in MATH-224 is to support aggregation in the following sense: SummaryStatistics instance 1 gets a stream of values and instance 2 gets another stream of values and we want to create a new instance or replace instance 1 with an instance that behaves as though it got all the data from both streams. The simplest way to do this would be to add an "aggregate" method to the StorelessUnivariateStatistic interface and then just implement aggregation in SummaryStatistics by delegation to the implementation instances. This is essentially what the patch attached to MATH-224 does. The problem with this approach is that supporting aggregation is a fairly strong requirement in general, stronger than just requiring that the statistic be computable without storing the data. Stronger still is the requirement that an implementation of a statistic be "aggregatable" with a possibly different implementation (since then it would have access only to the value of the other statistic). So the challenge is can we find a clean way to achieve the four objectives: 0) Maintain pluggability of statistics implementations 1) Support aggregation 2) Improve type safety 3) Minimize trauma for current users Dropping 0) makes things much simpler, but I would like to avoid that unless there is really no way to accomplish 1) and 2) without taking that step. Strictly speaking, 1) may be impossible as I know of no way to support this for the higher moments. I would be OK with aggregation forcing these to NaN (documented, of course). My first thought was to define a parameterized Aggregatable interface that requires the same types. Then two SummaryStatistics instances are aggregatable iff their implementation statistics match types. I am OK with these restrictions, but am having trouble actually making it work. Suggestions / patches welcome! Phil - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [math] MATH-224 - need a better idea
That is a fine answer for some things, but the parallel cases fail. My feeling is that there are a few cases where there are nice aggregatable summary statistics like moments and there are many cases where this just doesn't work well (such as rank statistics). For the latter, case I usually make do with a surrogate such as a random sub-sample or a recency weighted random sub-sample combined with a few aggregatable stats such as total samples, max, min, sum and second moment. That gives me most of what I want and if the sub-sample is reasonably large, I can sometimes estimate a few parameters such as total uniques. The sub-sampled data streams can be combined trivially so I now have a aggregatable approximation of non-aggregatable statistics. For descriptive quantiles this is generally just fine. On Sun, Apr 19, 2009 at 2:44 PM, John Bollinger wrote: > The key would be to generate the aggregate statistics at the same time as > the per-partition ones, instead of aggregating them after the fact. -- Ted Dunning, CTO DeepDyve
