Hello everyone,
Here's a small report on the news that I have so far:
1. I had a slow couple of weeks because of a quick holiday that I took.
I will make up for that.
2. I added the metric that considers number of test_runs since a
test_case ran for the last time. I graphed it, and it does not affect the
results much at all. -I still think this is useful to uncover hidden bugs
that might lurk in the code for a long time; but testing this condition is
difficult with our data. I would like to keep this measure, specially since
it doesn't seem to affect results negatively. Opinions?
3. I liked Sergei's idea about using the changes on the test files to
calculate the relevancy index. If a test has been changed recently, its
relevancy index should be high. This is also more realistic, and uses
information that it's easy for us to figure out.
- I am trying to match the change_files table or the mysql-test
directory with the test_failure table. I was confused about the name of
tests and test suites, but I am making progress on it. Once I am able to
match at least 90% of the test_names in test_failure with the filename in
the change_files table, I will incorporate this data into the
code and see
how it works out.
- *Question*: Looking at the change_files table, there are files that
have been *ADDED* several times. Why would this be? Maybe when a new
branch is created, all files are ADDED to it? Any ideas? : ) (If no one
knows, I'll figure it out, but maybe you do know ; ))
4. I uploaded inline comments for my code last week, let me know if it's
clear enough. You can start by run_basic_simulations.py, where the most
important functions are called... and after, you can dive into
basic_simulator.py, where the simulation is actually done. The repository
is a bit messy, I admit. I'll clean it up in the following commits.
This is all I have to report for now. Any advice on the way I'm proceeding
is welcome : )
Have a nice week, everyone.
Pablo
On Sun, May 25, 2014 at 11:43 PM, Sergei Golubchik <s...@mariadb.org> wrote:
> Hi, Pablo!
>
> On May 25, Pablo Estrada wrote:
> > On Thu, May 22, 2014 at 5:39 PM, Sergei Golubchik <s...@mariadb.org>
> wrote:
> > > I don't think you should introduce artificial limitations that make
> > > the recall worse, because they "look realistic".
> > >
> > > You can do it realistic instead, not look realistic - simply pretend
> > > that your code is already running on buildbot and limits the number
> > > of tests to run. So, if the test didn't run - you don't have any
> > > failure information about it.
> > >
> > > And then you only need to do what improves recall, nothing else :)
> > >
> > > (of course, to calculate the recall you need to use all failures,
> > > even for tests that you didn't run)
> >
> > Yes, my code *already works this way*. It doesn't consider failure
> > information from tests that were not supposed to run. The graphs that
> > I sent are from scripts that ran like this.
>
> Good. I hoped that'll be the case (but didn't check your scripts on
> github yet, sorry).
>
> > Of course, the recall is just the number of spotted failures from the
> 100%
> > of known failures : )
> >
> > Anyway, with all this, I will get to work on adapting the simulation a
> > little bit:
> >
> > - Time since last run will also affect the relevancy of a test
> > - I will try to use the list of changed files from commits to make
> sure
> > new tests start running right away
> >
> > Any other comments are welcome.
>
> Getting back to your "potential fallacy" at how you start taking tests
> into account only when they fail for the first time...
>
> I agree, in real life we cannot do that. Instead we start from a
> complete list of tests, that is known in advance. And you don't have it,
> unfortunately.
>
> An option would be to create a complete list of all tests that have
> ever failed (and perhaps remove tests that were added in some revision
> present in the history). And use that as a "starting set" of tests.
>
> Alternatively, we can generate a list of all tests currently present in
> the 10.0 tree - everything that you have in the history tables should be
> a subset of that.
>
> Regards,
> Sergei
>
>
_______________________________________________
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help : https://help.launchpad.net/ListHelp
