"Pace is nothing without guile"

[Neil Jerram]

... That's a comment from coverage of the current England v South
Africa cricket match
(http://uk.cricinfo.com/talk/content/current/multimedia/360921.html).

But is Guile nothing without pace?

Well obviously it isn't "nothing", but I think Guile is perceived,
among both Scheme implementations and free scripting languages, as
being a bit slow, and I think that a large part of the reason for this
is that we have no systematic benchmarking.

So this email is about systematic performance data.  I was wondering
what benchmarks we could run to get good coverage of all Guile's
function, and suddenly thought "of course, the test suite!"  The test
suite should, by definition, provide coverage of everything that we
care about.  Therefore I think that we should be able to start
collecting a lot of useful performance data by implementing a version
of "make check" that measures and stores off the time that each test
takes to run.

What I'd like input/advice on, is exactly how we store and collate
such data.  I think the system should ideally support

- arbitrary later analysis of the collected data

- correlation of the result for a specific test with the exact source
code of that test at the time it was run...

- ...and hence, being able to work out (later) that the results
changed because the content of the test changed

- anyone running the tests and uploading data, not just Guile core developers

- associating a set of results with the relevant information about the
machine that they were obtained on (CPUs, RAM) in such a way that the
information is trustable, but without invading the privacy of the
uploader.

So how do we do that?  Perhaps the test content identification could
be done by its Git (SHA-1) hash - together with the path of the repo
containing that version.  And I imagine that the form of the results
could be a file containing lines like:

("numbers.test" SHA1-HASH REPO-PATH DATE+TIME MACHINE-INFO MEASURED-DURATION)

That would allow sets of results to be concatenated for later
analysis.  But I'm not sure what the relevant MACHINE-INFO is and how
to represent that.

Any thoughts / comments / ideas?  Thanks for reading!

  Neil

Re: "Pace is nothing without guile"

[Greg Troxel]

My immediate reaction is that test suites aren't good benchmarks because
we will often want to add to test suites, while changing the benchmark
invalidates previous data so we will not want to change the benchmark.

Now, if you mean to use the test suite as a collection of
micro-benchmarks, so that we just have a rule that individual tests
aren't modified without serious cause, but new ones can be added, then
that makes senes.

Re: "Pace is nothing without guile"

[Neil Jerram]

2008/7/13 Greg Troxel <[EMAIL PROTECTED]>:
> My immediate reaction is that test suites aren't good benchmarks because
> we will often want to add to test suites, while changing the benchmark
> invalidates previous data so we will not want to change the benchmark.

Yes, but...

> Now, if you mean to use the test suite as a collection of
> micro-benchmarks, so that we just have a rule that individual tests
> aren't modified without serious cause, but new ones can be added, then
> that makes senes.

Yes, this is what I meant.  And I also think there will be enough test
cases not changing, over the long term, that we won't have to worry in
practice about changing a few here and there.

I've taken a better look now at the existing benchmark-suite - which
doesn't contain very many benchmarks, but does provide a sensible
discussion of timings, and useful library functions.  I don't think it
will be very hard to somehow incorporate the test-suite into the set
of benchmarks.

Regards,
  Neil

Re: Closure?

[Neil Jerram]

2008/7/13 Maciek Godek <[EMAIL PROTECTED]>:
>> except that the last line fails with a "Bad define placement" error.
>> That's because there are special rules for defines inside lexical
>> scopes.
>
> As the practise shows, although guile documentation says something
> different. In section 3.1.4.7 (A Shared Persistent Variable)
>
> "An important detail here is that the `get-balance' and `deposit'
> variables must be set up by `define'ing them at top level and then
> `set!'ing their values inside the `let' body.  Using `define' within
> the `let' body would not work: this would create variable bindings
> within the local `let' environment that would not be accessible at top
> level."
>
> So one might conclude that it _is_ possible to use define inside
> a 'let' form.

Which would be correct!  For example:

(let ((a 1))
  (define b 2)
  (+ a b))
=>
3

Whereas:

(let ((a 1))
  (display a)
  (newline)
  (define b 2)
  (+ a b))
=>
ERROR: Bad define placement

The "special rules" are just that  any defines have to come before
anything else in the body of the let.

I don't know exactly how it works out that using a define in
local-eval falls foul of the define placement rule, but it is not hard
to imagine that it could do.

> Yes, since there's local-eval and the-environment, everything I've
> ever dreamed of is possible :)
> But as I've concluded from the discourse,  neither of these is
> defined in R5RS (and it makes me wonder)

Well I've never thought this through before, but perhaps that is
because in many cases it is equivalent to create a lambda at the point
where you would call the-environment, containing the code that you
would later pass to local-eval.

For example, the ++ example then becomes:

(define ++ (let ((c 0)) (lambda () (begin (set! c (+ c 1)) c

- which is the traditional way of writing this example.

Regards,
 Neil

Re: Closure?

[Maciek Godek]

>> As the practise shows, although guile documentation says something
>> different. In section 3.1.4.7 (A Shared Persistent Variable)
>>
>> "An important detail here is that the `get-balance' and `deposit'
>> variables must be set up by `define'ing them at top level and then
>> `set!'ing their values inside the `let' body.  Using `define' within
>> the `let' body would not work: this would create variable bindings
>> within the local `let' environment that would not be accessible at top
>> level."
>>
>> So one might conclude that it _is_ possible to use define inside
>> a 'let' form.
>
> Which would be correct!  For example:
>
> (let ((a 1))
>  (define b 2)
>  (+ a b))
> =>
> 3
>
> Whereas:
>
> (let ((a 1))
>  (display a)
>  (newline)
>  (define b 2)
>  (+ a b))
> =>
> ERROR: Bad define placement
>
> The "special rules" are just that  any defines have to come before
> anything else in the body of the let.

Yeah, guess you're right (under certain circumstances :P)

> I don't know exactly how it works out that using a define in
> local-eval falls foul of the define placement rule, but it is not hard
> to imagine that it could do.

The other question is: is it really necessary to impose such
limitations on "define". Why is it required to make its position
inside let privileged?

>> Yes, since there's local-eval and the-environment, everything I've
>> ever dreamed of is possible :)
>> But as I've concluded from the discourse,  neither of these is
>> defined in R5RS (and it makes me wonder)
>
> Well I've never thought this through before, but perhaps that is
> because in many cases it is equivalent to create a lambda at the point
> where you would call the-environment, containing the code that you
> would later pass to local-eval.
>
> For example, the ++ example then becomes:
>
> (define ++ (let ((c 0)) (lambda () (begin (set! c (+ c 1)) c
>
> - which is the traditional way of writing this example.

You didn't focus :>
The whole idea of accessing a closure environment
was in fact to make scheme object oriented
programming more intuitive.

In guile info pages there's an oo closure example:

(section 3.1.4.9 "Example 4: Object Orientation")
"
 (define (make-account)
   (let ((balance 0))
 (define (get-balance)
   balance)
 (define (deposit amount)
   (set! balance (+ balance amount))
   balance)
 (define (withdraw amount)
   (deposit (- amount)))

 (lambda args
   (apply
 (case (car args)
   ((get-balance) get-balance)
   ((deposit) deposit)
   ((withdraw) withdraw)
   (else (error "Invalid method!")))
 (cdr args)

 (define my-account (make-account))
"
Notice the ugly "case" statement that requires
the variables to be accessed in the following manner
(the same example, a few lines later):
"
 (my-account 'get-balance)
 =>
 0

 (my-account 'withdraw 5)
 =>
 -5

 (my-account 'deposit 396)
 =>
 391

 (my-account 'get-balance)
 =>
 391
"

This is ugly as it requires doubling the names of functions.
Perhaps it could be overcome with some sort of macro,
but the "with" I proposed allows to avoid the whole "case"
and to write (after slight modifications in the "let" form):

(with my-account (get-balance))

Or maybe I think wrong; I'm new in the world of lisp,
so please forgive me my mistakes :)

Best regards,
M