Re: Guile foreign object interface
() David Kastrup () Thu, 09 Mar 2017 00:00:48 +0100 [...] rather than [...] fork Guile 1.8 in order to actually have some dependable functionality to base other work on. I intend to maintain 1.8 for the time being. More precisely, i seek to apply bug fixes, improve documentation (both method and content), modernize the build system (tracking gnulib, autotools evolution), and (time and gumption permitting) refactor some internals. If the gods smile, maybe a feature or two (always w/ an eye on upward compatibility). If the changes to 1.8 that Lilypond requires fall into the above categories, perhaps we can avoid a fork. Do you have a summary of those changes handy? -- Thien-Thi Nguyen --- (defun responsep (query) (pcase (context query) (`(technical ,ml) (correctp ml)) ...)) 748E A0E8 1CB8 A748 9BFA --- 6CE4 6703 2224 4C80 7502 signature.asc Description: PGP signature
Re: Lilypond speed (was Re: How to make GNU Guile more successful)
() Thomas Morley () Thu, 9 Mar 2017 00:17:12 +0100 Btw, I've improved my local setup to be able to test lilypond more quickly with different guile versions. Though I wasn't able to compile 1.8.8, neither from the repository Strange, because the repo should have... nor from the tarball downloaded from https://www.gnu.org/software/guile/download/ Due to: async.c: In function 'scm_i_queue_async_cell': async.c:243:14: error: variable 'count' set but not used [-Werror=unused-but-set-variable] size_t count; ^ Am I missing something? ...a fix installed from 2012-05-01: http://git.savannah.gnu.org/cgit/guile.git/commit/?h=branch_release-1-8&id=e2547476441 among other, more recent, changes. Note that this is on branch ‘branch_release-1-8’. Perhaps you were on a different branch? -- Thien-Thi Nguyen --- (defun responsep (query) (pcase (context query) (`(technical ,ml) (correctp ml)) ...)) 748E A0E8 1CB8 A748 9BFA --- 6CE4 6703 2224 4C80 7502 signature.asc Description: PGP signature
Re: Guile foreign object interface
David Kastrup skribis: > l...@gnu.org (Ludovic Courtès) writes: > >> David Kastrup skribis: >> >>> l...@gnu.org (Ludovic Courtès) writes: >>> Andy Wingo skribis: > I am not so sure about about this one. I think it's not accurate to > characterize beginning to replace a 25-year-old C API (SMOBs) as > "churn". I think the point is that there’s lots of code out there that rely on SMOBs and we shouldn’t break it overnight, precisely because that API is this old. Of course, I agree that pushing users towards an improved API is the right thing long term, no argument here. >>> >>> Shrug. LilyPond has all of its SMOB usage condensed into few C++ >>> classes, so it is comparatively easy to migrate to a different API as >>> long as it offers comparable functionality. >>> >>> Which it doesn't (namely the ability of marking objects reached through >>> STL-managed data structures). So it's pretty pointless to "push users >>> towards an improved API" and hardly "the right thing". >> >> Thanks for your feedback, this is a kind of use case we’d like to >> support. >> >> I’m sure we already discussed it and then I forgot, but would anything >> prevent the use of specific C++ allocators in this case? The STL data >> structures could be allocated on GC-scanned memory, in which case mark >> procedures are unneeded. > > We are not talking about STL data structures containing SCM values but > data structures containing other data structures and pointers to other > data structures. You cannot sensibly garbage collect when whole memory > areas in which allocation and deallocation is done are in GC-scanned > memory and there is no difference between data structures that have been > freed and data structures that haven't. Not sure what you mean. >> If this cannot be done, there’s always the possibility of having a >> weak hash table to keep the C++ and Scheme object life cycles in sync. >> That happens even with C APIs that record a pointer to an object we >> care about in non-scanned memory. Since 2.0 was out I’ve never used >> mark procedures for in C bindings. >> >> Thoughts? > > LilyPond is a large-scale application which will eat up a significant > ratio of the available address space in 32 bit applications and will > process, in a documentation run, thousands of files with little overlap. > Already in Guile 1.8 where _only_ the stack is conservatively marked and > gc is called explicitly at known points of low-stack usage, we have > occasional false positives in garbage collection (the debugging runs > flag them). With large memory areas being conservatively scanned, this > is going to get a whole lot worse. This sounds like speculation to me. I don’t think we’ve seen slowdowns when Guile then 1.9 switched to BDW-GC, quite the opposite. > The point of time where it makes sense to try to evaluate the impact or > feasibility of such functionality removals (and Boehm GC _does_ support > finalization) is when LilyPond is otherwise running fine on Guile 2. > With the current slowdown by a factor of 5-10 and a number of other > problems preventing the LilyPond test suite from running, there just > isn't any setup where a useful evaluation could take place. LilyPond on Guile 2.0 is 5–10 times slower than on 1.8? > As there is no technical necessity for planning the wreckage of existing > functionality when there hasn't even been a stable release containing > the prospective successor, I cannot see a viable point in raising yet > more hurdles for existing applications to migrate to Guile 2 rather than > bite the bullet and fork Guile 1.8 in order to actually have some > dependable functionality to base other work on. Note that SMOBs are still around in 2.2, not even deprecated, only discouraged. >> As an aside, please keep the tone friendly as is the norm on this >> mailing list. > > Disagreement is not the same as unfriendliness. I agree. However I found the tone of your messages patronizing and aggressive, assuming bad faith and incompetence on the side of the Guile developers (“planning for wreckage”, “pretty pointless”, etc. etc.) This is not OK on the Guile mailing lists. Thanks, Ludo’.
Re: Lilypond speed (was Re: How to make GNU Guile more successful)
Hello, Thomas Morley skribis: > Btw, I've improved my local setup to be able to test lilypond more > quickly with different guile versions. Though I wasn't able to compile > 1.8.8, neither from the repository nor from the tarball downloaded > from > https://www.gnu.org/software/guile/download/ > Due to: > async.c: In function 'scm_i_queue_async_cell': > async.c:243:14: error: variable 'count' set but not used > [-Werror=unused-but-set-variable] >size_t count; > ^ > > Am I missing something? Could you try configuring like this: ./configure --disable-error-on-warning ? > I'm aware noone is interested in developing 1.8.8 further, though I > would have prefered to build lilypond with that version as well, like > the other test-versions. The performance gap in LilyPond between 1.8 and 2.0 is terrible. I suppose LilyPond uses ‘eval’ to run Scheme code? What fraction of the Scheme code being run for this benchmark is pre-compiled (as a .go file)? Is auto-compilation enabled, and could it be that the figures include auto-compilation time? Thanks, Ludo’.
Re: Lilypond speed (was Re: How to make GNU Guile more successful)
On 03/09/2017 07:13 AM, Ludovic Courtès wrote: What fraction of the Scheme code being run for this benchmark is pre-compiled (as a .go file)? I don't think any of LilyPond's Scheme code is pre-compiled at this point... Yep, as David Kastrup wrote in the "GNU Guile 2.1.7 released (beta)" thread on Feb 28, 2017: Regular read and eval, however, is used a lot during the parsing of files and startup of LilyPond. But at least under Guile-1.8, the parsing and preprocessing took up a rather small part of the overall runtime (in the order of 15% or so), so it is unlikely to be responsible for the bulk of the slowdown. My personal guess is that the largest performance impact at the moment will be due to an absence of generation and installation of .go files. Since .go files are target-dependent (if I am not mistaken) and LilyPond is cross-compiled for a number of architectures with different byte orders and type sizes, it seems tricky to get this under wraps. The next largest performance impact will be redecoding issues. -Paul
Re: The status of JIT compiler of Guile
On Thu 09 Mar 2017 07:59, Nala Ginrut writes: > I've discussed with @Ludo, I think it's better to integrate it as a > plugin, and could be maintained separately. > It is possible to have some hooks in Guile VM to enable certain > optimizing, but I don't think we have it now, right? This sounds like a great plan :) For me, in an ideal world, Guile-TJIT would be a thing you just use if you need it, and "using it" is just loading up a library or something. If peoples' experience with TJIT is really positive then we can look at integrating it more. Of course it might not be possible to add TJIT support in this decoupled way -- we will see :) Cheers, Andy
Re: Guile foreign object interface
> From: l...@gnu.org (Ludovic Courtès) > Date: Thu, 09 Mar 2017 13:09:40 +0100 > > >> As an aside, please keep the tone friendly as is the norm on this > >> mailing list. > > > > Disagreement is not the same as unfriendliness. > > I agree. However I found the tone of your messages patronizing and > aggressive, assuming bad faith and incompetence on the side of the Guile > developers (“planning for wreckage”, “pretty pointless”, etc. etc.) FYI, I've communicated (and occasionally disagreed) with David for many years, and I can assure you that you see something that simply isn't there. He sometimes uses such "colorful" descriptions to make a point more clear, that's all. People should be allowed to use their personal style when writing, without being reprimanded, IMO.
Re: Guile foreign object interface
Eli Zaretskii skribis: >> From: l...@gnu.org (Ludovic Courtès) >> Date: Thu, 09 Mar 2017 13:09:40 +0100 >> >> >> As an aside, please keep the tone friendly as is the norm on this >> >> mailing list. >> > >> > Disagreement is not the same as unfriendliness. >> >> I agree. However I found the tone of your messages patronizing and >> aggressive, assuming bad faith and incompetence on the side of the Guile >> developers (“planning for wreckage”, “pretty pointless”, etc. etc.) > > FYI, I've communicated (and occasionally disagreed) with David for > many years, and I can assure you that you see something that simply > isn't there. He sometimes uses such "colorful" descriptions to make a > point more clear, that's all. People should be allowed to use their > personal style when writing, without being reprimanded, IMO. I’m all for personal style, but I’m against passive-aggressive or downright aggressive style. The problem is not whether the person who writes is well-meaning or not; the problem is how others perceive it. If I, as an old-timer and maintainer, feel attacked when reading these messages, I can only think that newcomers may feel uncomfortable joining the conversation, at best. We’d be doing a disservice to our group by sending the message that it’s OK to be harsh to others. Ludo’.
Re: Guile foreign object interface
> From: l...@gnu.org (Ludovic Courtès) > Cc: guile-user@gnu.org > Date: Thu, 09 Mar 2017 18:26:09 +0100 > > > FYI, I've communicated (and occasionally disagreed) with David for > > many years, and I can assure you that you see something that simply > > isn't there. He sometimes uses such "colorful" descriptions to make a > > point more clear, that's all. People should be allowed to use their > > personal style when writing, without being reprimanded, IMO. > > I’m all for personal style, but I’m against passive-aggressive or downright > aggressive style. That's what I'm trying to tell you: there's no aggression. I once thought like you, but experience of communicating with David taught me I was wrong. > The problem is not whether the person who writes is well-meaning or not; > the problem is how others perceive it. I'm saying that you are reading into David's words something that isn't there, not only in meaning, but also not in form. You are misinterpreting his words, which might not be surprising, given that we all are trying to communicate in a language that is not our first one. Things might sound aggressive when they really aren't. > If I, as an old-timer and maintainer, feel attacked when reading > these messages I don't see why you should feel attacked. FWIW, I see no attack in David's messages. He's opinionated, that's for sure. But there's a difference between that and an attack. > We’d be doing a disservice to our group by sending the message that it’s > OK to be harsh to others. Indeed, we would. But there's nothing particularly harsh in David's messages. I think it would make this list friendlier if people's words are not taken as an attack just because they disagree with the project leaders. We should be able to disagree and still stay colleagues in our common quest.
Re: Guile foreign object interface
On Thu 09 Mar 2017 19:31, Eli Zaretskii writes: >> From: l...@gnu.org (Ludovic Courtès) >> Cc: guile-user@gnu.org >> Date: Thu, 09 Mar 2017 18:26:09 +0100 >> >> I’m all for personal style, but I’m against passive-aggressive or downright >> aggressive style. > > That's what I'm trying to tell you: there's no aggression. I understand that different people can have different reactions and it's great that you can look through "style" to the substance. I and a number of other contributors (evidently including Ludovic) find it hard to do so, and the only reason we try is because we care about Guile. It's really weird though to try to ignore this "style" when the style often says precisely that we _don't_ care, in those words, and other times in as many words. Style also has an effect on how willing I am to work with someone, and with their work. I think for the similar reasons that we should encourage good coding styles (that it makes it easy to read and understand what we are doing), we should encourage good communications styles as well, and give negative feedback on those styles that need work. Finally, as Ludovic mentioned, one person's style has an effect on the composition of the body of Guile users and developers. Whether someone feels welcome in a group is a function of lots of subtle and not-so-subtle things. Feeling welcome is, for most people, a prerequisite to further involvement. Guile does OK here -- some good aspects, some needs-improvement aspects -- but I think we can all think of examples of online communities that have atrophied over time, and a hostile online environment can be one of the reasons for this; intervention of maintainers to avoid this failure mode should be expected, I think. I know it's a complicated issue. In the particular case of the mail that Ludovic responded to, I agree with his message and think he did a fine job -- focussing on issues but also making clear that we should be kind to each other in the way we communicate. It was more charitable than I was going to be :) Regards, Andy
Re: Guile foreign object interface
Thien-Thi Nguyen writes: > () David Kastrup > () Thu, 09 Mar 2017 00:00:48 +0100 > >[...] rather than [...] fork Guile 1.8 in order to actually >have some dependable functionality to base other work on. > > I intend to maintain 1.8 for the time being. More precisely, i > seek to apply bug fixes, improve documentation (both method and > content), modernize the build system (tracking gnulib, autotools > evolution), and (time and gumption permitting) refactor some > internals. If the gods smile, maybe a feature or two (always w/ > an eye on upward compatibility). > > If the changes to 1.8 that Lilypond requires fall into the above > categories, perhaps we can avoid a fork. Do you have a summary > of those changes handy? The changes to 1.8 that LilyPond requires is that Guile-1.8 needs to be compilable with newer compiler versions and standards and that distributions are confident enough about it to keep distributing it. That's all: LilyPond works fine with Guile-1.8 as it is. Yes, migrating to some sort of UTF-8 capable string scheme transcending the "byte array" approach of Guile-1 would be nice but the Guile-2 approach would likely be a major performance hog and it does not make sense for a Guile-1.8 fork to develop a different approach: any such "different approach" fork should, if at all, focus on the Guile-2 code base as a starting point. Unless there is an approach that is so lightweight that it just fits better when starting with the Guile-1.8 code base. But all in all the most important thing LilyPond would want from Guile-1.8 is not to die. -- David Kastrup
Re: Guile foreign object interface
l...@gnu.org (Ludovic Courtès) writes:
> David Kastrup skribis:
>
>> l...@gnu.org (Ludovic Courtès) writes:
>>
>>> I’m sure we already discussed it and then I forgot, but would
>>> anything prevent the use of specific C++ allocators in this case?
>>> The STL data structures could be allocated on GC-scanned memory, in
>>> which case mark procedures are unneeded.
>>
>> We are not talking about STL data structures containing SCM values
>> but data structures containing other data structures and pointers to
>> other data structures. You cannot sensibly garbage collect when
>> whole memory areas in which allocation and deallocation is done are
>> in GC-scanned memory and there is no difference between data
>> structures that have been freed and data structures that haven't.
>
> Not sure what you mean.
The code is out there. Typical stuff looks like
SCM
Context::mark_smob () const
{
scm_gc_mark (context_list_);
scm_gc_mark (aliases_);
scm_gc_mark (definition_);
scm_gc_mark (definition_mods_);
scm_gc_mark (properties_scm_);
scm_gc_mark (accepts_list_);
scm_gc_mark (default_child_);
if (implementation_)
scm_gc_mark (implementation_->self_scm ());
if (event_source_)
scm_gc_mark (event_source_->self_scm ());
if (events_below_)
scm_gc_mark (events_below_->self_scm ());
derived_mark ();
return properties_scm_;
}
The first members are marked directly as SCM. The others are marked
indirectly.
There is also stuff like
void
Slur_engraver::derived_mark () const
{
for (vsize i = start_events_.size (); i--;)
{
scm_gc_mark (start_events_[i].slur_->self_scm ());
if (start_events_[i].note_)
scm_gc_mark (start_events_[i].note_->self_scm ());
}
for (vsize i = stop_events_.size (); i--;)
{
scm_gc_mark (stop_events_[i].slur_->self_scm ());
if (stop_events_[i].note_)
scm_gc_mark (stop_events_[i].note_->self_scm ());
}
}
where the involved arrays are actually STL vectors.
>> LilyPond is a large-scale application which will eat up a significant
>> ratio of the available address space in 32 bit applications and will
>> process, in a documentation run, thousands of files with little overlap.
>> Already in Guile 1.8 where _only_ the stack is conservatively marked and
>> gc is called explicitly at known points of low-stack usage, we have
>> occasional false positives in garbage collection (the debugging runs
>> flag them). With large memory areas being conservatively scanned, this
>> is going to get a whole lot worse.
>
> This sounds like speculation to me. I don’t think we’ve seen
> slowdowns when Guile then 1.9 switched to BDW-GC, quite the opposite.
This is not a problem of slowdowns but of unreclaimed memory because of
false positives in reference detections. LilyPond has large sessions
where it basically frees all memory after the startup thousands of times
over.
>> The point of time where it makes sense to try to evaluate the impact
>> or feasibility of such functionality removals (and Boehm GC _does_
>> support finalization) is when LilyPond is otherwise running fine on
>> Guile 2. With the current slowdown by a factor of 5-10 and a number
>> of other problems preventing the LilyPond test suite from running,
>> there just isn't any setup where a useful evaluation could take
>> place.
>
> LilyPond on Guile 2.0 is 5–10 times slower than on 1.8?
Yes.
>> As there is no technical necessity for planning the wreckage of
>> existing functionality when there hasn't even been a stable release
>> containing the prospective successor, I cannot see a viable point in
>> raising yet more hurdles for existing applications to migrate to
>> Guile 2 rather than bite the bullet and fork Guile 1.8 in order to
>> actually have some dependable functionality to base other work on.
>
> Note that SMOBs are still around in 2.2, not even deprecated, only
> discouraged.
I have no problem with discouragement.
>>> As an aside, please keep the tone friendly as is the norm on this
>>> mailing list.
>>
>> Disagreement is not the same as unfriendliness.
>
> I agree. However I found the tone of your messages patronizing and
> aggressive, assuming bad faith and incompetence on the side of the
> Guile developers (“planning for wreckage”, “pretty pointless”,
> etc. etc.)
>
> This is not OK on the Guile mailing lists.
Given the current state of things and the current discussion and
wordings being used, the terms I used are describing the effects as they
appear when viewed through the lens of LilyPond development. I saw no
convincing _technical_ reason for the enthusiasm towards stopping users
from being able to use their existing code without a workable
replacement in light of the existing mechanisms.
As I said: LilyPond has its Guile object management encapsulated and
modularized to a degree where it was simple to implement some finalizer
workaround on the LilyPond side until the problem finally also got fixed
in Guile 2.0.13 or so.
But it woul
Re: How to make GNU Guile more successful
Hi, I'm relatively new to Guile and new to this list, but I saw this thread in the archives and I'd like to make some comments. First, I think it would be nice to have a Guile wiki, in the likes of cliki.net. A publicly-editable reference place where people can share knowledge about Guile and Guile libraries and projects. Sure, having a centralized package manager/repository would be nice, but just having a common place where people can look up what libraries exist and where they can be downloaded would be nice. (I don't know how easy would it be to set up a wiki at the Guile website.) On the topic of package management, I think that, more important than a central repository, is a standard format in which to distribute packages for easy installation. I've never used guildhall, and I guess it defines something like this, but I think just having an installation command where you can pass the URL of a package to download and install (say, guild install http://some.url/somepackage.zip, or even a git URL) would already go a long way towards making it easier for people to use third-party libraries in Guile. (Or maybe package _names_ could be URLs pointing to an index in a well-defined format, which tells how to download each version of a package; then we could have even dependency management without a central repository. I'm pipe-dreaming here, though.) Best regards, Vítor De Araújo -- Vítor De Araújo http://inf.ufrgs.br/~vbuaraujo
Re: The status of JIT compiler of Guile
Hi Andy! I have similar idea, TJIT could be an optional package and people may choose to install it seperately. We may choose to integrate it when we thought it's mature enough. In a conservative way, how about we keep the current status, say, I'll keep maintaining guilt-tjit as separated project, and rebasing it with the current master. Then you're not so hurry to take care of TJIT to implement the hook mechanism as we thought. The only problem is people may have to install two Guile on their system. But it's fine IMO. People who just wants to play/learn Guile may install guile-tjit. And people who wants to use Guile seriously (like me) should use upstream Guile. Anyway, in principle, if people don't enable --tjit when start Guile, then it should be the latest regular Guile even they use guile-tjit. But it's just in principle, I need more research to see if there's anything different. That's why I suggest people in serious use case should choose upstream Guile. WDYT? Best regards. On Thu, Mar 9, 2017 at 10:27 PM, Andy Wingo wrote: > On Thu 09 Mar 2017 07:59, Nala Ginrut writes: > >> I've discussed with @Ludo, I think it's better to integrate it as a >> plugin, and could be maintained separately. >> It is possible to have some hooks in Guile VM to enable certain >> optimizing, but I don't think we have it now, right? > > This sounds like a great plan :) For me, in an ideal world, Guile-TJIT > would be a thing you just use if you need it, and "using it" is just > loading up a library or something. If peoples' experience with TJIT is > really positive then we can look at integrating it more. Of course it > might not be possible to add TJIT support in this decoupled way -- we > will see :) > > Cheers, > > Andy
Re: Guile foreign object interface
> From: Andy Wingo > Cc: l...@gnu.org (Ludovic Courtès), guile-user@gnu.org > Date: Thu, 09 Mar 2017 20:56:09 +0100 > > > That's what I'm trying to tell you: there's no aggression. > > I understand that different people can have different reactions and it's > great that you can look through "style" to the substance. I and a > number of other contributors (evidently including Ludovic) find it hard > to do so, and the only reason we try is because we care about Guile. > It's really weird though to try to ignore this "style" when the style > often says precisely that we _don't_ care, in those words, and other > times in as many words. Given the certain amount of frustration over the real problems that didn't get solved until now, you can understand that, don't you? I had my share of problems reported here, mostly with working patches, some of which took many moons, sometimes years to get solved upstream. So I think I understand some of David's frustration, although the problems I reported were nowhere near the gravity of those Lilypond has. So maybe the project leadership should try to improve the efficiency of handling bug reports and of catering to the problems raised by projects using Guile, as a means to lower frustration and make the environment friendlier?
