Hi All,
Thanks, Grigori, for mentioning my plugin system, libplugin, which
can be found at http://libplugin.sourceforge.net/.
I have been meaning to release it but finding the time to finish
off the documentation and upload all the newest code to SourceForge
has been difficult (both code and docs on SourceForge are some months
out of date).
The plugin system was built to support MilePost goals for GCC, as
we need to be able to capture events during compilation as well as
change compilation behaviour. Here are some of the features of the
system:
*Application agnostic.*
The plugin system is not GCC specific but can be used to make any
application plugin aware. Plugin management is handled through a
shared library which GCC, or any other application, can link to.
I think if GCC took a similar approach then it would benefit from
the exposure the system received elsewhere and the wider community
would also have access to a professionally built plugin system. As
the plugin system becomes more powerful, GCC reaps the rewards without
having to change a line of code.
The other, huge advantage of this, together with the design I'll
describe below, is that GCC only has ~10 lines of plugin code; to
initialise the library. The rest is working out how to refactor GCC
to make it extensible. This way GCC won't be cluttered with nasty
plugin stuff that obscures the meaning of the code.
Finally, plugins for different applications can coexist. For
example, we might have some plugins for the linker, some for driver,
some for compiler and some that work in any of those.
*Eclipse Inspired.*
I've take inspiration from the excellent plugin system for the
Eclipse IDE, http://www.eclipse.org. It has proved very successful,
so it seemed like a good starting point.
*What it is.*
* Each plugin has an XML file describing it
* Plugins have ExtensionPoints that other plugins can extend
* Plugins can have shared libraries
* Requires libxml2, libffi, libltdl
An ExtensionPoint is one of the fundamental parts of the system.
It provides the links between plugins.
Each ExtensionPoint is really just an object with one method:
bool extend( ExtensionPoint* self, Plugin* extendingPlugin,
xmlNodePtr specfication )
This method tells the ExtensionPoint that some other plugin is
extending it and gives it the XML that plugin uses. The
ExtensionPoint can do whatever it likes with that XML. It might
contain symbols pointing to functions it should use, it might be
markup for logging text. It could be a list of unroll factors, one
for each function or a list of passes to apply to a particular
function. You can describe anything in XML.
*
An Example.*
Maybe that's a bit confusing, so here's an example.
Suppose we have a plugin which offers a logging or message
service. It would have a plugin specification in XML like this:
<?xml version="1.0"?>
<?gcc version="4.3.1"?>
<plugin id="simple-message">
<extension-point id="simple-message.print">
<extend symbol="simpleMessage_extend"/>
</extension-point>
<library path="path-to-shared-library"/>
</plugin>
That says it's a plugin for GCC, it has id "simple-message", it
uses a certain shared library. It also says it has an extension point
called "simple-message.print" and gives the function to call when
anyone extends that extension point.
This function is called "simpleMessage_extend" and is in the
shared library the plugin specified. It looks like this:
bool simpleMessage_extend( ExtensionPoint* self, Plugin*
extendingPlugin, xmlNodePtr specfication ) {
printf( "%s\n", xmlNodeGetContent( specification ));
return TRUE;
}
It simply prints the text content of any plugin that extends it.
Another plugin might come along later and have this as its
specification:
<?xml version="1.0"?>
<?gcc version="4.3.1"?>
<plugin id="hello-world">
<extension point="simple-message.print">
Hello,World!
</extension-point>
</plugin>
Hopefully that little guy should be clear, it prints "Hello, World!"
Now the plugin system has taken care of all the dependency
management, only required plugins are loaded, etc. All the
appropriate extension points are created (only those used) and
extensions are applied.
There's also a plugin lifecycle allowing things to happen at
appropriate times.
We could have had our plugins exchange code, remember things until
later, do anything in fact. If you can describe it in XML then the
plugin system lets you do it.
*Ease of Use with Events and JoinPoints*
Such simple extension points provide all the power you ever need,
but not the ease of use. So, the system also lets you do common
things with almost no code in GCC, just a slight refactoring and tiny
description in XML.
The most common things people want from a plugin system is to be
able to listen to events or to replace behaviours. I'll show you how
events are added here.
Suppose GCC (or another plugin) is going to raise an event called
"myEvent". The event will take an int and a string. Here's what
you'd like to write in GCC when you want to fire the event:
void fireMyEvent( int count, char* name ) { /*stuff we don't care
about*/ }
// And then call it in our code
fireMyEvent( 10, "fred" );
The plugin system lets you do pretty much that. It does it by
calling fireMyEvent through a function pointer. The default function
will do nothing, but if any other plugin is listening to the event
then the plugin system will replace the function pointer with a
function that notifies all the listeners. Here's what you write in GCC:
static void fireMyEvent_default( int count, char* name ) {}
void ( *fireMyEvent )( int count, char* name ) = fireMyEvent_default;
// And then call it in our code
fireMyEvent( 10, "fred" );
Now a bit of plugin XML tells the system about the event so that
other plugins can see it:
...
<event id="my-event" signature="void f(int,char*)">
<call symbol="fireMyEvent"/>
</event>
Here we've called the event "my-event" and told the system what
signature it has (so that we can dynamically build the firing function
with libffi). We also told the system what function pointer to
replace if any other plugins extend it.
Another plugin might come along and have this in its XML:
...
<extension point="my-event">
<callback symbol="handleMyEvent"/>
</extension>
Then whenever GCC calls fireMyEvent, the other plugin's function,
handleMyEvent, will be called with the arguments given.
The event system is very powerful, the example above has only
scratched the surface. You can, for example, have one event handler
which can be used for events with entirely different signatures, you
can also use object orientation. You can also have common event
handlers and pass additional information to the event handler through
the XML (which lets you, for example, have a single logging plugin
allowing people to print info about an event with their own markup
without writing any C code).
This setup requires the absolute minimum of changes to GCC. It is
very efficient if no one listens to the event. There is an overhead
if there are listeners due to dynamically built functions with FFI.
If that FFI overhead is too much, the system also provides much less
powerful but more efficient mechanisms.
We also need to be able to replace behaviours in GCC. For
example, we might want to replace the default loop unrolling
heuristic. Libplugin makes that just as easy as events, giving
JoinPoints.
A JoinPoint replaces a function pointer, just like events do, but
the function can return values rather than just void. We maintain a
stack of 'advice' which are replacement behaviours. Only the top
advice gets called but it can make use of advice lower in the stack
(for example, it might use the original implementation of the
heuristic, it can change the arguments there). This is a bit like
join points from aspect orientated programming.
A JoinPoint also fires an event before the call letting other
plugins know the arguments to the function and one after letting them
know both the arguments and the return value.
JoinPoints also have the same powerful capabilities as events
through FFI. They also have efficient, but not powerful alternatives.
For both of these mechanisms the GCC writer doesn't need to know
anything about plugins. All they need to know is: "call anything you
want to be extended through a function pointer". Other people can add
the XML to make those things extensible. GCC writers should have to
concentrate only on making the right refactorings. In fact, the GCC
writer doesn't even need to know whether the efficient or the powerful
versions will be used.
The system also provides easy mechanisms for having extensibe
lists, imports/exports and a number of other things.
*
Existing Plugins
* The system comes with a few example plugins.
* Message (logging). Extensible markup language for logging
events, etc. Could easily replace all dump file stuff.
* Command line options. These can be changed by a quick snippet
of XML. Very useful for iterative compilation.
* Printing internal structures to XML. GIMPLE, RTL, DDGs.
* Pass manager. Find out what passes are actually run or even
change which are run on each function - without writing any code
(or with code for extra power). Or add a whole new pass. You
can also run your own code before, after or instead of any pass.
* Control loop unrolling. Either with code - providing a new
heuristic function - or with XML, giving explicit unroll
factors/types for loops.
* Add performance counter code to x86 programs. Automatically
record cycles spent in any function, results finally dumped to
XML. No need to link to any library (except libc) or even to
change your make files. Which functions to instrument
controlled by the XML specification.
* A few other bits and pieces for iterative compilation stuff. I
also have a scripting plugin prototype letting you write python
in your plugin, but it's a long way from seeing the light of day.
*
* I'm sorry all that was a bit long winded. I'm also sorry not to
have finished the documentation for the system, it's a bit difficult
to evaluate without it. I might get it finished by mid October, but I
guess that will be too late?
Cheers,
Hugh.
Grigori Fursin wrote:
Dear all,
I noticed a long discussion about plugins for GCC.
It seems that it's currently moving toward important legal issues,
however, I wanted to backtrack and just mention that we
at INRIA and in the MILEPOST project are clearly interested
in having a proper plugin system in the mainline of GCC which
will simplify our work on automatically tuning optimization heuristics
(cost models, etc) or easily adding new transformations
and modules for program analysis.
We currently have a simple plugin system within Interactive Compilation
Interface (http://gcc-ici.sourceforge.net) that can load external DLL plugin
(transparently through the environment variables to avoid changing project
Makefiles or through command line), substitutes the original Pass Manager
to be able to call any passes (new analysis passes for example) in any legal order
and has an event mechanism to rise events in any place in GCC and
pass data to the plugin (such as information about cost model
dependencies) or return parameters (such as decisions about transformations
for example).
Since it's relatively simple, we are currently able to port it to the new versions
of GCC without problems, however, naturally, we would like to have
this functionality within the GCC mainline with the defined API (that
what I discussed with Taras from Mozilla during GCC Summit this year).
I believe it may help making GCC a modular compiler and simplify future
designs (and can be in line with the idea to use C++ for GCC development
if Ian moves this project forward). By the way, Hugh Leather
also developed an interesting plugin system for GCC that allows
to substitute internal GCC functions with the external ones within
plugins to enable hooks inside GCC (he mentioned that he plans to release it soon)...
Furthermore, we will then be able to use our current MILEPOST tools
and Collective Compilation Framework to automatically tune default GCC
optimization heuristic for performance, code size or power consumption
(instead of using -O1,2,3 levels) for a particular architecture before
new versions of GCC are actually released (or for additional testing
of a compiler using different combinations and orders of passes).
And when the compiler is released, users can further tune their particular
programs interactively or automatically through the external tools and GCC plugins.
By the way, we are extending current ICI for GCC 4.4 to add cost-model tuning
for major optimizations (GRAPHITE, vectorization, inlining, scheduling,
register allocation, unrolling, etc) and provide function cloning with
different optimizations, and naturally would like to make it compatible
with the potential future common GCC plugin system, so I hope we will
be able to agree on a common plugin design soon and move it forward ;) ...
Regards,
Grigori
=============================
Grigori Fursin, INRIA, France
http://fursin.net/research
-----Original Message-----
From: Taras Glek [mailto:[EMAIL PROTECTED]
Sent: Tuesday, September 16, 2008 11:43 PM
To: Diego Novillo
Cc: Basile STARYNKEVITCH; gcc@gcc.gnu.org; Sean Callanan; Albert Cohen;
[EMAIL PROTECTED]
Subject: Re: Defining a common plugin machinery
Basile STARYNKEVITCH wrote:
Hello Diego and all,
Diego Novillo wrote:
After the FSF gives final approval on the plugin feature, we will need
to coordinate towards one common plugin interface for GCC. I don't
think we should be adding two different and incompatible plugin
harnesses.
What exactly did happen on the FSF side after the last GCC summit? I
heard nothing more detailed than the Steeering Committee Q&A BOFS and
the early draft of some legal licence involving plugins. What happened
on the Steering Commitee or legal side since august 2008? Is there any
annoucement regarding FSF approving plugins?
I am CCing everyone who I know is working on plugin features.
Apologies if I missed anyone.
I would like to start by understanding what the plugin API should
have. What features do we want to support? What should we export?
What should be the user interface to incorporate plugin code? At
what point in the compilation stage should it kick in?
Once we define a common API and then we can take the implementation
from the existing branches. Perhaps create a common branch? I would
also like to understand what the status and features of the
different branches is.
The MELT plugin machinery is quite specific in its details, and I
don't believe it can be used -in its current form- for other plugins.
It really expects the plugin to be a MELT one.
>From what I remember of the plugin BOFS (but I may be wrong), an easy
consensus seems to be that plugins should be loadable thru the command
line (probably a -fplugin=foo meaning that some foo.so should be
dlopen-ed), that they could take a single string as an argument (using
-fplugin-arg=bar) and that plugins add essentially new passes into
pass.c - I am happy with such a picture (and could fit MELT easily
into it; in its current form MELT is adding a few basilys*passes into
passes.c, each having a gate which try using a MELT closure somewhere
so is equivalent of testing if the pass is enabled.).
Albert Cohen & Grigori Fursin (both in CC) from INRIA have an
"Interative Compiler Interface" (a big patch to GCC) which could be
helpful to enhance this idea of extending the pass manager. AFAIK,
they also have some plugin facility (or maybe I am wrong). Perhaps
some ideas or code could be reused for plugins.
I have no idea if some general plugin code evolved recently, i.e. if
someone is actively working on it.
If someone is coding on plugins, please tell the gcc@ list.
Hi Guys,
We are actively using and maintaining our plugin API at Mozilla. We
moderated the plugin discussion at the summit. From our discussion
sounds like we need to be able to hook into the middle-end(pass manager)
and the frontend. As far as I know nobody has needed anything of the
backend yet.
We are using the -fplugin syntax that Basile described above to dlopen
our plugins. As Basile mentioned, the pass manager API is extremely
important. Plugins need to be able to list passes(thus all passes need
unique names) and to insert a new pass at arbitrary positions. If I
recall correctly, Grigori's work also needs to be able to reorder
existing passes.
Beyond that, there need to be general callbacks sprinkled throughout the
code. At the plugin BoF we decided that something like
plugin_callback(EVENT_NAME, void *data) would suit us well. Here
EVENT_NAME would be an enum describing the callback. Examples:
* PLUGIN_FINISH_STRUCT(data would point at tree_node for the type
that gcc just finished building)
* PLUGIN_FINISH_GCC(before gcc exits)
* PLUGIN_CXX_CP_PRE_GENERICIZE (data is a function body tree_node)
for being able to look at fe gimple.
* PLUGIN_PLUGIN_SETUP for when it's time to register with the pass
manager
* other stuff such being notified before particular passes execute(I
think Grigori needs this).
For my treehydra plugin it is important to land the GTY patch that Diego
reviewed. I can start submitting patches for review whenever someone is
ready to review them. Our plugin code isn't yet structured exactly as
described but I will happily restructure the API to something acceptable
and submit stuff for review.
See http://developer.mozilla.org/en/Dehydra for more details on
Mozilla's plugin work.
Cheers,
Taras
