Here's the rest of my transcription of Larry's talk. He hasn't had
time to proofread it, so I'm posting it now in the interest of getting
it out. Ask, please add it to your page at dev.perl.org. When you
do, please send mail to pudge and mjd so they can announce the
completed transcription on perl.{com,org}.
Thanks,
Nat
As I look around here, I see a lot of young faces. Almost too young.
I like youth. I always planned to stay young all my life. I've always
been blessed with a face that's younger than my age, so I've tried to
always adopt a young attitude to go with my face.
You can fool some of the people all of the time, but you can't fool
Mom. I think she knows that I'm just a late bloomer (speaking of
procrastination). I'm late for a lot of things, not just blooming.
For instance, I only started writing this talk yesterday. Another one
of these Just In Time talks. I'll probably finish it sometime
tomorrow. Lateness is one of my specialties. Someday I'll be dead,
and when you refer to me as "the Late Larry Wall", people will think
you were joking.
Speaking of my long-lost youth, I remember sometime way back during
the Kennedy administraion, my grade school teacher asked us all to
calculate how old we'd be in the year 2000. I got the unfathomable
answer of 46. I could imagine being 10, maybe. But 46 is a lot closer
to infinity!
Anyway, I couldn't imagine being 46. Now I can. As of last month, I'm
officially in my late 40s. Whoopee! Does this mean I can schedule my
mid-life crisis soon?
Besides my birthday, I recently noticed several things that tell me
I'm not so young any more. The first thing I noticed is that I took
my first child, Heidi, off to college. Believe me, even if you don't
notice your kids while you still have them, you'll definitely notice
them when they go away.
The second thing I noticed is ... I forget what.
The third thing I noticed is that I want to start reminiscing about my
youth. Old people really only have three things to say: how good it
was back then, how bad it is now, and (if you're lucky) how much worse
it will be in the future.
So, I'd like to talk about how good it was back then, how bad it is
now, and how much worse it will be in the future.
Actually, I just said that to be funny. I really do think things are
getting better, and the getting betterness is what I would like to
talk about.
The name of this talk (for the moment, I might change it) is: Camel Lot
#6, The Once and Future Perl.
I going to do a little bit like Ken did last night, and take you
through how bad it used to be and how good it's getting, and on to the
future. A chronological outline is a great way to go. Some of you
may recognise some of these slides. In fact, these are probably about
10 years old, some of them. I put them in more for historical value.
If you look closely, you can see gritchineses--that's an overhead
transparency that I scanned. So this is the real original.
[How It Was Back Then (Sort Of)]
Way back in the dawn of time, this is what Unix looked like. Either you
programmed in C, or you programmed in shell. Of course, those were
two different realms, and it was sort of one or the other. We had
this dichotomy of our thinking, that you had to do one or the other.
[How It Really Was Back Then]
How it really was back then--one day it opened up to me, these were
not opposites, but think of an oyster shell opening up here. Instead
of opposites, they were two different dimensions in space that you
could graph things like this. You could consider sed and awk to be
sort of warts on the shell.
For those of you at the back who can't read this, the left hand axis
(what C is good at) is called "manipulexity" and the horizontal
attitude is called "whipuptitude".
With that realization I realized that there was a big gray area out
there. In biological terms, that's an ecological niche. So I decided
to fill it.
[The Hatching of an Idea]
Perl got hatched, and that's kinda why it's egg-shaped there. From
the beginning it was designed to grow into that space. Perl was
always designed to be an evolving language.
The reason I'm telling you some of this stuff is that, as some of you
know, we've announced Perl 6, and I'm going to be talking about that
later. And one of the very very high level goals of Perl 6 is to keep
Perl capable of evolving. Perl 5 was running into some limits and
we're going to figure out how to get around those limits.
So it appears that there are these two axes of C and shell. And there
could be other dimensions: the Python dimension, the Java dimenson.
But you don't want to think of either/or. It's really easy for people
(especially computer scientists) to get locked into this either-or
thing. Either I use the GPL or I don't. Well, why not use the GPL
*and* some other license? Is that right, Ken? [Ken Coar had been
lamenting the difficulties of the GPL the night before, but didn't
mention dual licensing.] It wasn't long after that that I invented
dual licensing, and I think that actually works pretty well.
So there are lots of ways in which we try to avoid falling into the
either-or trap. Perl got its philosophical inputs from four different
areas, not just from computer science. (Obviously).
[The Inputs]
>From this diagram, you can see that computer scientists tend to not
like things to be Art. They prefer things to be scientific. So
Computer Science is sort of the opposite of Art, maybe, but I don't
know that Linguistics is the opposite of Common Sense. Maybe it
is--I've been trying to learn Japanese lately .... People say that
Perl is hard to learn. Nu-uh! *Japanese* is hard to learn!
[What to Take, What to Leave Behind]
More specifically, here are some points that Perl took from those four
languages I listed earlier. The bold things are things that I
borrowed, and the non-bold things are things that I specifically left
behind. People often think that Perl is hodge-podge, everything
including the kitchen sink, but I actually left out a few of the
kitchen sinks along the way. Still, a few have snuck in and that's
what I'll talk about later.
The question is, when you combine things like this, is it good or is
it bad?
[Question: bastardization or hybrid vigor?]
Well, that's a personal judgement. I have my opinion on it.
[My Irrationalities]
I have a lot of opinions, actually, and some of them you'll
notice in the design of Perl. I don't like dangling syntax, and I
think that ordinary people like me hate abstractions. Hmm, that sorta
assumes that I'm ordinary. I do think of myself as ordinary.
Ordinary-weird, but ordinary.
And I believe contradictory things. I believe that C is wonderful and
I believe C is awful. And that awk is neither.
I think that language is an amoral medium, by which I mean that
language itself is like an artistic medium, like paint. You don't
have pretty paints and ugly paints, it's after you arrange them on a
canvas that they get pretty or ugly. Similarly, language is something
you can use. You can write beautiful poems, or ugly flames. Or
beautiful flames...
I think ugly can be beautiful, and that beautiful can get ugly real
quick. I think in particular of Lisp, which is the most beautiful
language in the world, and every program in Lisp is real ugly.
I think visual metaphor is for more than just poetry, so I think things
that are different oughta look different.
The next one applies to language designers: you have to simultaneously not
care what other people think, and care what other people think.
And I think that God has free will, which is why he designed us to be
creative. Which is why programmers prefer to have more than one way to
do it--they have choices, and that's what makes programming fun.
[Common Fallacies of Language Design]
Now lots of people design computer languages, and fall into all sorts
of traps:
"Now, we have to start over from scratch". That's almost any academic
language, which throws out all the existing art and says "we're going
to make people think entirely different from how they did before".
"If we put in English phrases, that makes it readable". That's
COBOL.
"Simple languages produce simple solutions". There are lots of simple
languages, but simple languages produce complicated solutions because
you have to be verbose.
"If I wanted it fast, I'd write it in C". That was the original
reasoning behind making awk run so slow.
"I thought of a way to do it so it must be right". That's a very
common trap: people just implement the first thing they think of.
"This is a very high level language--who cares about bits?" Again,
like awk.
"You can do anything you want with NAND gates". That's Turing
machines, I guess, or something like that.
"Users care about elegance." No, they care about getting their job
done. For some reason lots of languages that come out of Europe have
this thing about elegance that I don't quite get. Obviously.
"The specification is good enough". That's Ada.
"Abstraction = usability". That's Lisp or Scheme, or maybe ML.
"The core should be as small as possible." That's Forth.
"Let's make this easy for the computer." That's assembly language.
"Most programs are designed top down." That's Pascal.
"Text processing doesn't matter." Fortran.
"People should never have been given free will." Lots of languages.
[Larry's Conjecture]
I think that what actually makes a language useful is not the theory
involved in it. Theory is good, but if you drive a particular theory
into the ground, that just drives people away.
So a language's practicality is inversely proportional to the number
of theoretical axes the language has to grind. I don't mind theory, I
just don't like theoretical axes.
[Waterfall or Whirlpool?]
Now there's an interesting chart that I go back to look at again and
again, to remind me that programming is not a sequential activity.
We already saw the fallacy of either/or thinking. Another fallacy we
often fall into is sequential thinking.
Some of you may have heard of the waterfall chart, which talks about
doing specification, then you go to the design, then coding, then
testing. But really it's more of a chaotic strange attractor, where
all of these things are going on simultaneously. You know, "Ready!
Fire! Aim!" is how we program. There are some methodologies that
advocate testing before doing coding and then doing the design, and I
think that's one of the eXtreme Programming tenets.
Right now as I'm giving this talk, Camelot #6, it's about perl 6, I
think that Perls 1 through 5 were prototypes for Perl 6. And you plan
to throw away the prototypes ...
[The UNIX Family Tree]
Now Unix has an interesting history (audience laughter). I think this
down here is the little dance being done by BSD and Linux.
[The C Family Tree]
The C family tree kinda looks like this (more laughter). We had K&R
C to begin with, then we had gazillion different types of C, then we
had ANSI C, and then ... something happened. (laughter)
[The C Family Tree, Cont.]
And then something else happened. (more laughter)
[How Microsoft Sees It]
And then something else happened, and this
is the Microsoft view of reality (yet more laughter).
[The Perl Family Tree]
Here's the Perl family tree. Actually, this is slightly disingenuous,
because there was a little branch off to the side when ActiveState
first ported Perl to the Windows platform, but we made a big effort
and pulled it back together. So the line got a little wider there and
got narrower again. The nice thing about open source is that you can
pull things back together again if you really need to.
The slide has said "under construction" down there from the beginning,
and it still says that.
[The Other Perl Family Tree]
Here's the other way of looking at the Perl family tree. Needless to
say this is notional.
[Natural Language Concepts]
Now Perl's design, from the beginning, since my background is both in
linguistics and computer science, has a lot of linguistics
notions. Not on the level of COBOL, where it just looks like English,
but on a deeper level.
A natural language you learn it once, and you use it many times. So
you should optimize for expressiveness, not for learnability. I think
somebody had told that to the Japanese before they designed their
language, and I wish their language was a little easier to learn.
Another thing about language is that you learn as you go, and many
acceptable levels of competence are ... acceptable. That means we
don't expect a 5 year old to speak with the same level of competence
as a 50 year old. Or a 46-year old. And it's perfectly okay for a kid
to use kid-talk and a baby to use baby-talk. Contrast that with your
typical C++ shop where if you don't know the whole language and
program in a subset of C++, you get laughed out of the office.
To me it should be acceptable to program in Perl subsets. If you want
to program in the subset that looks like awk, sed, or C, that's fine.
Or the subset that looks like Python ... (slow laughter) Some of you
are awake.
There are more points here that I'm not going to go through. The
explanations are on my website if you're interested. There are many
more things that are based on how we use natural language, and I don't
want to lose that as we go forward and evolve Perl.
[Perl 5: The Big Nouns]
So for more recent history, meaning 5 years ago plus or minus, when
Perl 5 first came out this was what we announced as the goals of Perl
5. Perl 5 did improve in all of these areas. When I look back at it
now, in all of these areas it can continue to improve. So most of the
rest of this talk is going to be about that.
[Constraints on Perl 5]
Now Perl 5 had some constraints on its design. First of all was the
need for backward compatibility. We have deprecated a few things along
the way, but most of them were what I call 'eternally deprecated':
"Don't do that, but it still works". So most of your Perl 1 scripts
(I'm sure you've all written Perl 1 scripts right? Some of you
weren't born then!) still work in Perl 5 for the most part. You might
get some warnings, but we had to keep the backward compatibility at
that point.
Of course another constraint on the design of Perl 5 was my own
idiocy. That was the last time I designed the language by myself, so
I was limited by my own imagination. And the cultural model of Perl 5
worked for Perl 5, but it had bottle necks in it, in terms of how we
developed it. Having one person in charge of all the integration
tends to be a bottleneck. Yes, you know that.
Of course, there was ignorance of the future. And there were some
places where I was ignorant of the present and the past, but we'll let
that go.
And a lack of round tuits.
And another way in which Perl 5 was constrained was that in dealing
with all the other things there, the core was quite complex. You like
it when Perl rips through a file really quickly, doing pattern
matching or whatever it's doing, but a lot of that is because of the
internals doing optimizations and the like. So it's a little hard to
get in there and actually change it, particularly if you're starting
out. There aren't that many people that can do it, and that limits
how much Perl 5 can evolve.
And then of course there's the issue of complacency. You can easily
sit there and say "well, it's good enough". That was sort of starting
to happen.
[The Summer of our Discontent]
Between version 5.005 and 5.6, it took about two years.
There were many reasons for that, some of them I mentioned on the last
slide. A personal reason is that I was busy writing the new Camel
book, Camel 3 as we call it. I was busy with the Perl Whirl cruise
(you know, there's a Linux Lunacy cruise coming up. I highly
recommend it. It's cheaper than you think, because the cruise lines
are overbuilt so they're trying to get people to go on all these
cruises. I was expecting the Perl cruise to be all these old fogeys
pretending to be Perl programmers, but it was really much of the same
demographics as you. But I spent a good part of it proofing the Camel
book in my cabin. You look up every five minutes and "there's a
beautiful view, oh nice", then go back to proofing).
After that I was busy getting ready for Yet Another Perl Conference
19100 (that's what they called it), and after that getting ready for
the Perl Conference 4.0.
As much as a year before the Perl conference, we'd said to ourselves
"you know, we should have ourselves a constitutional convention and
figure out how to reorganize the community to be more effective". And
yeah, that's a great idea, but finally this last summer Chip
Salzenberg said, "Let's have one of those meetings."
We started having one, and an interesting thing happened. We spent
the first hour gabbing about all sorts of political and organizational
issues of a fairly boring and mundane nature. Partway through, Jon
Orwant comes in, and stands there for a few minutes listening, and
then he very calmly walks over to the coffee service table in the
corner, and there were about 20 of us in the room, and he picks up a
coffee mug and throws it against the other wall and he keeps throwing
coffee mugs against the other wall, and he says "we are fucked unless
we can come up with something that will excite the community, because
everyone's getting bored and going off and doing other things".
And he was right. His motivation was, perhaps, to make bigger Perl
conferences, or he likes Perl doing well, or something like that. But
in actual fact he was right, so that sort of galvanized the meeting.
He said "I don't care what you do, but you gotta do something big."
And then he went away.
Don't misunderstand me. This was the most perfectly planned tantrum
you have ever seen. If any of you know Jon, he likes control. This
was a perfectly controlled tantrum. It was amazing to see. I was
thinking, "Should I get up and throw mugs too?"
Anyway, so we started talking after that and the idea popped up that
maybe we oughta rewrite Perl. The idea that occurred to me then was
that we had some new technology: Perl has a decompiler now. You can
take a Perl 5 script and compile it down to the bytecode, and take
that and compile it back to Perl 5 code. And if you can do that, why
not compile it back to Perl 6 code?
So with that realization, we are now free to actually do a major
redesign. What can we do that we would have done differently if we
had known better, 14 years ago?
So later that day we also had an open meeting of the perl porters,
with about 50 people there, and we started handing out tasks about how
we would do the redesign. And the next day in my keynote I announced
that we were rewriting Perl. I only got about three hours of sleep that
night because I was rewriting my speech. That would have been okay
except that I'd only had three hours the night before. So if you
listen to that audio on the net, you'll hear that I sound very spaced
out. Er, more than usual.
So anyway, Perl 6. As I was flying here on Delta Airlines, the
monitor was showing all sorts of happy sayings.
["If you can dream it, you can do it"--Walt Disney]
"If you can dream it, you can do it"--Walt Disney. Now this is
actually false (massive laughter). I think Walt was confused between
necessary and sufficient conditions. If you *don't* dream it, you can't
do it; that is certainly accurate.
[Reinventing Everything]
So we've been dreaming. Everything's fair game. We've been thinking
how to reinvent the lexer, the parser, the interpreter, the code
generators, the internal APIs, the external APIs, the language itself,
and the culture surrounding it. And interestingly, the first thing we
started reinventing was the culture.
[New Working Group Model]
Here are some mailing lists that have sprung up since August 1, and a
rough idea of the number of messages that have been posted to each of
those. These correspond to various working groups. We have a working
group model where there's an official person in charge of each working
group, and there's a charter, and there's a sunset law where at a
certain date we will shut ourselves down unless we can justify
continuing our existence. So we're doing something very different
from the perl5-porters mailing list that has gone on forever, and
isn't much about porting any more.
So there's much more sense of responsibility, and people are holding
themselves accountable. And the amount of raw flamage has been less
than in the old days. So, so far so good on the culture, and we'll
continue evolving the culture.
As you can see, some of these working groups look very general.
perl6-language obviously has a lot of activity and will probably go
on for a long time. Some of these were working groups set up to
discuss one little issue, one RFC, and sometimes the timing was a
little off. If you look at the perl6-language-unlink one, they
created the mailing list just as the discussion was dying, so it never
got anything on it. But this tells you of some of the ways we're
breaking down the culture here.
In particular, there's an official way of announcing RFCs. We're
splitting the language issues from the internals issues. Build issues
are being covered. QA. Notice that that's perl-qa -- we're hoping
that we can back-apply some of the QA things we develop in Perl 6 back
to Perl 5.
Now one thing I want to make clear about this Perl 6 activity is that
we're not abandoning Perl 5 any time soon. We don't know how long this
Perl 6 will take. There are still some people using Perl 4 out there.
[The Constraints on Perl 6]
Now Perl 6 has a different set of constraints. Hopefully these are
more liberal constraints than Perl 5 had. It's still the case that
Larry needs to understand Perl 6. For some strange reason, even though
they split off into all these working groups, and they have a language
working group, they appointed me Chief Language Designer. So I said
"okay".
So I need to make sure that I grok everything that's out there. That
doesn't mean that I had to come up with it in the first place. That
doesn't mean that I have to understand how to do it, only how it's
going to interact with other things.
Other constraints that we were putting on the design: you can propose
a new feature, but if it's not something that people really need,
we're not going to put it in. The underlying philosophy of Perl is
to serve, and if it doesn't serve our needs then what good is it?
Can we implement it efficiently and robustly?
And a biggie here--does it preclude translation from Perl 5? We need
to be able to have a migration path from Perl 5 to Perl 6, so if you
distort the semantics too violently then it becomes too difficult to
translate. So Perl 6 will bear a close resemblance in some ways to
Perl 5.
Does the utility of the change grow faster than the complexity? And
does the size of the executable grow slower than Moore's law? These
things always grow over time, but is the computer getting faster faster?
And we have to take the time to maintain Perl 5, and we also have to
make sure that we take the time to design Perl 6 right. There's never
time to do it right, but there's always time to do it over--but we
want to do it over first. Or something like that...
[The RFC process]
The RFC process. We call them RFCs, and they really are Requests For
Comments, they're not standards! There are over 360 standards,
they're written in Plain Old Document format, and they must be
well-formed to be acceptable to be announced and posted on
dev.perl.org. They're subject to revision, retraction, rethink.
The first one came out right after the Perl conference, and we had it
open for brainstorming essentially until the end of last month. And
that's when I got a snapshot of it. The theory was that I would read
all these RFCs in two weeks and come up with a language design by today.
Riiiight. I've read the abstracts of all of them but the last 50. If
I run out of things to say here, I'll read the last 50 and take a
master class approach here.
[Handwaving Requirements]
These are things that will be in Perl 6, for some definition or other,
but they are somewhat nebulous. Perl needs to support higher level
programming, and I'll go into some of these as we go along, and it
must support lower level programming. It'd be nice if it supported
more paradigms, yes the p-word. And we really want to avoid the
equivalent of DLL hell. It's getting harder and harder with all these
modules out on CPAN, with different versions of them, to keep them all
straight. There are ways of handling these sorts of things.
"Perl must support absolute flexibility". There's actually an RFC
that says "Perl 6 will be the last version of Perl". By which they
mean, "we'll do it good enough that there'll be no need for a Perl 7."
In fact, they say that the version number will asymptotically approach
pi*2. That's what the RFC says, I haven't said whether it's right or
not.
It needs to have a mean and lean core, and we'll talk about that.
Perl must not surprise people unnecessarily, and there are still ways
in which Perl surprises people and we can reduce some of that. It
needs to be as simple as possible, but no simpler. It needs to map
well onto your problem. The better your solution space maps onto your
problem space, the more compact your solution will be. And Perl must
remain Perl. There's an RFC that says that too.
[Simplifications]
There's a lot of things we can simplify, going from Perl 5 into Perl 6.
If the core is going to be lean and mean, that implies that a lot of
stuff are going to move out of the core. Non-critical built-ins,
sockets, operating system stuff, and so on, will be moved out to
modules. Now they might be autoloaded, so they look like they're in
the core, but there's no reason they have to be there to begin with.
So we can discount those in the internals design, provided of course
that calling out to those modules is efficient. So that's another
goal.
All variables will be usable as objects. There are ways in which you
would like to be able to treat arrays and hashes and scalars as
objects, so for instance (and this comes back to the category of not
surprising people) people expect length() to work on arrays. That's
because they're thinking of the array as an object, not as a list in
scalar context. So probably length() is going to become a method on
the array object. So the translator for Perl 5 will actually have to
translate the mistake of saying length(@array)!
In any event, all the variables are actually objects internally. And
rather than having this funny conditional code like Perl 5 does,
everything will be run off vtables as C++ does. That does not
necessarily imply thtat we are writing it in C++, probably not, but
that's all implementation. That will be very efficient and it will
allow alternate implemenations to be swapped in and out very quickly,
loaded as modules.
Now many of the strange global variables that you know and love dearly
will be lexically scoped. That will simplify things like
multithreading. Filehandles become objects using object-oriented
notation, meaning we've decided that they'll be regular scalar
variables instead of bare filehandles as they are now. This
simplifies a whole bunch of things, one of those things is the parsing
rules for indirect objects. We can probably simplify the parsing
rules in other places as well, disambiguation rules, and might even go
pretty wild there in terms of how you specify a slice--it might be
entirely different. As long as we can translate it!
We plan to unconfuse homonyms, like the different types of do() and the
different types of eval(). For example eval BLOCK will probably
become try BLOCK as that's how other languages are going these days,
and that's what eval BLOCK means. So we may disambiguate that way.
Typeglobs we would very much like to go away. XS we would dearly
*love* to go away. And I think we can do that, if we can make Perl a
low-enough level language to have you specify the interfaces.
I had this realization a few years ago when I wrote this thing called
JPL, Java-Perl Lingo. You link Java and Perl interpreters into the
same process and call back and forward between them. I realized that
Java had sufficient information to write all the glue to call back and
forth between Java and Perl, and there was no need to have a glue
language. I thought "that's cool, I wish Perl could do that".
So part of the design of Perl will be that you can specify enough type
information in your function and method signatures to be able to do
that. To be able to produce that glue automatically.
We're going to be standardizing the way that objects are set up. Not
that we're going to be undoing all the flexibility that you have right
now, but you'll have some standard ways that will be easier and more
efficient.
Perl 5 by design made it easy to use objects, but not easy to use
object classes. We'd like to make it easy to write object classes.
It's becoming obvious that we need a real garbage collector, either
one of our own or we need to be able to relate to one that is
supplied. For example if we compile Perl down to the C# virtual
machine, it provides its own garbage collector.
Exception handling becomes completely object-oriented. You can still
pretend like it's string-oriented. If you die with a string, you can
still treat it like an object. This gives you many benefits.
You'll have easier ways to interpolate expressions in strings. We
might even think, and I haven't thought about this a lot as you can
tell by the question mark on the end, about simplifying the precedence
levels. And the translator can handle that, if it knows about the
precedence levels in the different languages. So we might be able to
do that.
[How much complexity can we allow in exchange?]
So the question is, with all this simplification going on, how much
can we get back out? Well, I don't know. We can handle a certain
amount of complexity so long as we can hide it from people. As long
as you're not exposed to the part of the complexity that you're not
interested in at the moment, that's a virtue. It goes back to
programming in a subset of Perl: the standard Perl has a certain
level of complexity, and if you put a declaration at the top then you
get the full level of complexity, but you don't *have* to worry about
it. So there are things you can do with the complexity.
[A Tour of the RFCs]
So now let's look at the RFCs. As I said, there are lots of them.
I've sorted out about the first 20 of these, and I've read all but
about the last 50 of them. There are some very interesting things
here, and I'm sad I don't have time to read them all to you.
To let you know the way I'm going through these is. Well, let me just
show you one as a for instance here. Here's one that probably won't
be done. [I don't remember which one was on my screen at that point.]
Let's talk about my classification scheme. There are the first 361,
except for the first 20 which have already disappeared into
subdirectories.
Now you can't read it, but that one says "good", that one says "bad",
and that one says "ugly". So I really am sorting them into the good,
the bad, and the ugly. There's also a culture directory, and other
things I don't want to deal with: culture, internals, retracted,
metasyntax.
If you look down into the "good" directory here, I'm not just
classifying them into good, but I'm also giving them grades. Not
because I want to flunk them, but because I want to sort them in
order. The two grades I'm giving them, the first one is "is this a
real problem that we need to deal with". And the second grade is "how
well is the solution specified". And a lot of these say, "I don't know
how to solve this". This shouldn't be taken to mean I'm castigating
them for brainstorming, but it means I can see them in order.
Yes, it would be great to have higher-resolution time values. I think
they should be floating point. Some of these I've gone through and
put notes up to myself: should we modify open() to support objects and
extensibility? Yes. I think that open() should return a filehandle
object rather than take a filehandle as a first argument.
Rename the local operator? Yeah, I think we ought to do that. It
confuses people when we call it local(). The problem is, of course,
that this is not a perfect solution--they haven't come up with the
right name here: savetmp, tmpsave, scratchpad, etc. The consideration
for me is how it's used grammatically. It'll often be used with an
our() declaration. It depends on whether it should come before or
after the our() keyword. If it came after, you'd say "our temporary
$foo = 2", but if it came before, you'd say "temporarily our $foo = 2".
Yes, we'll reorganize all these funny variables. Rather than put them
into other globals, we'll make it object-oriented as far as possible.
Here's one that is rated "BF": pretty good problem, rather lousy
solution. There should be a way to tailor error messages to different
languages. We'll have to do that for localization anyway, so fine.
Here are some from the "bad" directory. (reads from one) ... and they
want us to solve the halting problem. No.
Highlander variable types. This is an interesting one. The idea is
that an array is really just a funny name for a reference in the
corresopnding scalar. It makes for some interesting ramifications,
but I think that it would probably be more confusing in the long run.
Multiline comments in Perl. I'm sorry, I was brainwashed by the Ada
rationale. Ada has comments to the end of the line and I think that
if your editor can't put a row of sharps down the left, then you need
a better editor.
And so on.
Ugly things are ones that sort-of are good ideas but maybe we need to
think about them. Yeah, we need to talk a lot about threads. Immediate
subroutines, yes. We need ways of having subroutines that are
evaluated at compile time, but this mechanism probably isn't it. Lexical
variables made default. Uh, no.
Keep default Perl free of constraints such as warnings and strict.
Yeah, I can argue this one both ways too. Overloadable logic
operators. This has ramifications on a semantic level because at
least currently the Perl internals don't distinguish between these
funny C-like && and || and the higher precedence if(), unless() and so
on. These compile down to the same thing, and it would be strange to
make some of them behave one way and others the other. We might allow
it if you declare that you want it this way.
Anyway, that's kinda how I've been going through it. They're great
fun, you should read through them if you want to be entertained.
They're all over the map. Some of them contradict others. There's no
way we're going to implement them all, because they are
self-contradictory.
In fact, some people have written their own contradictory RFCs. "We'll
do it this way!" "No, we'll do it this way!" [Bradley Kuhn laughs on
the front row.] Yes, I was going to point out Brad here. Let's see:
"Artistic License must be changed" fair enough. "Perl6 License must be
a bugfix of Perl5's License". "Perl6's Artistic License should be
GPL/Artistic 2.0", which is a complete change from the old Artistic
License. I haven't actually given this one too much thought yet, so I
can't tell you whether I approve or not.
A big category in my mind is "things I don't want to think about yet"
and I'm deferring this one. In fact, licensing is probably something
we can defer for a while, at least until we start publishing alphas.
[Perl as a Low-Level Language]
Perl as a low-level language. Polymorphism is your enemy if you're
trying to do low-level programming. If you want to get early
compile-time binding as soon as possible, you want the compiler to spit
out very efficient code, so you write your loop to declare i as an
int, then by golly you want your compiler to spit out very efficient C
code. "num" is a number, "str" is a string. For some reason they're
all coming out to three letters. You'll be able to get nice
declarations like this:
my num $pi :const = 3;
And similarly if you have homogenous arrays or hashes, you'd like to
be able to declare the type of all the elements and have compact
storage. And this will be hugely more efficient than the way arrrays
and hashes are currently stored. Furthermore you specify not only
what the types are and how they're stored, but give hints to the
compiler like "I'm only going to use these four bits so if you want to
pack them in tight, go ahead".
If you want to have mathematically compact arrays, multidimensional,
then we'll have some way of declaring those. There's a whole series
of RFCs of what the mathematicians want out of Perl, if we make them
declare at the top "use MathematicallyFunnyArrays" or something like
that.
Non-homogenous arrays are going to be done primarily through the
object interface. I'd like to borrow the idea of unsafe code from
C#, where you can go in and do things with low-level pointers, so long
as you're careful.
[Perl as a High-Level Language]
By way of contrast, if Perl's going to become more a high-level
language then polymorphism is your friend. You want to delay your
binding as long as possible as to what implements what. Perl has
always been in the business of allowing, but not requiring,
abstraction. We'd like to put in more support for functional
programming, for logic programming, and for what are called "little
languages".
The folks at Bell Labs invented this notion, soon after they invented
yacc. "Cool, yacc lets you make a grammar for your own language". So
all these itty bitty languages sprang up that were for their own
purpose. So each time you wrote a new program, you wrote your own
language for your program. This was cool except that you had to learn
the new language each time, and it was always different. We'd like to
explore the notion of using a big language as a little language.
If it's okay to program in a Perl subset, and you define a subset that
looks like a little language, then how do you get around paying the
price of the generality of the big language? We have ways of thinking
about that.
--new--
Perl4's approach to high-level programming, or metasyntax, was the
eval operator. That was what you had, and maybe typeglob assignment
if you were really lucky. Perl5's approach to metasyntax was much
more general. Not only does it have the eval operator, it has BEGIN
blocks which are very powerful with the whole notion that you can run
the interpreter at compile-time and do what you jolly well please.
Well, up to a point.
By that means, perl5 implemented importation in a completely general
fashion. It allows overloading and overriding of certain builtins on
the fly as you're compiling. And also there's the concept of source
filtering. If you declare a source filter at some point, you can
completely change the language from there on down. There's a filter,
"use Pythonesque". There's a filter by Damian Conway called
Lingua::Romana::Perligata, which lets you program Perl in Latin. I
kid you not.
One of the RFCs out there is another of Damian's talks where he
proposes adding quantum mechanical operators which will do quantum
superpositions. Seriously he's proposing this in case some day we get
quantum computers. Actually, it has some notational convenience
because he has any() and all() operators that work interestingly as
quantifiers. But anyway.
In perl6, not only do we have all the perl5 approaches, but we keep
getting asked "we'd like to have a site-wide configuration file that
Perl will read when it starts up". Well, no. Not if it's going to be
implicit. But if you're willing at the beginning of your site's Perl
scripts to say "use Policy" then you can have your site policy in
one file, and it can in turn turn on all the strictures that you want.
As long as you declare that, it can be a meta-declaration of all the
other things you want to use.
I intend that the lexer and parser of Perl6 be written in Perl, not in
C. Perl's low-levelness needs to be made efficient enough so that the
lexer and parser in Perl will be fast enough. The lexer should be
written with Perl regular expressions. The parser? If we can get
away with it, I'd like to do a recursive descent parser because it can
do much better error diagnostics and we can easily do the last point,
that is add new lexer or parser rules on the fly.
The problem with the source filtering was that it was all or nothing:
you could filter the text as it came through, but then if you just
wanted to tweak the Perl code that was coming through, you had to
parse the Perl code as it was coming through. This lets you use the
Perl parser for what it's good at, and then tweak it at whatever level
you want.
So any point where the lexer or parser is recognizing something, we
will have a hook that lets you redefine how you recognize numbers,
strings, quoted constructs, bracketed constructs like subscripts,
operators, functions, expressions, blocks, lists of blocks like try
... catch ... catch .. catch ... finally, subroutine definitions to
include compilation notes. That ought to give us enough flexibility
to evolve a little bit further. For a while.
Now, if you can redefine operators then why not have Unicode
operators. Some of you are squeamish about that. Well, I say "why
not?" The mechanism will be abused, so what? There are a billion
mathematicians out there who are always inventing their own notation.
When they write a program and want one or two mathematical operators,
they'd like to write their script in terms of it. Fine, let them.
They can redefine their own operators, but they're what I call a
swearing operator. If you limit yourself to ASCII, all you can do is
make yourself longer and longer operators that look weirder and
weirder.
One of the things I like about Tcl is that it's always had a model of
semantic delegation, where the first verb in the command can be kind
of agnostic as to the meaning of the rest of the sentence. It just
hands it off to the next level, and that can work out what the next
level means. This ties back into being able to define little
languages, and is another good use for having these metasyntactic hooks.
This indicates that we're not doing macro processing here, not
rewriting text. We're thinking about putting hooks at the level where
you can tweak the semantics not just the textual program.
Now hooks like this will be lexically scoped, so there'll be no action
at a distance. If I declare "use this funny-looking Perl code from
here down in the file" it just changes your file, not someone else's
file. If you want to do it in another file, you have to do it there
too.
A lot of Perl so far has been "how do I limit the scope of the
complexity?" If you want to do backtracking in logic programming,
it's limited inside a regular expression. So limiting the complexity
is important. And to me, anything is fair so long as you predeclare
it. Even rewriting Perl in Latin. That's crazy, but fair. I think
that's okay.
Things we must do: we must manage the documentation well, so if people
want to chase down that complexity. They look at this new thing, they
see the declaration at the top and it says "use this funny syntactic
module". Gotta make sure the documentation is there and easily found.
And we have to manage the warpage at the semantic level, where there
are interactions between modules.
And there are other things we could talk about, but we're out of time.
I'd like to leave you with something that popped up on my screensaver:
"never promise to complete a project within six months of the end of
the year, in either direction". So we won't.
I don't know how long we can go here, but I'd be quite glad to take
questions. I presume that when they turn down the lights that'll be a
subtle hint.
(inaudible question)
My current leaning is that I really like the new ithreads model
because I think it simplifies the way you can think about it. For
those of you who are not aware, there are two models of threading in
perl5. One is more of your standard threading model, called pthreads,
where everything is shared (more or less) by default. That has
advantages in terms of efficiency, but not as much as you'd think.
There's another threading model, ithreads, which is more like a fork
inside the process. That could be inefficient, but I think if we do
what we're thinking of doing with copy-on-write semantics then it will
not be as inefficient as all that to clone a new thread, and I think
in some ways it's more safer to assume that all your global variables
are not shared and make you specifically declare all the things that
you want shared and how you want them shared. So that's my leaning.
I do expect that both models will be there, up to and including
sharing everything.
(inaudible question)
You're too obvious. The question is: what languages are we
considering for writing perl6 in. As I said, the parser and lexer I
want to write in Perl. I'm under no illusions that we can write the
runtime engine in Perl. Especially the interpreter. That would be
silly.
I think at the moment that we are leaning toward writing it in C but
with some amount of preprocessing of our own devising, which manages
some things. Maybe we're reinventing some of C++, but while there's
some interest in the Perl community in using C++, but not that much.
Yes, Bradley says "there's some interest in using Java". Well, yes
there is. Let me say that we're specifically thinking about not just
targeting a C back end, but also a Java backend and a C# backend. And
not just slapping it on at the end, but thinking in the design about
how we divide up the opcodes to make it map nicely onto these other
virtual machines as well as onto more C-ish things.
(inaudible question)
Which garbage collection methods are we looking into? That's one of
those things that we're deferring on purpose. We currently use a
reference counting methods which has some nice features and some
not-so-nice features. There's a number of different garbage
collection algorithms out there, which have different features from
your basic mark-and-sweep to generational and hybrid approaches.
What we're hoping is that we can keep the interface generic enough so
that we can plug different garbage collectors in. Part of the reason
for having a preprocessor for our own language is that we can write it
generically enough that we can preprocess it into C or Java or
whatever, but also that we can preprocess it into handling whatever
garbage collector that virtual machine wanted. Some virtual machines
specify their own garbage collector, and some don't.
(inaudible question)
The question is: what do I want to say about XML and Unicode. We've
had XML and Unicode support in one way or another in Perl for some
time now. The XML support is pretty heavily used, and there's been an
XML mailing list for several years, and there are a lot of XML modules
for Perl. Some work better than others, and there's some talk about
what needs to be done to make them work better.
In some ways there's an impedance mismatch between XML and Perl. If
you take a parser written in C and hook it up to Perl, you end up
building all these object hierarchies that use up a lot of memory and
it runs real slow. Part of what I didn't go into in the object design
is that if there are standard ways of mapping what look like Perl
objects onto low-level C-style objects so that where you could quickly
load them in and store them compactly in the way that C or C++ could
do, then you could speed up things like XML parsers.
Unicode is required to support XML. Perl 5.6 supports Unicode, for
some definition of supports. The story on Unicode is basically that
Perl's model of characters is going to be Unicode-based. Even if
there are different encodings floating around inside Perl, in the
abstract you look at a character and it has a particular character
number and it has a length of 1, regardless of how it is encoded. And
we carry this abstraction completely throug the language itself, and
the only place we have to worry what the real encoding is, is in
interfaces to the outside world, or in older Perl modules that were
making assumptions about bytes.
So we've got a pretty good migration path set out on that, and it's
pretty much there for Perl 5.6. Perl 5 characters are always coded
internally in utf8 or in bytes, and it keeps track of which it is for
you. IN Perl 6, since we have the vtable approach to the objects, we
can have different encodings and mix and match. This gives us a great
deal of scalability and flexibility.
Suppose you want to shoehorn Perl into your Pilot or your TV set top
box. You probably will say "there's only one internal format for
strings, maybe utf8". If anything else comes along in another form,
then you'll just translate it. That's probably not so efficient. On
the other hand, a big server pulling in all sorts of stuff might have
encodings going around: utf8, utf16, utf32, arrays of integers, big5,
whatever. All this stuff floating around in there, and if you have to
convert then you do it lazily to what you need. If you really want
everything to be fast, you run a complete crossbar, where you can
directly translate anything to anything else. It'll run really fast.
It'll be really big, but Moore's Law is on our side.
You get that scalability, where you can make the choices trading off
more implementation for speed. So that's the direction we're heading
with Unicode.
(long question)
That's a very good question. I don't know the answers. I can tell
you my basic philosophy on all these things: I'd like to have my cake
and eat it too.
(audient interjection)
Component models are very interesting and very important for some
people, and I think they ought to be supported. I don't know what to
tell you. Activestate has already been looking at putting perl5 into
the component model on Windows. And If we can program down to the C#
engine, then we get the integration for free.
Okay, thank you very much.
(applause).
