Matthias, all:
Thanks very much for the considered replies. As a Racket newbie,
this gives me insight into how Racket is used to solve complex
problems.
Complexity exists, and require complex solutions. But often I
am faced (or even have designed) complex solutions that either
did not solve the problem, or solved it at a cost that was too
high.
I don't think the article is vacuously true - it is not against
complexity, it is against complexity that costs too much.
My personal experiences with DSLs has been in situations
where they were overused, or used with poor tools (and
again I hasten to add, not with Racket). I've worked with
too many people (including myself) that fall in love with
their own complex designs.
Thanks
John
On 6/21/13 1:35 PM, Matthias Felleisen wrote:
On Jun 21, 2013, at 8:26 AM, John Gateley wrote:
Subject for discussion:
http://firstround.com/article/The-one-cost-engineers-and-product-managers-dont-consider#
Interesting sentence in the middle:
Consider DSLs, abstractions and the attraction to being the one to
build a framework that gets leveraged for years.
I think Racket is a different target: education vs. engineering (is
this true?). As a software
engineer, I really agree with the article. Complexity is almost
always a terrible thing,
whether it is a DSL, a complex implementation of a simple interface,
or just the
one additional thing requested by product management that didn't fit.
For Racket: are DSLs a source of complexity? Or would you argue that
they reduce the
complexity normally introduced with DSLs?
John,
this article's claim concerning abstractions and DLS is vacuously true
so it's also devoid of any information. I grant the
maintenance-construction cost ratio; I teach to this slogan --
starting with How to Design Programs through HtD Components and HtD
Systems. It is the guiding principle.
If we wanted to turn this person's essay into a well-founded statement
that helps engineers, we would first have to clarify what complexity
is or what simplicity is. No, "I know it when I see" it won't work
here. To clarify, I am sure that many people will say that C is a
simpler language than Racket. If we follow the article's
recommendation then, we should use C. But as you know, C lacks safety
and memory safety and these gaps seriously impeded software
development and maintenance. The lack of safety means that you never
know whether the output of a C program is serious or whether it's some
random bits from some place in memory interpreted as, say, an int. The
lack of memory safety in particular destroys modularity. Every dyn mem
handed over from one component to another must be tracked and
accounted for.
Not every language that is more complex than C will reduce the cost of
software construction and maintenance. To wit, C++ started out as a
more complex variant of C and to this day it is 'sold' that way even
though it moved away from its roots over the past 10 years. Its
complexities introduce seriously deeper safety problems, which
measurably impact software construction and maintenance problems. IBM
believes that this cost is a factor of 3x to 5x when compared to Java,
another language that is definitely more complex than C. The San
Francisco project under Kathy Bohrer, a Rice grad from around your
time, ran the project in C++, switched to Java, and convinced a lot of
people at IBM to measure this cost. A few years later the company
switched all software to Java for these reasons. This is not to say
that Java is good; but it does say that they actually measured cost,
compared, and went from simple and complex languages to other complex
languages. If I were a senior software dev manager at a company, I
would pay attention to someone who measures and compares instead of
someone who writes content-free polemics that actually sound correct.
The toggles examples from the article is apt here. The lack of safety
in C means that there is no isolation and every line in a system may
potentially affect the behavior of every other line -- just like the
toggles/switches mentioned. But now imagine, the guy had first built a
box around the first two switches so that only their relevant behavior
is visible -- say two states -- and then added a third one. In that
case, the interactions would be fine.
LESSON 1: simplicity by itself is not an advantage in software
development.
LESSON 2: complexity comes in many flavors, some good, some bad.
Now let's move on to DSLs. A DSL, like any abstraction, helps you
reduce cost if it is well designed and meets your needs. If you don't
have a need in existing code for an abstraction, don't build it. If
you do have a need,
-- understand the abstraction mechanisms of your language
-- study the concrete cases of repetition, extensive verbiage to say
things
-- use the abstraction mechanisms of your language to create an
abstraction that removes your repetitions, extensive verbiage.
If you work in Java, you don't have good abstraction mechanisms to
eliminate domain-specific verbiage from your systems. My recent
reading experience with three books on industrial DSL building tools
firmly convinced me that current practice can easily flip into
counter-productive architecture acrobatics. If you build these DSLs,
you may make the system more costly. If you work in Racket, you have
great tools for building internal DSLs and you can smoothly integrate
modules written in different DSLs. As I said, this thoughts apply to
any abstraction mechanism, internal DSLs are just the most powerful
form of abstraction.
LESSON 3: if you have bad tools, building an abstraction may increase
the cost of building a system
LESSON 4: with good tools, you're likely to reduce the cost, but
ill-trained programmers can and do mess up
Let me finally address internal complexity vs external complexity.
When you construct a language like Java, you are actually building an
extremely complex system. But, the Java community succeeded beyond
imagination and possibly beyond justification with making their
language appear well-layered, well-structured and simple. Guy Steele
has stated this as "you have to make things sufficiently complex to
make them appear simple" and Matthew has stated similar experiences
with the design of Racket again and again. The key is that very few
Java programmers experience the complexity that the Java box hides;
most have to cope with the complexities of the language. Here is an
example that is more straightforward. AMPL is an external DSL for
writing down mathematical programs (linear, integer, binary, graphs,
networks). If you are an applied math person, the only difference
between AMPL programs and paper and pencil is that the former uses
ASCII. (Perhaps they use unicode these days.) It is definitely at the
right level and reduces the huge cost of maintaining mathematical
programs in, say, Fortran, a vastly simpler language than AMPL -- as
far as the internals are concerned. AMPL is after all a complex PL,
with niffty parsers and abstraction capabilities for plugging in all
kinds of solvers. But it neatly separates the solver from the problem
statement, for example, which while internally complex, makes it
externally simple.
LESSON 5: Do not confuse external and internal complexity. A proper
separation may vastly decrease the cost of creating programs.
Very last thought on 'complexity.' All of the above assumes that the
time of programmers is more costly than what it takes to run the
computation. I know of cases where this is not true. In that case, you
might be better off sacrificing programmers on the altar of machine
time. The cases I know are extremely rare and involve expensive
programmers who understand a lot of continuous mathematics and who
work with computers that are extremely expensive to use.
;; ---
As for Racket: yes, it is good at DSLs because we had to pay attention
to building languages for our original educational goals. But for the
last 10 years, the focus has widened to include a lot of industrial
and quasi-industrial uses. It is no longer the TI Scheme of yore.
Your milage may vary. -- Matthias
____________________
Racket Users list:
http://lists.racket-lang.org/users
