Lack of excitement for Pharo 1.4 seems symptomatic

Here's Lukas Renggli, the working horse that pulls Smalltalk behind him, announcing his plans for Pharo 1.4, or his lack thereof:

Just to repeat myself: With Pharo 1.4 having uncountable changes in core parts of the system and with the system including more and more forked and increasingly incompatible versions of packages (AST, RB, FS, Shout, Regex, …) I am unwilling to go through the same pain as with Pharo 1.3 again. In the current state, I don't see any of the code I am involved with (including Seaside, Magritte, Pier, OB, PetitParser, …) to move forward. I suspect that moving to another development platform soon causes less pain than to move to adopt the next Pharo :-(

In a word, nobody wants to upgrade to new versions of Pharo. But that was the whole point of Pharo: progress, moving ahead. There isn't a Smalltalk with traction right now. You can watch the Smalltalk community fail to answer which part of the Smalltalk universe is worth looking at at Stackoverflow.
Compare this with the spirit in the Objective-C/Cocoa world:

It’s OK to support only the newest version of iOS.

Now, you might say that this isn't a fair comparison, because writing iOS apps is the key to a new and exciting and fast-growing market, the mobile platform. But I think that's the same argument that Eric Schmidt is getting wrong about Android: “Whether you like Android or not, you will support that platform”. As Gruber points out, he's having cause and effect backwards: developers love programming for iOS, and only therefore is iOS as exciting as it is.
Smalltalk simply isn't as inviting to developers as it used to be.

Point-free programming in Smalltalk

Smalltalk does not, ordinarily, support point-free programming. Point-free programming is programming without "points", or variables. In a loop, rather than assigning every item to loop over to a variable, you only write which function should be called on them.

Pharo Smalltalk has limited support for that. There, you can write:

#(1 2 3 4) select: #even     -> #(2 4)

The above is implemented using an easy equivalence. Every block that sends only one message is equivalent to the selector of that message. Thus, the above is equivalent to:

#(1 2 3 4) select: [:e | e even]    -> #(2 4)

Which is just standard Smalltalk. The equivalence can be made work by implementing #value: etc. on Symbol.

Well, of course this is rather limited, we still haven't got rid of blocks. We still need blocks for everything that does not only send one message. You can push a little farther by allowing composition of symbols:

{'100' . '20' . '3'} sort: #<= * #first.     ->  #('100' '20' '3') 

The idea is to read the asterisk as "after", like the function compositor in Mathematics.

So you read this as "sort the area by less than, after first was applied". Thus, the strings are sorted by their first letter. Awesome, eh?

This is not standard Pharo, but it can be implemented in a couple of minutes. The code is on Snipt, but it's more easily written than read.

This can still not replace blocks. For example, the following block still cannot be expressed point-free:

[:a | Transcript show: a]

.

But nothing is easier than inventing a syntax for that (can you implement it in Pharo, so it'll work?):

 Transcript delayedSend: #show: 

Finally, for blocks where the left-hand-side is the argument:

 [:stream | stream nextPutAll: 'hello']

I propose that could be written as follows (can you implement it in Pharo, so it'll work?):

Delayed nextPutAll: 'hello'

Open questions

Would it be more enjoyable to write entirely point-free in Smalltalk? What do you think?

Full text search in Pharo?

Pharo does have something like full text search across all methods, and it works like this: Rightclick the text you'd like to search for, and choose "extended search" from the options. Then, choose "method source with it." A progress bar will appear, and you should get your answer within a minute. That has to do with the source not being loaded into Pharo at all times, it is read on demand out of the sources file.

However, the compiled sources are in memory from the start. While you can't meaningfully search compiled sources, you can search the string literals within. Again, rightclick the text you'd like to search for, and choose "extended search" from the options. But now, choose "method strings with it." That will bring up a list of all methods that contain the selected text as a string literal—instantly.

One of my most-hated methods in Pharo

    TestCase>fail
         ^self assert: false

What's wrong? Well, I often write self fail at the end of a unit test to remind myself that the test isn't yet done. Now, if I run the test, I care whether an assertion failed or whether the test solely failed. Of course I have no right to expect should detailed feedback from a test runner. But at least I shouldn't be misinformed. But that's exactly what happens. The TestRunner reports that an assertion failed if you call fail, even if none did.

Why's the Smalltalk debugger not focused on more in Smalltalk IDEs?

I can't help but wonder why Smalltalk's lovable trait of letting you develop in the debugger isn't leveraged far more by the IDEs. Suppose you just noticed that a method call doesn't have enough arguments. That is, you wrote this:

self doSomething.

But you should have written:

self doSomethingWith: anArgument

Well now, that's a fantastic situation for refactoring, isn't it? You could automatically rename the method in the receiver, and you'd know quite precisely who that receiver is. Little of that is leveraged. But of course, if Smalltalk is to play any serious role in the future, we should play by our strengths.

Well, here's a strength: the most powerful debugger in any modern IDE.

A more robust doesNotUnderstand:?

Smalltalk makes it really convenient to write code in the debugger. So, how I often work is as follows:

1. Write a rough skeleton that calls "self fail" at some point.
   
2. Write a unit test that calls the skeleton with real data.
   
3. Finish the skeleton in the debugger, where I have test data right at hand to see if things work as intended.

Well, that's alright as far as it goes, but you can't work like that with doesNotUnderstand: methods.

doesNotUnderstand: aMessage
 self fail.

If you now call a non-existing method on an object of a thus modified class, you've built an endless recursion that is so close to the VM that you can't get into a debugger anymore.

Smalltalk's doesNotUnderstand: method is a dangerous superpower. And perhaps that's why just about nobody overwrites it in a standard Pharo image.

Open problems

Do you know a more robust implementation?

• Non-existing calls inside the handler for such should be caught, I think.
  
• It should work nicely with code completion. (Code completion seems to be an integral part of just about every current approach to help programmer's productivity, so it better work nice.)

expandMacrosWith:

In Pharo, there's a very useful method String>expandMacrosWith:, virtually without documentation. It's much like printf in C. I guess it's easily understood from a few examples, so here go:

"<1s> is replaced by the first argument, a string.
 <n> is a newline, <t> is a tab. 
 % escapes brackets."
'<1s><n><t>%<inject><n><t>^<1s>' expandMacrosWith: 'player'
➔
'player
 <inject>
 ^player'


"p calls printString":
'<1p>' expandMacrosWith: #(1 2 3)
➔
'#(1 2 3)'


"? prints one string or another, depending on a Boolean"
'Dear <1?Mr:Mrs> <2s>, your contract has expired.' 
  expandMacrosWith: false   with: 'Smith'
➔
'Dear Mrs Smith, your contract has expired.'

Pharo dojo in Bern, October 23–24

There will be a Pharo dojo in Bern. The announcement reads:

We will organize a joint Pharo sprint / Moose dojo during October 23-24, in Bern (at the Software Composition Group, University of Bern). There will be free food and drinks!
Some action points are mentioned on the dedicated page (of course, other ideas and interests are welcome as well).

For planning purposes, please let me, Tudor Girba, know if you will attend.

How to test post-conditions

A pretty good student, Joshua Muheim, just asked me what the simplest way was to check postconditions. And my answer was the following:

Joshua, this is a research question! To the best of my knowledge, currently and in practice, developers use steamroller tactics to check postconditions. The standard way to verify that the LHC works as intended, in a method exterminate on a class EarthPopulation:

exterminate
    | oldSize | 
    oldSize := self size.
    LHC switchOn.
    self size should < oldSize.

That is, you copy whatever you intend to change and then compare new with old value. It isn’t difficult to imagine that this can lead to trouble if whatever you want to change isn’t cloneable. And of course it isn't possible to switch off the overhead of these assertions.* And that’s where Histoory (by Pluquet et al) comes to the rescue. Not only does it work independent of the cloneability, it also provides a much cleaner syntax:

exterminate
    [LHC swithOn]
        postCond: [:old | old size should < self size]

Alas, it isn’t in the latest flavor of any programming language, so I truly hope that Histoory will make it into Pharo :).

* I added this sentence after Joshua's correct observation of this problem.

ThothCompiler revamped

ThothCompiler has been revamped very near completely. Get it by executing:

Gofer it
    squeaksource: 'ASTBridge';
    addPackage: 'ThothCompiler';
    load

Then go to the Preference Browser (System->Preferences in the world menu) and choose ThothCompiler for the option "whichCompiler".

ThothCompiler allows you to use a modern AST in Pharo, rather than the 20 year old default one. The code is parsed with Lukas' really fast RBParser and then transformed to the 20 year old nodes, from there compiled. What's the advantage?

Well, you can transform the modern AST nodes before compilation! It has a great visitor API, that lets you do all kinds of transformations. You can use ThothCompiler as is just to feel fancy, or better: subclass it and overwrite the transform: method, in which you can transform the AST to your liking.

ThothCompiler is now used in Helvetica (albeit under the name of RBCompiler, because things can never have enough names!).

Changes since the last blog post:

• Now uses the RBParser, which is much faster than SmaCC
  
• No more dependency on the NewCompiler or AST package
  
• Adds a preference to the system (in the system menu), where you can choose the compiler, even when you can't compile anymore :)
  

What to keep in mind when playing with compilers in Pharo

If you play with the compiler in Pharo, here's an urgent hint: don't lock yourself out of your image! If you break the compiler, you want to have something to fall back to. So here's what you do.

The compiler for the system is whatever Behavior compilerClass returns, so change that method to read out a preference:

Behavior>compilerClass
 "Answer a compiler class appropriate for source methods of this class."
 |default|
 default := Compiler.
 ^(Smalltalk classNamed:  (Preferences valueOfPreference: #whichCompiler ifAbsent: [^ default]) ) ifNil: [default] 

That's step 1. Second step is to create the preference, so you can read it out at all:

Preferences addTextPreference: #whichCompiler category: #compiler default: 'Compiler' balloonHelp: 'Compiler to be used throughout the system' .

That's it, you're set. Happy hacking!

And it's done, Phexample has no more lolcats

Lolcats have left the building, Phexample reads easier now.

Stack new isEmpty should not be true.

To get a meaningful error message out of this test, you're faced with a small dilemma. The Phexample framework lies in the should method, thus, for all that should sees, it is called on true, so the test might as well have been

true should not be true.

Now, to squeeze a reasonable error message out of this, such as "isEmpty should return false, but got true," it helps finding the unexecuted code snippet in the caller. Thus, the execution depends on who called you. Maybe not perfectly object-oriented, but context-oriented! There we have a new buzz-word: COP, context-oriented programming.

I learned a few things on the way: Bytecodes in Smalltalk have different sizes, thus you can't just walk backwards from where you are. The containing method must be read forwards, and once you hit the position of the current execution, you know you're a bit too far. Also, you find the current stack frame using the pseudo-variable thisContext, though there are some performance issues with it.

By the way, you can't really ice and copy the stack (someone forbade it in the source, for whatever reason), you're analyzing it as it is being used for the current computation. A bit scary, but it works quite nice. Pharo ftw! Try this in JavaScript!

Get the latest version of Phexample at SqueakSource (there's a Gofer-Script on that page that will do the downloading for you!.

Phexample's lolcats syntax

In this post I'll clear my throat on what there is to come. The number one criticism of Phexample is that Phexample sometimes reads like lolcats, because real code that asserts that a stack be empty reads like

stack should not be isEmpty.

The complaint feels like an instance of the complaint of how stupid it is that in OSX you need to drag CDs to the waste bin to eject them, or how in Windows the bin stands on the desk. Of course the metaphor is broken there, but you only notice it because the metaphor works great.

And now you compare that to the usual alternative, which is assert. The first thing you know about assert is that nobody can remember which one is the expected and which is the actual value (I'll take this fact and some of the below discussion from Josh Graham's discussion of the assert syntax). We're talking about assertEquals(stack.isEmpty, true). Or the other way around. I certainly can't remember that either.

In Java, improvements have been suggested:
assertEquals(expected(true), actual(stack.isEmpty());

I find that while Phexample does not allow you to express your examples in a way that completely parses as natural language, it certainly reads easy enough and gives a natural and easy distinction between the expected and the actual.

So there's my analogy: no other testing framework is required to express its tests in a way that perfectly parses as human language. Only because the tests already look so close to perfection is it that people single out the mismatch and decry it. I don't think this is an interesting problem worth solving at all.

Yet, the criticism follows Phexample wherever it goes, so I'll try and allow phrases like

stack isEmpty should be true.

The difficulty in making the above code work is that the testing framework is only activated after isEmpty was evaluated already. Which means that it isn't in the stack anymore, so we need to analyze the source code to find out which method caused the failure to display a meaningful error message.

Telling other people how to install your code

Previously, I created dirty ScriptLoader tricks to make it easier for others to download my code.

I am pleased to find out that Gofer is even easier than my dirty tricks, well without being a dirty trick: Gofer on Lukas's blog.

Phexample, because examples expand on one another

Imagine you want to test the Stack class, and your first test just
creates a test and checks if it's empty.

Then in your second test, you create the same empty stack, but now
push an element 'apple' and check if the size is one.

Then in your third test, you do all the same, and then you pop the
element and check that it is an apple.

Wouldn't it be great if you could write your test cases such that they
could expand on one another? If you didn't need to copy paste the code
of the previous test and just say that you require the previous one!

Well, that's what PhExample lets you do. And as a bonus, tests are
ignored if the test they expand on does not pass. This gives you less
failed tests, because examples are executed only if the examples they
expand on pass too! That gives you the simplest failing examples in
your set to look at!

And best of all, you can run all examples with ye old TestRunner!

The example discussed above would read in code as follows:

EGExample subclass: #ForExampleStack
       …

shouldBeEmpty
       "Create the empty stack"
       | stack |
       stack := Stack new.
       stack should be isEmpty.
       stack size should = 0.
       ^ stack

shouldPushElement
       "Push one element"
       | stack |
       stack := self given: #shouldBeEmpty.
       stack push: 42.
       stack should not be isEmpty.
       stack size should = 1.
       stack top should = 42.
       ^ stack

shouldPopElement
       "And pop it again"
       | stack |
       stack := self given: #shouldPushElement.
       stack pop should = 42.
       stack should be isEmpty.
       stack size should = 0.

Find the code at http://www.squeaksource.com/phexample.html

You can install it as follows using Gofer:

 Gofer new
  squeaksource: 'phexample';
  addPackage: 'Phexample';
  load 

The repository is write-global, so feel free to drop your ideas!

PhExample is based on JExample by Lea Haensenberger, Adrian Kuhn, and
Markus Gaelli. See JExample

Introducing ThothCompiler

ThothCompiler combines the strengths of the two most popular smalltalk compilers. It takes the extensibility and wealth of the AST implementation from the NewCompiler and combines it with the most important asset of the old Compiler that per default ships with Squeak: it actually works.

Installation

The installation is fantastically easy. Go to Thoth on Squeaksource and get ASTBridgeLoader.

In a workspace, then execute:

ASTBridgeLoader load


Add the method compilerClass to the CLASS side of your class to set the compiler for the class. Thoth installs itself as the default compiler of the image. If you're unhappy with that, for example because you don't like the lack of error messages when you accept uncompilable code, you can choose for each class which compiler is to be used. Behavior>compilerClasss sets the default.

How do I hack into it?

Subclass ThothCompiler and override #compile:in:classified:notifying:ifFail: and #evaluate:in:to:notifying:ifFail:logged: Transform the AST before generate is called. Perhaps I should provide some infrastructure here …

How it works

It is a really simple thing: It uses the parser of the NewCompiler, then transforms the AST into the old model, and then uses the backend of the old compiler to compile.

Blocks in Smalltalk can have their own temporary variables

Did you know that the following appears to be Smalltalk code? Runs fine in the current Pharo.

[:a | |b| 2] value: 3

Pharo – perhaps Squeak 2.0?

Saturday, two days ago, I had the pleasure of being present at the Berne Pharo sprint. It was not my first contact with Pharo, but close.

If I understand it correctly, then Pharo emerged because Squeak was torn into too many directions. Squeak was the one platform for child education, multimedia websites, and webserver development. Some code in the Squeak image was never released under a free licence, and the Squeak-licence itself is not an OSI-certified open-source licence. Squeak has more than 2000 open issues in its bugtracker, Mantis.

Pharo is a fork from Squeak that aims at removing cruft and turnung Squeak smalltalk into a more traditional programming environment, with a small core, stable and fast-evolving.

I am slightly suspicious about the reducing of the Image size. I don't know anything about it, but I still recall how in an interview, Dan Ingalls spoke about the advantages of an image that is NOT split into packages, where everything calls everything. I recall for myself that when I wrote a web application that was to send emails by itself, I was delighted to see that an SMTP client is included in a standard image already. It was fantastically easy to send an email message. On the other hand, I discovered at least one bug in the SMTP client code.

Still, I will use Pharo, and not Squeak, henceforward. The user interface is revised and does not look childish anymore. The number of open issues is small, and despite the alpha status, Pharo appears to be a lot more stable than Squeak. Pharo is already used as the reference implementation of Seaside, and thus, it is only a matter of time until it will draw a significant share of the Smalltalk world into using it for web development. Depending on how well it will manage to faciliate the installation of external packages, I would like to believe that for non-educational purposes, Pharo will take over all of the Squeak world, and serve as Squeak 2.0.

How fast is Comanche?

The other day I came across this guy who wants to serve webpages in less than a millisecond. Without parallelization. So, his web page is really simple, it looks like this: <h1>Hello TWUG!</h1>.

His point is that things like Twitter are difficult to scale across several machines. The slowness of dynamic pages, he claims, is tried to be made up by advanced caching strategies. He goes on to check how fast certain languages are at serving dynamic content. He gets these numbers:


Clearly, his machine is faster than mine, because serving the same file statically, my Apache (single-threaded using httpd -X) can do only 1850 requests per second, that is 0.5 ms per request. (I use this benchmark: httperf --hog --num-conn 10000 --uri http://localhost:9090/?name=Niko). Here are the numbers I get:


Anyway, I find interesting to mention that Comanche is not far from Apache. Comanche can serve the same page dynamically (the name is a GET parameter) in 1.1 ms, for which Apache takes statically 0.5 ms. As for Apache-php vs. Comanche, I'd call it a tie.

The Squeak code for the Http server would be this:

ma := ModuleAssembly core.
ma addPlug: [:request | HttpResponse fromString: 
   (String streamContents: [:s| 
   s nextPutAll:'<h1>Hello! '; 
   nextPutAll:( request  fields at: 'name'); 
   nextPutAll: '</h1>']).].
(HttpService startOn: 9090 named: 'Example') module: ma rootModule

By this way, as the above snippet suggests, a standard Seaside installation for Squeak is quite able to make RESTful applications. Just avoid the Seaside framework and plug right into the Comanche web server.

For completeness, the PHP-code:

<h1>Hello <?=$_REQUEST['name']?></h1>

Update

By the way, this seems to be about 10 % faster:

s:=(HttpService on: 8080 named: 'Example Http Service').
s onRequestDo: [ :request |
 HttpResponse fromString: 
  (String streamContents: [:s| s nextPutAll:'<h1>Hello! '; 
  nextPutAll:( request  fields at: 'name'); 
  nextPutAll: '</h1>']) ].
s start

Adding new properties to a class

When you load a new toy into Squeak and toy with it in a workspace, that is usually fun. So much more fun than creating a new class. I always suspect that I use the Squeak tools all wrong, but honestly, to create a new class with a couple of properties, it takes me fivethousand clicks to set all the instance variables, set the accessors and get them all properly initialized.

So, I still don't know if there might not be a much easier way to do it, but I'm fine with what I've dreamt up today. If I want to add a new instance variable to a class, including all accessors, I call this from a workspace:

ManyManyRelation addPropertyName: #oneLot.

And if I want to initialize the property to something, I use this:

ManyManyRelation addPropertyName: #otherLot lazyInitializeWith: [Set new].

The following method goes to the instance side of Class:

addPropertyName: aSymbol
  self addInstVarName: aSymbol.
  self compileAccessorsFor: aSymbol

And then:

compileAccessorsFor: aSymbol

"Compile instance-variable accessor methods
  for the given variable name "
   self compileSilently: aSymbol asString , '
       ^ ' , aSymbol classified: 'access'.
   self compileSilently: aSymbol asString , ': anObject' 
        , aSymbol , ' := anObject' classified: 'access'

You can now call something like ManyManyRelation addPropertyName: #oneLot, and it adds an instance variable to ManyManyRelation, including both accessors.

Let's push it one step further. Hooking into initialize really doesn't feel right––if objects are about modelling the real world, then what exactly corresponds to object initialization? While still not being optimal, the following creates lazy initizers in the accessors.

The following two methods go as instance methods into Class:

addPropertyName: aSymbol lazyInitializeWith: aBlock
   self addInstVarName: aSymbol.
   self compileAccessorsFor: aSymbol lazyInitializeWith: aBlock

compileAccessorsFor: aSymbol lazyInitializeWith: aBlock 
    | initializeCommand decompileString |
    decompileString := aBlock decompileString.
    initializeCommand := decompileString copyFrom: 
                2 to: decompileString size - 1.
    self compileSilently: aSymbol asString , '
        ^ ' , aSymbol , ' ifNil: [ ' , aSymbol , 
          ' := ' , initializeCommand , ']'  classified: 'access'.
    self compileSilently: aSymbol asString , ': anObject
    ' , aSymbol , ' := anObject' classified: 'access'

You use it by calling ManyManyRelation addPropertyName: #otherLot lazyInitializeWith: [Set new].

By the way, I would refrain from using (CreateAccessorsForVariableRefactoring variable: #hasMany class: OneManyRelation classVariable: false) execute, which is what happens when you create accessors using OmniBrowsers, the reason being it behaves strangely when the object already responds to a message, most typically like name.