Saturday, December 31, 2011

Anonymous Functions in Dart

‹prev | My Chain | next›

Now that I know how to execute Dart locally, I am ready to begin poking through the language a bit. First, I would like to explore the different ways to write anonymous functions. Be they used for currying, partial application, or simple callbacks, anonymous functions are a must in any language. Interestingly, I have already seen two different formats in Dart. I wonder how many more there are...

The one that I have been using for the past two days is a sort of naked argument list followed by the function body inside curly braces:
main() {
  var list = [0,1,2,3,4];
  list.forEach( (i) { print("Item: " + i); } );
}
(try.dartlang.org)

That format is very reminiscent of Ruby blocks:
list = [0,1,2,3,4]
list.each { |i| $stderr.puts "Item: " + i }
To be sure, there are more curly braces and semi-colons in the Dart equivalent, but it is nice being able to drop the function keyword from the Javascript equivalent.

If I save and run that Dart script, I get:
➜  command_line git:(master) ✗ dart iterator01.dart
Item: 0
Item: 1
Item: 2
Item: 3
Item: 4
Interestingly, it is possible to make this Dart code look even more Javascript-y with a "function" before the argument list:
main() {
  var list = [0,1,2,3,4];
  list.forEach( function(i) { print("Item: " + i); } );
}
(try.dartlang.org)

That "function" is not a keyword in dart however, it is the name of the function (in effect, I have named my function "function"). I could have called it "f":
main() {
  var list = [0,1,2,3,4];
  list.forEach( f(i) { print("Item: " + i); } );
}
(try.dartlang.org)

So what are those things? Named anonymous functions? Somehow that does not quite work. Named blocks? Maybe.

At any rate, they lend themselves to recursion nicely:
main() {
  var list = [0,1,2,3,4];

  list.forEach( f(i) {
    print("Item: " + i);
    if (i > 0) f(i-1);
  });
}
(try.dartlang.org)

Yah, it is pretty silly to do recursion in an iterator block, but still, it works:
➜  command_line git:(master) ✗ dart iterator04.dart
Item: 0
Item: 1
Item: 0
Item: 2
Item: 1
Item: 0
Item: 3
Item: 2
Item: 1
Item: 0
Item: 4
Item: 3
Item: 2
Item: 1
Item: 0
Just as in Javascript, it is possible to assign anonymous functions to variables:
  var f = (i) {
    print("Item: " + i);
  };

  var list = [0,1,2,3,4];
  list.forEach(f);
(try.dartlang.org)

And, yes, regular function naming works as well:
  f(i) {
    print("Item: " + i);
  };

  var list = [0,1,2,3,4];
  list.forEach(f);
(try.dartlang.org)

That pretty much covers this anonymous function format, but then there was also a hash rocket format that I encountered:
  var list = [0,1,2,3,4];
  list.forEach( (i) => print("Item: " + i) );
(try.dartlang.org)

This compact format seems to only work with a single expression. That is, both of the following are syntax errors:
  // wrong #1
  var list = [0,1,2,3,4];
  list.forEach( (i) =>
      print("Item: " + i);
      print("      " + i);
  );

  // #wrong #2
  var list = [0,1,2,3,4];
  list.forEach( (i) =>
      print("Item: " + i)
      print("      " + i)
  );
(try.dartlang.org)

The first fails on the semi-colon:
'/home/cstrom/repos/dart-site/examples/command_line/iterator10.dart': Error: line 4 pos 26: ')' expected
      print("Item: " + i);
                         ^
The second fails on the second print statement in what is supposed to be a single expression:
'/home/cstrom/repos/dart-site/examples/command_line/iterator10.dart': Error: line 5 pos 7: ')' expected
      print("      " + i)
      ^
But it does work if I combine those two statements into a function that can be called as a single expression:
  var list = [0,1,2,3,4];

  double_print(i) {
    print("Item: " + i);
    print("      " + i);
  }

  list.forEach( (i) => double_print(i) );
(try it on try.dartlang.org)

Checking through the language specification, it seems like those two formats (argument list + block and hash-rocket + expression) are the two available anonymous function formats allowed in Dart.

While looking through the language specification, I notice that there do appear to be optional arguments. While messing about, I had tried a Ruby-ish:
f(i, j=0) { /* ... */ ]
But that had not worked. Looking through the formal specification, I do not see any examples of optional arguments, but some of the discussion seems to show optional arguments inside square brackets:
main() {
  pretty_int(i, [label="Number: "]) {
    print(label + i);
  }

  pretty_int(1);
  pretty_int(42, "Answer to everything: ");
}
(try it on try.dartlang.org)

And that does seem to work as expected:
➜  command_line git:(master) ✗ dart optional_arg.dart 
Number: 1
Answer to everything: 42
With a pretty good handle of functions under my belt, I call it a night here. Up tomorrow, I think that I will play with currying functions in Dart. I love me some good currying.


Day #251

Friday, December 30, 2011

Command-Line Dart and Frogc (Wha?)

‹prev | My Chain | next›

Thanks to some pointers from @dartbits, it seems that there may be some hope of using a real editor, namely Emacs, with Dart.

To do so, I need to download the Dart SDK, which can be found amongst the Dart builds. The SDK are the ones without "DartBuild" in the filename. For my Ubuntu system, that means that I need "dart-linux.zip":
➜  Downloads  wget http://gsdview.appspot.com/dart-editor-archive-continuous/latest/dart-linux.zip
...
2011-12-30 16:08:40 (130 KB/s) - `dart-linux.zip' saved [6683485/6683485]
Using the mnemonic suggested in yesterday's comments (unzip -d -- I might actually remember that), I extract the SDK into a temporary directory until I can decide what to do with it:
➜  Downloads  unzip -d ~/tmp/dart dart-linux.zip
Taking a look at the contents, I find:
➜  dart  cd ~/Downloads 
➜  Downloads  cd ~/tmp/dart
➜  dart  ls
dart-sdk
➜  dart  ls dart-sdk/*
dart-sdk/revision

dart-sdk/bin:
dart  frogc  frogc.dart

dart-sdk/lib:
builtin  core  coreimpl  dom  frog  html  htmlimpl  json

dart-sdk/util:
dartdoc
For what it is worth to future readers, the revision that I am using here is 2886:
➜  dart  cat dart-sdk/revision 
2886
Since the dart SDK follows an almost UNIX like directory structure (with bin and lib sub-directories), I am almost tempted to move those sub-directories directly into my $HOME/local. I worry about confusion with the likes of $HOME/local/lib/builtin, so, at least for now, I move the dart SDK directory into $HOME/local and sym-link the executables:
➜  dart  mv dart-sdk ~/local 
➜  dart  cd ~/local/bin/
➜  bin  ln -s ../dart-sdk/bin/dart 
➜  bin  ln -s ../dart-sdk/bin/frogc
➜  bin  ln -s ../dart-sdk/bin/frogc.dart 
First, I think that I will try to replicate yesterday's Dart to Javascript compilation with these tools. If I am successful, I can eschew the Eclipse-based Dart Editor in favor of my beloved Emacs.

Back in the directory containing my simple iterating code, I "frogc" (hunh?) the dart code:
➜  iterator  frogc iterator.dart             
error: File not found: /home/cstrom/local/lib/core/core_frog.dart
error: File not found: /home/cstrom/local/lib/html/html.dart
iterator.dart:6:5: warning: can not resolve "document" on "iterator"
    document.query('#status').innerHTML = '';
    ^^^^^^^^
iterator.dart:6:5: warning: document is not defined anywhere in the world.
    document.query('#status').innerHTML = '';
    ^^^^^^^^
iterator.dart:6:5: warning: can not resolve "noSuchMethod" on "iterator"
    document.query('#status').innerHTML = '';
    ^^^^^^^^
iterator.dart:6:5: warning: noSuchMethod is not defined anywhere in the world.
    document.query('#status').innerHTML = '';
    ^^^^^^^^
compilation failed with 2 errors
Unhandled exception:
Compilation failed
 0. Function: '::main' url: '/home/cstrom/local/dart-sdk/lib/frog/minfrogc.dart' line:32 col:5
 1. Function: '::main' url: '/home/cstrom/local/dart-sdk/bin/frogc.dart' line:11 col:16
Yikes! Well, not too bad I suppose. The reason for the failure is that the compiler cannot find the core libraries. Looks as though I need to sym-link them into my $HOME/local/lib directory after all:
➜  iterator  cd ~/local/lib
➜  lib  ln -s ../dart-sdk/lib/* .
➜  lib  ls -l
total 14068
lrwxrwxrwx  1 cstrom cstrom      23 2011-12-30 21:51 builtin -> ../dart-sdk/lib/builtin
lrwxrwxrwx  1 cstrom cstrom      20 2011-12-30 21:51 core -> ../dart-sdk/lib/core
lrwxrwxrwx  1 cstrom cstrom      24 2011-12-30 21:51 coreimpl -> ../dart-sdk/lib/coreimpl
lrwxrwxrwx  1 cstrom cstrom      19 2011-12-30 21:51 dom -> ../dart-sdk/lib/dom
drwxr-xr-x  2 cstrom cstrom    4096 2011-05-14 21:39 engines
lrwxrwxrwx  1 cstrom cstrom      20 2011-12-30 21:51 frog -> ../dart-sdk/lib/frog
lrwxrwxrwx  1 cstrom cstrom      20 2011-12-30 21:51 html -> ../dart-sdk/lib/html
lrwxrwxrwx  1 cstrom cstrom      24 2011-12-30 21:51 htmlimpl -> ../dart-sdk/lib/htmlimpl
drwxr-xr-x  3 cstrom cstrom    4096 2011-03-06 12:00 io
lrwxrwxrwx  1 cstrom cstrom      20 2011-12-30 21:51 json -> ../dart-sdk/lib/json
...
At least that is a little better than moving those Dart sub-directories directly into $HOME/local/lib—the sym-links describe the source of what might otherwise be mysterious entries 6 months hence.

Back in my simple iterator directory, I try again:
➜  iterator  frogc iterator.dart
/home/cstrom/local/lib/htmlimpl/htmlimpl.dart:23094:21: warning: a map literal takes one type argument specifying the value type
    _listenerMap = <String, EventListenerList>{};
                    ^^^^^^
Er... What?

That is not an error from my code. That is from Dart itself. Ugh. Checking line 23,094 of htmlimpl.dart, I see:
EventsImplementation._wrap(this._ptr) {
    // TODO(sigmund): the key type (String) yields a warning in frog and the vm,
    // but it is currently necessary to compile with dartc.
    _listenerMap = {};
  }
Ah, so it is an expected warning. But why then did the expected Javascript version of the app not compile?
➜  iterator  ls -l
total 2272
-rw-rw-r-- 1 cstrom cstrom     416 2011-12-30 21:58 iterator.dart
-rw-rw-r-- 1 cstrom cstrom 2125072 2011-12-30 00:05 iterator.dart.app.js.bak
-rw-rw-r-- 1 cstrom cstrom  182516 2011-12-30 21:58 iterator.dart.js
-rw-rw-r-- 1 cstrom cstrom     220 2011-12-29 23:07 iterator.html
drwxrwxr-x 4 cstrom cstrom    4096 2011-12-30 00:04 workspace
It turns out that it did compile. I had expected it to be named iterator.dart.app.js like it was when the Dart Editor generated it. Instead it is just iterator.dart.js. And wow, it is an order of magnitude smaller than the Editor generated version.

Of course, it does not mean squat unless the smaller version actually works. So I point the HTML to the new Javascript file and load the page in the browser to find:


Awesome. That is a huge win for me. Not only was the Dart Editor not Emacs, it was also a bit slow with the code auto-completion (which I detest anyway) and took up much memory.

In addition to compiled Dart into Javascript, I expect to want to execute pure Dart from the command line as well. That should be what the dart command is for:
➜  iterator  dart iterator.dart
'/home/cstrom/dart/iterator/iterator.dart': Error: line 1 pos 1: library handler failed: Do not know how to load 'dart:html'
#import('dart:html');
^
Hrm... One the one hand, why would I want to load dart:html when I am working on the command line? On the other hand, if I want to load it, why can't I?

I can kinda-sorta get it working by changing the #import statement to:
#import('./lib/html/html.dart');
// #import('dart:html');
And sym-linking the html library:
$ ln -s ~/local/dart-sdk/lib .
does the trick. Sort of:
➜  iterator  dart iterator.dart
'/home/cstrom/dart/iterator/iterator.dart': Error: line 1 pos 1: library handler failed: '/home/cstrom/local/dart-sdk/lib/html/html.dart': Error: line 3 pos 1: library handler failed: Do not know how to load 'dart:dom'
#import('dart:dom', prefix:'dom');
^

#import('./lib/html/html.dart');
^
Of course dart:html references dart:dom so now that does no resolve. I am not going to worry about that too much. Not yet at least. For now, it is good to be able to call and compile Dart code.

I call it a night here. Tomorrow, I plan to get a little more familiar with this dart executable. As well as Dart itself.



Day #250

Thursday, December 29, 2011

Getting Started with the Dart Editor (and Dart Mode)

‹prev | My Chain | next›

While my Dart-enabled browser continues to download and build (seriously, do I really need 4.5G of "LayoutTests" to do that?), I think it wise to skip ahead and try alternate means of running actual Dart code.

That means downloading the latest, stable (for now) version of the "Dart Editor":
➜  Downloads  wget http://gsdview.appspot.com/dart-editor-archive-integration/latest/DartBuild-linux.gtk.x86_64.zip
Bah! I hate ZIP files. I never know if they contain a single directory with everything in it or a bunch of files and sub-directories that are going to get mixed up with my current working directory. No doubt there is a tar tzf equivalent, but I know I will never remember it. So I create a temporary directory into which I can extract the ZIP file:
➜  ~  cd ~/tmp
➜  tmp  mkdir dart-editor
➜  tmp  cd !$
➜  tmp  cd dart-editor
➜  dart-editor  unzip ~/Downloads/DartBuild-linux.gtk.x86_64.zip
...
It turns out that there is a single directory, dart, in the ZIP which contains:
➜  dart-editor  ls -1 dart
about_files
about.html
configuration
DartEditor
DartEditor.ini
features
icon.xpm
libraries
plugins
samples
workspace
I normally install local code like this in my $HOME/local directory (rather than risk a sudo install). That directory seems a reasonable place for this:
➜  dart-editor  mv dart ~/local 
➜  dart-editor  cd ~/local/bin 
➜  bin  ln -s ../dart/DartEditor
The ~/local/bin directory path is already in my $PATH, so, by sym-linking the editor in there, I can run DartEditor from anywhere (including Gnome Do):
➜  ~  DartEditor
When I do that, I get a lovely splash screen:


Followed by a very IDE looking editor:


Which contains some simple getting started instructions:


Following those instructions, I would like to experiment with simple iterators:


Quite nicely, that creates a new iterator.dart file and an iterator.html file that will load the Javascript compiled version of that Dart code. The generated Dart is a pretty decent start. It simply appends "Hello World!" to the "status" element in the calling web page:


Since I desire to try a simple iterator, I modify the code slightly:
#import('dart:html');

class iterator {

  iterator() {
    document.query('#status').innerHTML = '';
  }

  void run() {
    [0,1,2,3,4,5,6,7,8,9].forEach((i) {
      write("Hello World #" + i + "!<br/>");
    });
  }

  void write(String message) {
    // the HTML library defines a global "document" variable
    document.query('#status').innerHTML += message;
  }
}

void main() {
  new iterator().run();
}
In there, I blank out the "status" <div> in the iterator() constructor. Then, in run() I iterate through a list of integer elements (I will have to explore ranges another day) with the forEach() method. There seem to be a number of different ways to express an anonymous function in Dart. Here, I use the argument list followed by curly braces. Last up, I change write() to append to the "status" <div>.

When I try to run that code with the "Run in Browser" button, however, I am greeted by:


So I manually compile the dart into Javascript by selecting "Generate Optimized Javascript..." from the "Tools" menu and supplying the output path as: /home/cstrom/dart/iterator/iterator.dart.app.js. That seems to work (at least it didn't error out).

Checking it out in my browser, I see:


Yay!

In the end, I am going to use Emacs to edit Dart code just like I use Emacs to edit everything else. Happily, there is an Emacs wonk or two at Google, meaning that there is already a dart-mode major Emacs mode. It is not in elpa yet, so I have to install it the old-fashioned way:
➜  ~  cd ~/.emacs.d 
➜  .emacs.d git:(24) ✗ 
➜  .emacs.d git:(24) ✗ wget http://dart-mode.googlecode.com/git/dart-mode.el
...
2011-12-29 22:35:48 (169 KB/s) - `dart-mode.el' saved [15705]
Then, in my dot emacs file, I require and auto-load dart-mode:
(require 'dart-mode)
(add-to-list 'auto-mode-alist '("\\.dart\\'" . dart-mode))

With that, I have beautiful Dart code (well, not hideously ugly) in a beautiful editor:


That is a good stopping point for tonight. Hopefully tomorrow "Dartium" will have finished compiling.

Day #249

Wednesday, December 28, 2011

My First Attempt at a Dart Program

‹prev | My Chain | next›

For my first efforts at Dart, I would like to get a simple "hello world" application running. As a stretch goal, it would be nice to play with iterators and the DOM, but let's see how far I get with the simplest case first.

I create a very simple web page with minimal Dart (or what I think passes for Dart) in it:
<!DOCTYPE html>
<html>
<head>
<script type="application/dart">
#import('dart:html');

main() {
  Console.log("hello");
}
</script>
</head>
<body>
<h1>Hello Dart
<div id="target"></div>
</body>
</html>
When I load that up in Chrome... nothing.

Thinking that maybe I have the lt;script> tag wrong, I change strategy. Instead of defining the dart script in the HTML, I load it from a separate file:
...
<script type="application/dart" src="iterator.dart"></script>
...
It is at this point that I realize that it is not possible to do Dart natively in Chrome yet. Bummer.

Almost resigned to doing all of this in the "Editor" thingy, I happen to stumble across build instructions for a dart-enabled Chrome.

I learn the hard way that this will not run well from an interactive zsh prompt. So first I switch to bash and then run the installer:
➜  dart-site git:(master) ✗ cd ~/src                            
➜  src  bash
1.9.2 ~/src $ source install-build-deps.sh 
Eventually, it runs to the point that I am asked what to do with 32 bit libraries:
...
Installing 32bit libraries not already provided by the system

This is only needed to build a 32-bit Chrome on your 64-bit system.

While we only need to install a relatively small number of library
files, we temporarily need to download a lot of large *.deb packages
that contain these files. We will create new *.deb packages that
include just the 32bit libraries. These files will then be found on
your system in places like /lib32, /usr/lib32, /usr/lib/debug/lib32,
/usr/lib/debug/usr/lib32. If you ever need to uninstall these files,
look for packages named *-ia32.deb.
Do you want me to download all packages needed to build new 32bit
package files (y/N)
Curious, I answer "No". This turns out to be a bad idea because the installer exits. Re-running the script, I am then greeted with:
...
It produces the following output:
Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 97 reinstalled, 0 to remove and 0 not upgraded.
Need to get 0 B/61.9 MB of archives.
After this operation, 0 B of additional disk space will be used.
E: Internal Error, No file name for libstdc++6

You will have to install the above packages yourself.
Bah! This seems to be a bug in the installer script. Rather than troubleshooting it myself (I just want some progress tonight), I remove one of the packages previously installed:
➜  dart-site git:(master) ✗ sudo apt-get remove language-pack-fr
And then re-try the script, this time answering "Yes" to installing the 32-bit libraries.

I am forced to call it a night there as the installer is still downloading the entire contents of the Ubuntu repository. Hopefully tomorrow, I will have my very own Dart-enabled browser.

Day #248

Why DART?

‹prev | My Chain | next›

My next book is going to be on the Dart programming language, lovingly entitled Dart for Hipsters.

tl;dr

  • Dart looks to be an interesting language
  • Dart addresses real-world, legitimate limitations of Javascript
  • Google is not evil for proposing Dart
  • Follow along daily in this blog or sign-up for the mailing list

Why?!

Why Dart? More to the point, why Dart given that Javascript is the lingua franca of the web and has already won?

Simply put, it is silly to suggest that Javascript is the best that we can ever do. Don't get me wrong, I love Javascript. Of the 700+ posts on this blog, I'd estimate that 600+ discuss Javascript in some way. I love node.js, jQuery, and Backbone.js (heck I co-authored a book on it). I don't think Javascript is going away any time soon.

But that does not mean that we should be satisfied with Javascript. Nor should we be satisfied with the proposed additions to the Javascript standards.

For the past several years, I have developed something of an obsession for speed. One of the things that drew me to node in the first place was the speed promised (and more or less realized). After node, I began to look at HTTP as a bottleneck for responsive applications. The result, of course was The SPDY Book. Even as I grew to understand and appreciate the power offered by SPDY, it became apparent that it only solves part of the problem. It gets stuff to the browser blazingly fast, but does nothing to speed up application boot and execution.

I see many parallels between SPDY and Dart. With SPDY, rather than trying to work around the limitations of the status quo, (e.g. HTTP "optimization"), Google produced a completely different protocol adapted for the modern web. With Dart, it seems that Google is once again trying to do something completely different rather than patch more cruft onto a language barely newer than the internet itself.

I am not buying the argument that Dart (or SPDY) are Google's attempt to subvert the web. Google has been very good about embracing the community with SPDY. Google has been extremely open about the standard. I could not have written The SPDY Book if they were trying to keep tight control of SPDY. Surely Amazon could not have used SPDY to power the Kindle Fire if Google were hell bent on tight control of the protocol.

Similarly, I see Dart as a logical response to an obvious performance bottle neck in web applications. Javascript is slow. It lacks fundamental things that most "real" languages take for granted: libraries, iterators, dates. To be sure, there are workarounds that more or less address most of these concerns, but why settle? Why not start from the ground up with support for the things that cause the most pain?

Is Dart the language to do all that? Is Dart today better than Javascript? I dunno and prolly not.

But it has potential. And it's trying something different. And it's inherently different than the status quo.

That makes it worth learning.

Join me as I blog about this every day for 3+ months straight. At the end of January, an alpha version of the book will be available to purchase at a discounted rate. At the end of February, the beta will arrive. The first edition will be available at the end of March. Want more? Sign-up for the mailing list.

Tuesday, December 27, 2011

Git-Scribe Syntax Highlighting on Ubuntu

‹prev | My Chain | next›

I had a couple of surprises while playing with the latest version of git-scribe yesterday. First, I was able to generate different size output (7.5in x 9in) (that did not, in fact, work -- I was using the wrong gem). Second, I could not get code syntax highlighting working.

I tried both over a few months back and failed to get the re-sizing working, but definitely got the syntax highlighting working:


Unfortunately, I seem to have deleted the spike that allowed that syntax highlighting to work.

The commit that added syntax highlighting to upstream hard-coded the xslt highlighting library to the most likely installation location under OSX. Since I am on Ubuntu, I need to switch the location of that file accordingly:
<!-- xsl:param name="highlight.xslthl.config">file:///usr/local/Cellar/docbook/5.0/docbook/xsl/1.76.1/highlighting/xslthl-config.xml</xsl:param -->
<xsl:param name="highlight.xslthl.config">file:///usr/share/xslthl/highlighters/xslthl-config.xml</xsl:param>
But that has no effect—there is still no syntax highlighting present in the resultant PDF.

There are at least three locations for xslthl-config.xml on my system:
➜  ~  find /usr/share | grep xslthl-config.xml
/usr/share/xslthl/highlighters/xslthl-config.xml
/usr/share/xml/docbook/stylesheet/docbook-xsl/highlighting/xslthl-config.xml
/usr/share/xml/docbook/stylesheet/docbook-xsl-ns/highlighting/xslthl-config.xml
None of them do the trick.

My next step is to switch back to the last commit in the upstream gem:
➜  git-scribe git:(upstream) ✗ git stash save
Saved working directory and index state WIP on upstream: dd5cce2 bump to 0.1.1
HEAD is now at dd5cce2 bump to 0.1.1
With that, I am greeted with:
➜  backbone-recipes git:(master) ✗ git scribe gen pdf
GENERATING PDF
GENERATING DOCBOOK

Making portrait pages on A4 paper (210mmx297mm)

[warning] /usr/bin/fop: Unable to locate servlet-api in /usr/share/java
log4j:WARN No appenders could be found for logger (org.apache.fop.util.ContentHandlerFactoryRegistry).
log4j:WARN Please initialize the log4j system properly.
That error is apparently known by some. Following this documentation, I edit /usr/bin/fop on my system so that the last line in there reads:
run_java -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.SimpleLog $HEADLESS org.apache.fop.cli.Main "$@"
Fixing that, I see:
➜  backbone-recipes git:(master) ✗ git scribe gen pdf
GENERATING PDF
GENERATING DOCBOOK

Making portrait pages on A4 paper (210mmx297mm)
Loading Xslthl configuration from file:///usr/share/xslthl/highlighters/xslthl-config.xml...
[warning] /usr/bin/fop: Unable to locate servlet-api in /usr/share/java
[ERROR] FOP - Exception javax.xml.transform.TransformerException: org.apache.fop.fo.ValidationException: file:/home/cstrom/repos/backbone-recipes/output/book.fo:1:108583: Error(1/108583): fo:page-sequence is not a valid child element of fo:flow.
        at org.apache.fop.cli.InputHandler.transformTo(InputHandler.java:217)
        at org.apache.fop.cli.InputHandler.renderTo(InputHandler.java:125)
        at org.apache.fop.cli.Main.startFOP(Main.java:166)
        at org.apache.fop.cli.Main.main(Main.java:197)
Caused by: javax.xml.transform.TransformerException: org.apache.fop.fo.ValidationException: file:/home/cstrom/repos/backbone-recipes/output/book.fo:1:108583: Error(1/108583): fo:page-sequence is not a valid child element of fo:flow.
        at org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:502)
        at org.apache.fop.cli.InputHandler.transformTo(InputHandler.java:214)
        ... 3 more
Caused by: org.apache.fop.fo.ValidationException: file:/home/cstrom/repos/backbone-recipes/output/book.fo:1:108583: Error(1/108583): fo:page-sequence is not a valid child element of fo:flow.
        at org.apache.fop.fo.FONode.invalidChildError(FONode.java:435)
        at org.apache.fop.fo.FONode.invalidChildError(FONode.java:420)
        at org.apache.fop.fo.pagination.Flow.validateChildNode(Flow.java:105)
...
Ugh. I think it is starting to come back to me. I believe that I ended up hand editing the fo output this fall. I don't think that is a viable long term solution, so I call it a night here to ruminate on next steps...

Update: I eventually track down the FO / page-sequence errors to the preface. If I eliminate the preface, I again get PDF with syntax highlighting:


But now I have two problems on my hands: the page size is again wrong and I cannot include preface chapters in my PDF anymore ugh. That is definitely something on which to ruminate.

Day #147

Monday, December 26, 2011

Tech PDFs should be 7.5" x 9"

‹prev | My Chain | next›

I have made a right old mess out of my fork of the git-scribe tool chain for producing ebooks. A mess it may be, but the resultant mobi is a bit better in that code blocks no longer split across pages and some Kindles better recognize the result as an ebook.

But really, I should do what I can to get back to upstream. Most of the work that I have in my fork is better epub/mobi out, but there is at least one PDF fix that I consider a show-stopper. Technical PDFs should be 7.5" x 9". Go on, check your PDFs from "real" publishers. That's how big they are. 8.5" x 11" is fine for printing on US-letter, but it is just too darn tall for computer reading.

Anyhow, I had got 7.5"x9" working back on my original fork fairly easily. Let's see how well that applies to recent upstream changes.
➜  git-scribe git:(master) git fetch upstream
➜  git-scribe git:(master) git co -b upstream -t upstream/master
Branch upstream set up to track remote branch master from upstream.
Switched to a new branch 'upstream'
The most obvious thing to try is cherry picking my old commit into this upstream branch:
➜  git-scribe git:(upstream) git cherry-pick b94110f
error: could not apply b94110f... Certain publishers prefer 7.5x9 inch paper size
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add ' or 'git rm '
hint: and commit the result with 'git commit'
The conflict is in the lib/generate.rb file responsible for, well, just about everything in an ebook generating library:
➜  git-scribe git:(upstream) ✗ gst
# On branch upstream
# Changes to be committed:
#
#       modified:   docbook-xsl/fo.xsl
#
# Unmerged paths:
#   (use "git add/rm <file>..." as appropriate to mark resolution)
#
#       both modified:      lib/git-scribe/generate.rb
#
More specifically, it looks as though the xsltproc command-line is missing the page-size options:
      java_options = {
        'callout.graphics' => 0,
        'navig.graphics'   => 0,
        'admon.textlabel'  => 1,
        'admon.graphics'   => 0,
      }
      run_xslt "-o #{local('book.fo')} #{local('book.xml')} #{base('docbook-xsl/fo.xsl')}", java_options
Back in my version, I had added the page-size:
strparams = {'callout.graphics' => 0,
                   'navig.graphics' => 0,
                   'admon.textlabel' => 1,
                   'admon.graphics' => 0,
                   'page.width' => '7.5in',
                   'page.height' => '9in'
      }
      param = strparams.map { |k, v| "--stringparam #{k} #{v}" }.join(' ')
      cmd = "xsltproc  --nonet #{param} --output #{local('book.fo')} #{base('docbook-xsl/fo.xsl')} #{local('book.xml')}"
      ex(cmd)
The easiest thing to try is to simply add those options to the new version:
java_options = {
        'callout.graphics' => 0,
        'navig.graphics'   => 0,
        'admon.textlabel'  => 1,
        'admon.graphics'   => 0,
        'page.width' => '7.5in',
        'page.height' => '9in'
      }
      run_xslt "-o #{local('book.fo')} #{local('book.xml')} #{base('docbook-xsl/fo.xsl')}", java_options
      ex "fop -fo #{local('book.fo')} -pdf #{local('book.pdf')}"
After re-building and re-installing the gem, I give it another try:
➜  backbone-recipes git:(master) ✗ git-scribe gen pdf
GENERATING PDF
GENERATING DOCBOOK
asciidoc: reading: /etc/asciidoc/asciidoc.conf
asciidoc: reading: /home/cstrom/.asciidoc/asciidoc.conf
asciidoc: reading: /etc/asciidoc/asciidoc.conf
asciidoc: reading: /home/cstrom/.asciidoc/asciidoc.conf
asciidoc: reading: /home/cstrom/repos/backbone-recipes/output/book.asc
asciidoc: reading: /etc/asciidoc/docbook45.conf
asciidoc: reading: /etc/asciidoc/filters/graphviz/graphviz-filter.conf
asciidoc: reading: /etc/asciidoc/filters/music/music-filter.conf
asciidoc: reading: /etc/asciidoc/filters/code/code-filter.conf
asciidoc: reading: /etc/asciidoc/filters/source/source-highlight-filter.conf
asciidoc: reading: /etc/asciidoc/filters/latex/latex-filter.conf
asciidoc: reading: /etc/asciidoc/lang-en.conf
asciidoc: writing: /home/cstrom/repos/backbone-recipes/output/book.xml
asciidoc: book.asc: line 7: reading: /home/cstrom/repos/backbone-recipes/output/history.asc
asciidoc: book.asc: line 16: reading: /home/cstrom/repos/backbone-recipes/output/introduction/namespacing.asc
Making portrait pages on USletter paper (7.5inx9in)
...
And it works! I could have sworn that I tried that a couple months back without success.

Just to be sure, I check the properties of the resultant PDF and, sure enough, it is 7.5" x 9":


I call it a night here. Up tomorrow, I need to see if I can get syntax highlighting working again with upstream. I had it working a few months ago, but am unable to find the magical combination of configuration options and installed packages to get it working tonight.

Update: Ugh. It turns out that I was re-installing my old version (0.0.9) of the gem rather than the new version (0.1.1). And so, page width and height are not, in fact, being honored.

Day #246

Sunday, December 25, 2011

Post-Mobi Clean-up in Git-Scribe

‹prev | My Chain | next›

Running the mobi generated by git-scribe through a Calibre command-line conversion now seems like a useful thing. The resultant mobi is an "EBOK" ebook (rather than a "PDOC" personal document) which works better on some Kindles. I believe that I have resolved any display issues from the resultant Calibre mobi, so there is no reason not to use it.

No reason not to switch to it save that Calibre's command line options do not include the "EBOK" setting. Rather, it needs to be configure via the GUI. That is a not a recipe for a successful pull request back to upstream.

I think, for now, that I will introduce a post-mobi build step to the mobi generation. In my fork of git-scribe, I alter the do_mobi to optionally execute a shell script if present in the current working directory:
   def do_mobi
      return true if @done['mobi']

      do_epub

      info "GENERATING MOBI"

      decorate_epub_for_mobi

      cmd = "kindlegen -verbose book_for_mobi.epub -o book.mobi"
      return false unless ex(cmd)

      cmd = @wd +  '/scripts/post-mobi.sh'
      if File.exists?(cmd) && File.executable?(cmd)
        return false unless ex(cmd)
      end

      @done['mobi'] = true
    end
I rebuild the gem and install it:
➜  git-scribe git:(master) ✗ gem build git-scribe.gemspec  
...
➜  git-scribe git:(master) ✗ gem install git-scribe-0.0.9.gem
Successfully installed git-scribe-0.0.9
1 gem installed
Then, in the Recipes with Backbone source, I create scripts/post-mobi.sh to include:
#!/bin/sh

echo "doing post mobi things..."
ebook-convert book.mobi book_ebok.mobi --chapter-mark=none --page-breaks-before='/'
echo "done!"
Now, when I run git scribe gen mobi, I see entirely too much output (from git-scribe, from the dependent kindlegen tool and from Calibre's ebook-convert, but in there, I do see my local script echo statements:
➜  backbone-recipes git:(master) ✗ git-scribe gen mobi
...
doing post mobi things...
Converting input to HTML...
InputFormatPlugin: MOBI Input running
on /home/cstrom/repos/backbone-recipes/output/book.mobi
Parsing all content...
Forcing Recipes_with_Backbone.html into XHTML namespace
34% Running transforms on ebook... 
Merging user specified metadata... 
Detecting structure...
        Detected chapter: Chapter 1. Namespacing
Flattening CSS and remapping font sizes...
Source base font size is 12.00000pt
Removing fake margins...
Cleaning up manifest...
Trimming unused files from manifest...
Trimming 'images/00002.jpg' from manifest
Creating MOBI Output...
67% Creating MOBI Output
Generating in-line TOC...
Applying case-transforming CSS...  
Rasterizing SVG images...
Converting XHTML to Mobipocket markup...
Serializing markup content...
  Compressing markup content...
Generating flat CTOC ...
  CNCX utilization: 1 record, 0% full
Indexing navPoints ...
Generating INDX ...
Serializing images...
MOBI output written to /home/cstrom/repos/backbone-recipes/output/book_ebok.mobi
Output saved to   /home/cstrom/repos/backbone-recipes/output/book_ebok.mobi
done!
That will do to clean up my personal toolchain. Unfortunately, I am getting further and further away from the upstream version of git-scribe. I think tomorrow I shall have a look to see what it would take to get me back on track.

Day #245

Saturday, December 24, 2011

Too Many Page Breaks in Calibre Mobi

‹prev | My Chain | next›

As of yesterday, I am strongly considering adding Calibre to my already hacked up version of the git-scribe toolchain for producing ebooks. This seems like a good idea so that I can get "Book" support on the Kindle Fire. Calibre is able to add a Mobi header (cdetype=EBOK) that the Fire recognizes as a book rather than a "personal document".

A quick glance through the resulting book on the Fire, however, reveals a show-stopper for me: there are page breaks before every heading in the book. In Recipes with Backbone, this means that our 1-2 sentence introductions now appear on a nearly blank page. The mobi generated by git-scribe does not suffer from this problem, so it must be Calibre that is adding the page breaks.

I try reducing the page breaks from h1 and h2 tags on the command line:
ebook-convert book.mobi book_ebok.mobi --chapter-mark=none --page-breaks-before="//*[name()='h1']"
but this seems to have no effect.

So I take a look at the various mobis in the Calibre display tool. For both the git-scribe and the Calibre versions of the document, the display in the viewer looks good:


Taking a look at the git-scribe DOM (by inspecting the element in Calibre's viewer), I see just the one manual page break (mpb_pagebreak):


By contrast, the Calibre DOM has manual page breaks before each sub-heading in the book:


It is pretty clear from the git-scribe DOM that the problem is all of the headings are h1 tags—even the sub-headings inside the chapters. Rather than dig through the toolchain to see where this is occurring (likely somewhere in asciidoc), I change the command line option to insert no page breaks:
ebook-convert book.mobi book_ebok.mobi --chapter-mark=none --page-breaks-before='/'
With that, my DOM only retains the original manual page break before the chapter start and no in-chapter breaks:


Hopefully that does it for the Calibre clean-up. I will root through Recipes with Backbone a little more today. If all looks good, I will add a post-mobi.sh script to my git-scribe tool chain tomorrow.


Day #144

Friday, December 23, 2011

Command Line Calibre

‹prev | My Chain | next›

I spend a good chunk of time today researching. I had hoped to find an easy solution that would allow me to create and modify the "exth" header in mobi files. If I can just do that, then I can set the "cdetype" in that header to "EBOK", which would signal to Kindles that books like Recipes with Backbone are ebooks and not "personal documents" ("PDOC").

Last night I was able to get this working with the GUI tool Calibre. The problem is that it is a GUI tool. A big appeal of git-scribe is that automates so much of the ebook chain. In my fork of git-scribe, I have it to the point that I can generate PDF, epub and mobi versions of a book and zip them all into a single file with just one command. I do not relish adding a manual, GUI step to that process.

The best I can come up with for command-line solutions is mobiperl. It seems like it would do the trick -- it was the inspiration for Calibre's EBOK support. But I am not going to pursue that. There are already an alarming number of dependencies in git-scribe, I do not think adding perl and all of mobiperl's requirements is going to help.

In ruby-land (git-scribe is written in ruby), there is the mobi gem, but that is for reading mobi, not writing. And it does not seem to be exth-aware. I could try writing my own, but that just feels like too much of a rabbit hole.

So it seems as though I am stuck with Calibre. It turns out that there is a set of command-line tools that are installed along with the GUI. One in particular, ebook-meta seems like it could be of some use, except... it does not support updating "cdetype" field in a mobi.

Another command-line tool installed along with Calibre is ebook-convert. It does not support editing the cdetype either, but I have already used the GUI to always generate EBOK output.

So I give the command a try, converting the mobi version of Recipes with Backbone to... mobi:
➜  output git:(master) ebook-convert book.mobi book_ebok.mobi
1% Converting input to HTML...
InputFormatPlugin: MOBI Input running
on /home/cstrom/repos/backbone-recipes/output/book.mobi
Parsing all content...
Forcing Recipes_with_Backbone.html into XHTML namespace
34% Running transforms on ebook...
Merging user specified metadata...
Detecting structure...
        Detected chapter: 1. Who Should Read this Book
        Detected chapter: 3. How this Book is Organized
        Detected chapter: Chapter 1. Writing Client Side Apps (Without Backb
        Detected chapter: Chapter 2. Writing Backbone Applications
        Detected chapter: Chapter 3. Namespacing
        Detected chapter: Chapter 4. View Templates with Underscore.js
        Detected chapter: Chapter 5. Instantiated View
        Detected chapter: Chapter 6. Collection View
        Detected chapter: Chapter 7. View Signature
        Detected chapter: Chapter 8. Fill-In Rendering
        Detected chapter: Chapter 9. Actions and Animations
        Detected chapter: Chapter 10. Reduced Models and Collections
        Detected chapter: Chapter 11. Non-REST Models
        Detected chapter: Chapter 12. Changes Feed
        Detected chapter: Chapter 13. Pagination and Search
        Detected chapter: Chapter 14. Constructor Route
        Detected chapter: Chapter 15. Router Redirection
        Detected chapter: Chapter 16. Evented Routers
        Detected chapter: Chapter 17. Object References in Backbone
        Detected chapter: Chapter 18. Custom Events
Flattening CSS and remapping font sizes...
Source base font size is 12.00000pt
Removing fake margins...
Cleaning up manifest...
Trimming unused files from manifest...
Trimming 'images/00007.jpg' from manifest
Creating MOBI Output...             
67% Creating MOBI Output
Generating in-line TOC...
Applying case-transforming CSS...
Rasterizing SVG images...
Converting XHTML to Mobipocket markup...
Serializing markup content...
  Compressing markup content...
Generating flat CTOC ...
  CNCX utilization: 1 record, 0% full
Indexing navPoints ...
Generating INDX ...
Serializing images...
MOBI output written to /home/cstrom/repos/backbone-recipes/output/book_ebok.mobi
Output saved to   /home/cstrom/repos/backbone-recipes/output/book_ebok.mobi
Hrm... it really seems to do quite a bit of changing and re-arranging of things. I had expected, based on a cursory reading of the mobi-to-mobi options for ebook-convert that it would do less.

Still, it does the job. When I copy the generated book back to my Kindle Fire, it shows up under the "Books" section rather than the "Documents" section.

Hrm... perhaps I can add an optional "clean-up" step to git-scribe that can invoke an arbitrary command-line script. I cannot explicitly include Calibre since it requires the GUI configuration from last night. But first, I think, I need to compare the output from ebook-convert with the original git-scribe mobi. I worked hard three months ago to get it just right. I will pick back up with that tomorrow.


Day #243

Thursday, December 22, 2011

Kindle Fire Book vs. Document

‹prev | My Chain | next›

One of the hardest things to get right with ebook generation is the on-device experience. In part, this is because it is so time-consuming to try out even small changes. I have no idea how to speed up the process of generating the book, mounting an associated device, copying the stinking thing over to the device and then unmounting the device so that the device is back in read mode.

To be sure, sending the book to the device wirelessly is generally a better bet, but then you are at the mercy of network issues. You are never quite sure if the copy on the device is the one you just sent or the one that seemed to get lost 30 minutes ago.

I bring all of this up, because we received a report from Derick Bailey that copying a the mobi version of Recipes with Backbone onto the Kindle Fire fails to register the book as an actual book on the device. Rather it shows up as a "document".

So I regenerate the mobi version of the book:
➜  backbone-recipes git:(master) git-scribe gen mobi
GENERATING EPUB
GENERATING HTML
...
Info(prcgen): The file format version is V6
Info(prcgen): Saving MOBI file
Info(prcgen): MOBI File generated with WARNINGS!
Then I copy the output over to my Fire:
➜  backbone-recipes git:(master) cp output/book.mobi /media/KINDLE/Books 
➜  backbone-recipes git:(master) umount /media/KINDLE 
And sure enough, the book does not show up under "Books" in the Fire, but under "Documents".

But, you know what? So do all of the PragProg books that have been getting sent to the Fire via my @free.kindle.com account. So, at least I am in good company.

Even so, it would be nice if I could get that working.

Derick was kind enough to provide a pointer to get started. In that thread, it is suggested that Calibre might be able to solve the problem if I convert my output to "EBOK" (instead of "PDOC"). So, under preferences / output, I change the output from "[PDOC]" to "[EBOK]":


Then, I load the mobi version of Recipes with Backbone into Calibre, and convert it to... mobi. Again, I copy the book over to the Fire:
➜  backbone-recipes git:(master) cp ~/Calibre\ Library/Nick\ Gauthier/Recipes\ with\ Backbone\ \(30\)/Recipes\ with\ Backbone\ -\ Nick\ Gauthier.mobi /media/KINDLE/Books/book.mobi
➜  backbone-recipes git:(master) umount /media/KINDLE   
And, what do you know? It worked. Recipes with Backbone now shows up under Books on the Fire.

But needing to run the book through a manual process like Calibre is less than ideal.

Unfortunately, the kindlegen tool provided by Amazon does not support setting this value. This looks to be a binary field somewhere in the mobi, so I am hesitant even get into trying to figure this out manually. And to make matters worse, I cannot reverse engineer the mobi because the generated mobi is not unzip-able:
➜  rwb_mobi  unzip book.mobi  
Archive:  book.mobi
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of book.mobi or
        book.mobi.zip, and cannot find book.mobi.ZIP, period.
If I do nothing, I am no worse off than other ebook vendors. Still, it would be cool to solve this. I will have to ruminate some...

Day #242

Wednesday, December 21, 2011

Git-Scribe Syntax Highlighting

‹prev | My Chain | next›

Yesterday I was able to cherry pick a commit from the upstream git-scribe into my fork of git-scribe that prevented code blocks from splitting across pages in the generated PDF. An even more exciting development in upstream is syntax highlight in code blocks—something I was never able to get working on my own.

Ultimately, I should try to get back on upstream, but I had invested quite a bit of time in getting the epub and mobi formatting working nicely in my fork (I did submit pull requests) before upstream went a different direction. Also, when I had tried out upstream after those changes went it, I was no longer able to control the PDF page size (I am use 7"x9"). It is going to be hard to get that rebased onto upstream if I cannot get my local a little closer.

Anyhow, I am pretty sure that I could not get highlighting working when I cherry picked the commit previously, but let's try it out to be sure....
➜  git-scribe git:(master) git cherry-pick cddb437d27e01c4fda7ae5a51bb47cdd57595acb
error: could not apply cddb437... Add syntax highlighting for PDFs
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'
Checking the conflicts, I see:
➜  git-scribe git:(master) ✗ gst
# On branch master
# Changes to be committed:
#
#       modified:   docbook-xsl/fo/highlight.xsl
#       modified:   test/gen_test.rb
#       new file:   vendor/saxon.jar
#       new file:   vendor/xslthl-2.0.2.jar
#
# Unmerged paths:
#   (use "git add/rm ..." as appropriate to mark resolution)
#
#       both modified:      .gitignore
#       both modified:      docbook-xsl/fo.xsl
#       both modified:      lib/git-scribe/generate.rb
Ah, wait a second. Now it's all coming back to me. Upstream is now using saxon (java) to convert so that it has access to the libxslthl library which is written in java. Ugh.

Before I go down that path, I would like to see if alternatives might work. To the best of my knowledge, this means pygmentize. Rather than fiddle with it directly, I give the python script in this article a try (copying the html.xslt code listing from that same article):
➜  output git:(master) docbook_build.py --html html.xslt book.xml
Note: namesp. add : added namespace before processing              Recipes with Backbone
Writing /home/cstrom/repos/backbone-recipes/output/pr01.html for preface(_history)
Writing /home/cstrom/repos/backbone-recipes/output/pr02s02.html for section(_contact_us)
Writing /home/cstrom/repos/backbone-recipes/output/pr02s03.html for section(_how_this_book_is_organized)
Writing /home/cstrom/repos/backbone-recipes/output/pr02.html for preface(_introduction)
Writing /home/cstrom/repos/backbone-recipes/output/ch01.html for chapter(chapter_non_backbone)
....
Writing /home/cstrom/repos/backbone-recipes/output/index.html for book
When I take a look at the output, I find:


Aw, dang, that's really no improvement over:


To be sure the pygmentize formatting is a little prettier, but the same things are getting highlighted: this, keywords (return, function), and strings. None of the class names are highlighted. None of the method names in either is highlighted. So there really is no benefit to sticking with the old xsltproc-based way.

So it looks as though I am going to have to follow upstream down the saxon path. I will get started with that tomorrow, but, instead of working with the syntax highlighting, I am going to see if I can solve the page size first. 8.5in x 11in just won't cut it for me.

Day #241

Tuesday, December 20, 2011

Code Blocks in Git-Scribe

‹prev | My Chain | next›

With my last remaining PhantomJS questions answered last night, I think I am ready to bid a fond adieu to Backbone.js. I will still have need of the occasional follow-up post, but the regular material needed to support Recipes with Backbone is complete. And so, I turn my attention back to my toolchain for generating ebooks, git-scribe.

Between the time that I wrote The SPDY Book and Recipes with Backbone, there was quite a bit of activity on git-scribe. Much of it was fairly exciting. One of the features added was one with I had wrestled unsuccessfully: code-highlighting.

The thing about highlighting is that not all highlighters are created equal. The code in our book ended up looking like:


In Emacs, by contrast, methods properties and top-level class names are highlighted:


That coupled with an unsatisfactory control over the page size led my co-author, Nick Gauthier, and I to stick with my my fork of git-scribe. I hope to take a few days now ideally to switch back over to the mainstream git-scribe. Failing that, I hope to at least get a little closer.

Tonight, I start with the getting closer. One of the annoyances in my git-scribe fork is that code sample can span pages:


That just sucks. Fortunately there's a commit for that. A commit, that is, in the upstream version of git-scribe. I am already tracking upstream, so I can just cherry-pick that into my local copy:
➜  git-scribe git:(master)  git cherry-pick 8e35785
Now, to test that out. It is not exactly the kind of thing that I can BDD since the desired output is entirely visual. So I start with the bit of code that was not wrapping previously. I delete enough lines so that it is just at the bottom of one page:


Then I add just enough text to add one line of text above that code sample and re-check the output:


Yay! The one line that would have otherwise pushed a single line of a closing parenthesis onto the next page now sends the entire code block onto the next page.

That is a small victory, but I'll take it. If nothing else, it is a good way to get me back in the swing of git-scribe coding and I can definitely make use of that immediately to improve Recipes with Backbone.


Day #240

Monday, December 19, 2011

Exit Codes from PhantomJS Jasmine Runs

‹prev | My Chain | next›

Last night, I was able to get some pretty OK output from my jasmine test suite when run with PhantomJS. I had to use the jasmine server provided by the jasmine ruby gem to get my require.js structured Backbone.js application running under test, but it was worth it if it makes testing easy on a continuous integration server.

The thing about continuous integration servers is that they like proper exit codes from builds. A non-zero exit code indicates a failure on Unix systems, but the run-jasmine.js script that is supplied in the PhantomJS examples always returns 0:
        waitFor(
          function(){
            return page.evaluate(function(){
              // Run the Jasmine suite on the page
            });
          }, 
          function(){
            page.evaluate(function(){ 
              // Inspect the results
            });
            phantom.exit();
          }
        );
The problem is that, inside the page.evaluate the anonymous function has no access to PhantomJS variables. So something like this will not work:
        var passed;
        waitFor(
          function(){ // Run the Jasmine suite on the page }, 
          function(){
            page.evaluate(function(){ 
              // Inspect the results

              passed = jasmineEnv.currentRunner().results().passed();
            });
            phantom.exit(passed);
          }
        );
The value of passed at the very end will always be undefined.

I could return a value from page.evaluate, but the enclosing waitFor() does not return a value.

Without any better solution, I try to write to the file system:
page.evaluate(function(){
  var fs = require('fs');
  // ...
})
But even that does not work for me:
Error: Module name 'fs' has not been loaded yet for context: _
http://requirejs.org/docs/errors.html#notloaded
Unfortunately, I seem to be stuck here.

Update: Figured it out. The indentation in my Javascript was confusing me. It turns out that I can use the return value from page.evaluate():
        waitFor(
          function(){ // Run the Jasmine suite on the page }, 
          function(){
            var passed = page.evaluate(function(){ 
              // Inspect the results

              return jasmineEnv.currentRunner().results().passed();
            });
            phantom.exit(passed ? 0 : 1);
          }
        );
Now, if I run the test suite with an and-and echo, I do not see the and-and echo when the suite is failing:
➜  calendar git:(requirejs-jasmine-gem) ✗ phantomjs run-jasmine.js http://localhost:8888 && echo "\n\n    PASSED\!\!\!\!\n\n"
...
'waitFor()' finished in 980ms.

Failed: 1

Calendar
  the initial view
    the page title
      Expected '<h1>Funky Calendar<span> (2011-12) </span></h1>' to have text 'Remember, Remember, the Fifth of Novemeber'.
But, when I fix that spec and re-run the suite, I do see the and-and echo output:
➜  calendar git:(requirejs-jasmine-gem) ✗ phantomjs run-jasmine.js http://localhost:8888 && echo "\n\n    PASSED\!\!\!\!\n\n"
...
'waitFor()' finished in 1138ms.

Succeeded.




    PASSED!!!!

Yay!


Day #239

Sunday, December 18, 2011

Somewhat Pretty Printing with PhantomJS, Jasmine and Backbone.js

‹prev | My Chain | next›

I now have the jasmine specs covering my Backbone.js application running under PhantomJS. Running, but the failure leaves something to be desired. The exit code is a success and the failure message simply parrots the entire suite with no indication of which specs failed.

I know that there are terminal reporters, a la the node-jasmine. They do not seem to be a quick drop-in replacement in my little PhantomJS setup. Besides, I hope that I can hack together a little something that will be good enough™.

I still have my intentionally failing spec in place:


Back in the run.html.erb page runner for the jasmine gem, I make the jasmineEnv variable global so that I might access it from within PhantomJS:
require(['Calendar', 'backbone'], function(Calendar, Backbone){
  window.Cal = Calendar;
  window.Backbone = Backbone;

  window.jasmineEnv = jasmine.getEnv();
  jasmineEnv.updateInterval = 1000;

  var reporter = new jasmine.TrivialReporter();

  jasmineEnv.addReporter(reporter);

  jasmineEnv.specFilter = function(spec) {
    return reporter.specFilter(spec);
  };

  jasmineEnv.execute();
});
Then, in the PhantomJS contributed run-jasmine.js, I log the number of failed specs found in the current spec runner:
  waitFor(
    function(){ /* finished at to display */ }, 
    function(){
      page.evaluate(function(){
        console.log('Failed: ' + jasmineEnv.currentRunner().results().failedCount);
        // ...
      });
  });
(I uncover that chain by fiddling with things in Chrome's Javascript console)

With that logger in place, when I run my intentionally failing specs, I see:
# lots of normal application console.log output ...

[object Arguments]
'waitFor()' finished in 1164ms.
Failed: 1

# The text for all specs in the suite...
Looking more closely at the PhantomJS run-jasmine.js script, I notice that it is quite shallow:
  waitFor(
    function(){ /* finished at to display */ }, 
    function(){
      page.evaluate(function(){
        console.log('Failed: ' + jasmineEnv.currentRunner().results().failedCount);
        list = document.body.querySelectorAll('div.jasmine_reporter > div.suite.failed');
        for (i = 0; i < list.length; ++i) { /* ... */ }
      });
  });
By "shallow", I mean that the script is simply grabbing the top-level failed spec and then logging the inner text of each immediate-child:
           list = document.body.querySelectorAll('div.jasmine_reporter > div.suite.failed');
           for (i = 0; i < list.length; ++i) {
               el = list[i];
               desc = el.querySelectorAll('.description');
               console.log('');
               for (j = 0; j < desc.length; ++j) {
                   console.log(desc[j].innerText);
               }
           }
As can be seen from the screen shot of my failing spec, I am nest things quite a bit. What this translates into is a bunch of nested spec groups, each of which have some failing and some passing specs:


To paraphrase the immortal philospher of movies, I know this... it's recursion! So I create a log_failure recursive function to log the specs properly:
  waitFor(
    function(){ /* finished at to display */ }, 
    function(){
      page.evaluate(function(){
        console.log('Failed: ' + jasmineEnv.currentRunner().results().failedCount);

        function has_message(el) { /* children className == spec failed */ }
        function failed_children(el) { /* all children w/ className == 'suite failed' */ }

        function log_failure(failure_el, indent) {
          if (typeof(indent) == 'undefined') indent = '';

          console.log(indent + failure_el.querySelector('.description').innerText);

          if (has_message(failure_el)) {
            console.log(indent + '  ' + failure_el.querySelector('.messages > .fail').innerText);
          }
          else {
            failed_children(failure_el).forEach(function (failed_child) {
              log_failure(failed_child, indent + '  ');
            });
          }

          log_failure(document.body.querySelectorAll('div.jasmine_reporter > div.suite.failed')[0]);
        }
      })
    });
That is kinda ugly because I had to work with HTML Collections instead of arrays. Still, it seems to work because the output produces:
'waitFor()' finished in 1117ms.
Failed: 1
Calendar
  the initial view
    the page title
      Expected '<h1>Funky Calendar<span> (2011-12) </span></h1>' to have text 'Remember, Remember the Fifth of Novemeber'.
I think I can live with that.



Day #238

Saturday, December 17, 2011

Phantom.js and Backbone.js (and require.js)

‹prev | My Chain | next›

Yesterday I was able to make use of my mad ruby hacker skills to get the jasmine server to run the jasmine specs for my backbone.js and require.js application. Today, I hope to use my lovely specs:


The hope is to get them running under PhantomJS—a headless webkit and Javascript engine. First I need to install the thing. For that, I follow the build instructions for Ubuntu. I already have the QT dependencies installed, so I download the source code and compile:
➜  Downloads  wget http://phantomjs.googlecode.com/files/phantomjs-1.3.0-source.tar.gz
...
2011-12-17 23:00:29 (144 KB/s) - `phantomjs-1.3.0-source.tar.gz' saved [409428/409428]

➜  Downloads  sha1sum phantomjs-1.3.0-source.tar.gz 
76902ad0956cf212cc9bb845f290690f53eca576  phantomjs-1.3.0-source.tar.gz


➜  src  tar zxf ../Downloads/phantomjs-1.3.0-source.tar.gz
➜  src  cd phantomjs-1.3.0

➜  phantomjs-1.3.0  qmake-qt4 && make
cd src/ && /usr/bin/qmake-qt4 /home/cstrom/src/phantomjs-1.3.0/src/phantomjs.pro -o Makefile.phantomjs
cd src/ && make -f Makefile.phantomjs
make[1]: Entering directory `/home/cstrom/src/phantomjs-1.3.0/src'
...
make[1]: Leaving directory `/home/cstrom/src/phantomjs-1.3.0/src'

➜  phantomjs-1.3.0  ls -l bin
total 324
-rwxrwxr-x 1 cstrom cstrom 330534 2011-12-17 23:05 phantomjs
I copy that phantomjs executable into my bin directory (which is in my $PATH):
➜  phantomjs-1.3.0  cp bin/phantomjs ~/bin/
I am not quite ready just yet. In the same source code, PhantomJS includes a script for running jasmine tests. That sounds like exactly what I want, I copy it into my applications source directory:
➜  phantomjs-1.3.0  cp examples/run-jasmine.js ~/repos/calendar
With that, I am ready to give PhantomJS and the jasmine server a try. First, I spin up the jasmine server:
➜  calendar git:(master) ✗ rake jasmine
your tests are here:
  http://localhost:8888/
In a separate terminal, I then point PhantomJS at the run-jasmine.js script and my running gem server:
➜  calendar git:(requirejs-jasmine-gem) ✗ phantomjs run-jasmine.js http://localhost:8888
[setDefault]
[setMonth] %s
Error: Backbone.history has already been started
[setMonth] %s
...
[setDefault]
[setMonth] %s
editClick
Error: Backbone.history has already been started
[setMonth] %s
[fetch] Dang
[object Arguments]
'waitFor()' finished in 1253ms.
Hrm... I guess that worked OK. The exit status was cool:
➜  calendar git:(requirejs-jasmine-gem) ✗ echo $?                                       
0
All of the output was just my various calls to console.log(). Really, I should remove them because they will break older browsers like IE. It is interesting that PhantomJS does not seem to honor the sprintf format of console.log:
define(function(require) {
  var Backbone = require('backbone')
    , to_iso8601 = require('Calendar/Helpers.to_iso8601');

  return Backbone.Router.extend({
    // ....
    setMonth: function(date) {
      console.log("[setMonth] %s", date);
      this.application.setDate(date);
    }
  });
});
The "[fetch] Dang" error seems legit. It is not causing anything to fail, but I seem to have missed a sinon.js stub somewhere in my tests.

All of the Backbone.history messages are a minor headache incurred when testing Backbone.js applications with jasmine.

So it all seems to be working. To be sure, I intentionally break a test:
    describe("the page title", function() {
      it("contains the current month", function() {
        expect($('h1')).toHaveText("Remember, Remember the Fifth of Novemeber");
      });
    });
In the browser, this failure looks like:


In PhantomJS, it looks like:
➜  calendar git:(requirejs-jasmine-gem) ✗ phantomjs run-jasmine.js http://localhost:8888
...
'waitFor()' finished in 1044ms.


Calendar
routing
defaults to the current month
sets the date of the appointment collection
the initial view
the page title
contains the current month
a collection of appointments
populates the calendar with appointments
adding an appointment
sends clicks on day to an add dialog
displays the date clicked in the add dialog
adds a new appointment to the UI when saved
deleting an appointment
clicking the "X" removes records from the data store and UI
updating an appointment
binds click events on the appointment to an edit dialog
displays model updates
can edit appointments through an edit dialog
navigated view
I guess that failed. The output was different. Even so that exist code remained the same:
➜  calendar git:(requirejs-jasmine-gem) ✗ echo $?
0
Hrm... that output is nothing more than the entire text of the test suite. Hopefully fixing that is just a simple matter of futzing with the run-jasmine.js script. I think I will leave that for tomorrow.

For now, I am happy with the progress I made tonight. I can run my Jasmine test suite under the Jasmine server (standalone still works as well) and under PhantomJS. The output from PhantomJS leaves a little to be desired, but it is still workable in a continuous integration environment. Hopefully, by tomorrow, I can make it even better.


Day #237

Friday, December 16, 2011

Jasmine Server and Backbone.js + Require.js

‹prev | My Chain | next›

I can't quite tear myself away from Backbone.js (and really, why would I want to?). So tonight I am going to try to get a Backbone application built with require.js to run under the jasmine server (part of the jasmine ruby gem). Ultimately, I hope to parlay this into something that can be run under PhantomJS, but first things first.

The main problem with the jasmine server, with respect to require.js code, is that the jasmine server explicitly slurps in a bunch of Javascript files. It does this by checking the spec/javascripts/support/jasmine.yml file for the src_files entry. Back before require.js, that entry looked like:
src_files:
    - public/scripts/jquery.min.js
    - public/scripts/jquery-ui.min.js
    - public/scripts/underscore.js
    - public/scripts/backbone.js
    - public/scripts/**/*.js
Inserting those into the server's HTML is just going to cause require.js related trouble, so I empty out that entry.

But how do I get require.js into the server spec runner and, more importantly, how do I also add the data-main attribute? For now, I manually copy the run.html.erb file from the Ruby gem into my application's home directory. In there, I can manually add the require.js <script> tag:
<script data-main="/public/scripts/main"  src="/public/scripts/require.js"></script>
I was already able to get standalone jasmine to work with require.js the other night, so I can copy the same configuration into run.html.erb now:
<script data-main="/public/scripts/main"  src="/public/scripts/require.js"></script>

<script type="text/javascript">
require.config({
  baseUrl: '../public/scripts',
  paths: {
    'jquery': 'jquery.min',
    'jquery-ui': 'jquery-ui.min'
  }
});

require(['Calendar', 'backbone'], function(Calendar, Backbone){
  window.Cal = Calendar;
  window.Backbone = Backbone;

  var jasmineEnv = jasmine.getEnv();
  jasmineEnv.updateInterval = 1000;

  var trivialReporter = new jasmine.TrivialReporter();

  jasmineEnv.addReporter(trivialReporter);

  jasmineEnv.specFilter = function(spec) {
    return trivialReporter.specFilter(spec);
  };

  jasmineEnv.execute();
});
</script>
That is a combination of the usual Jasmine initialization and require.js configuration.

At this point, I have my spec runner ready to go, but the jasmine gem's server is still going to look for run.html.erb inside the library instead of using my new require.js copy. To get around that, I edit the Rakefile, to pull in a specialized jasmine server:
begin
  require 'jasmine'
  require './jasmine_server'
  load 'jasmine/tasks/jasmine.rake'
rescue LoadError
  task :jasmine do
    abort "Jasmine is not available. In order to run jasmine, you must: (sudo) gem install jasmine"
  end
end
In the jasmine_server.rb file, I make use of Ruby's open classes to modify the server's run method:
module Jasmine
  class RunAdapter
    def run(focused_suite = nil)
      jasmine_files = @jasmine_files
      css_files = @jasmine_stylesheets + (@config.css_files || [])
      js_files = @config.js_files(focused_suite)
      body = ERB.new(File.read(File.join(File.dirname(__FILE__), "run.html.erb"))).result(binding)
      [
        200,
        { 'Content-Type' => 'text/html' },
        [body]
      ]
    end
  end
end
With that, I have my require.js jasmine server running my specs. Tomorrow, I will go through the exercise of fixing them:


The switch to the node-dirty data store broke those—not the switch to the jasmine server. For now, this is a good stopping point.

Day #236