My name is Caspar Krieger.

I'm a developer at Atlassian, living in Sydney, Australia.

In my free time, I experiment with game and web development.

You can find me on Twitter, BitBucket, GitHub, Stack Overflow and LinkedIn. You can also contact me via email addressed to my first name at this domain.

D3 Impressions

I used D3 to do a few graphs and animations for a recent company hackathon, and I thought it might be worthwhile to jot down some notes on my impressions.

Why D3?

The rough problem we were solving was visualizing a tree-based data structure, with a centered root node and child nodes splaying out to the left and right (depending on the type of child) for several different levels.

I've messed around with HTML5 Canvas elements before, but I wanted the tree to be interactive. I wanted to be able to click on nodes of the tree and have the tree be recentered so that the clicked-on node becomes the root node (which might require some new data to be loaded).

That seemed like a pain to do with Canvas, because state information like "there's a circle rendered here" is discarded as soon as the drawing operation is completed. I'd have to manually keep track of where nodes were and translate "a click at a given (x, y)" to "a click on node z".

On the other hand, I knew just enough about rendering SVG in the browser to realize that it was probably a better fit for this scenario. SVG elements are DOM elements, which lets us sidestep the process of mapping xy-click coordinates to tree nodes and rely on the information contained in the click event instead.

But, having never worked with SVG before (except for messing about in Inkscape drawing logos), I thought it best to "go with the flow" and explore it in the context of a popular library.

What's the point of D3?

Depending on your web front-end (& SVG) experience, D3 as shown by tutorials seems either arcane, or completely pointless.

     .attr("width", 50)
     .attr("height", 50)
     .attr("cx", 25)
     .attr("cy", 25)
     .attr("r", 25)
     .style("fill", "purple");

(try this example)

Arcane because it magically knows about body, svg & circle (does it also know about dotted-squigly-line?) and the syntax for allowable attributes vs styles doesn't seem to be documented anywhere in D3. In reality, 'circle' is just the name of the svg element, much like p or div, attr() sets whatever attribute you want, and style() sets whatever css property you want.

Pointless because, well, if that's all it's doing, can't we do the exact same thing with jQuery?

     .attr("width", 50)
     .attr("height", 50).append(
         .attr("cx", 25)
         .attr("cy", 25)
         .attr("r", 25)
         .css("fill", "purple")));

A little uglier, but close enough? If you try this jQuery example you'll see that although the DOM nodes get added, there's no purple circle shown. What the?

As pointed out by answers on trusty StackOverflow, the reason behind this is that SVG and HTML elements belong to different namespaces; you can create an SVG element using jQuery, but it ends up in the wrong namespace (see also MDN's Namespaces Crash Course. Anyway, some quick hackery and we're back:


Now we can observe our purple circle in all its glory, but woops! We just lost events we had bound to any SVG elements because that snippet recreates them from scratch.

If all you want to do is draw a non-interactive SVG animation (or you have masochistic tendencies) and don't want to drag in another dependency, then use jQuery.

So D3 is a jQuery that works for SVG?

Yes, but the more interesting thing is how D3 helps you deal with changing data. Most tutorials do an average job of explaining this, so I'll have a swing at explaining it in the way that I wish I could have had it explained to me.

D3, Data, & Binding

You have an array of Javascript objects, and you want to use this data to affect some set of DOM nodes, such as a bunch of circle elements. So, first you use a select() to do CSS selection (just like jQuery), and then you bind your data array to the selection using data().

var circleSelection = d3.select("svg").selectAll("circle").data([10, 20, 40]);

Each time you bind data like this, you can access 3 possible "stages" (D3 terminology), which are selections themselves:

  • enter() - you have more data than elements you selected. You probably want to append() a new element for each of your excess data points, and then use the relevant data point to do something: circleSelection.enter().append("circle").attr("cy", 30).attr("r", 5).attr("cx", function(d) { return d; }); (here we use the data point as the x position)
  • update() - you have some data for which there is already a matching element. You probably want to use the new data to change the data node; in our case we want to update the x position again: circleSelection.update().attr("cx", function(d) { return d; }); (note that unlike in enter(), there's no need to re-set cy or r because those will already have been set for the elements when they are created)
  • exit() - you have less data than elements you selected. You probably want to make those elements disappear: circleSelection.exit().remove(); (remove() is a D3 function to disappear selected elements)

That's all you need to know to get started (+ the MDN SVG docs).

D3 features you should probably know about

D3 includes a bunch of other functionality; skimming through this list will avoid you reinventing the wheel a few times:

  • Want fancy animations? Call transition() on a selection before doing something like calling attr() or style() on it (By the way, D3 calls these operators).
  • Loading data from the server? You've got d3.json() and more.

  • Want to scale your values so your elements don't go off the edge of your SVG element? D3 has Scales.

  • Drawing curvy lines? D3 has Path data generators
  • Trying to position nodes in a tree layout, or make a histogram, or a pie chart? D3 has layouts, which can do useful things like augment your data array objects with x & y coordinates.
  • Text overlapping? D3 has some support for zooming.

These things were useful to me, but it's worth skimming the API docs yourself.

What about that hackathon project?

Success! We implemented a more advanced version of this tree.

We won a minor prize too, but the jury's out on whether it was the utility or the animations which got us that.

Either way, it's a vote of confidence in D3 ;)

Functional Programming in Scala

Posted Sun 03 November 2013

Tagged: Scala

Functional Programming in Scala is a Coursera course taught by Martin Odersky. Odersky contributed substantially to Java's generics and built the current incarnation of the Java compiler; you could say he knows his Javas. It seems that was too easy for him, so he created Scala.

My experience with Massively Open Online Courses ("MOOC"s) is that they're slow and highly theoretical, with just the odd quiz thrown in to keep you from falling asleep. I was hesitant to sign up for it.

I'm glad I did, because I ended up learning a great deal.

The course structure is about 1-2 hours of lectures (depending on whether you watch at 2x speed or not - I recommend it), followed by a homework assignment.

The assignments are almost always completely solvable using only the techniques covered in the lectures so far. They're great for forcing you to put the theory into practice, but there's a lot of variance in how much time it takes to finish them, because they often rely on "aha" moments. On the other hand, that's part of what makes them so valuable.

That Functional Programming Thing

I work in Java fulltime, so I've been exposed to a tiny amount of programming in functional style through Google's excellent Guava library. However, Java's lack of lambdas / anonymous functions left some things to be desired:

List<String> myList = ImmutableList.of("dog", "cat", "dingo", "leopard");
List<String> dAnimals = Collections2.filter(myList, new Predicate<String>() {}
    public boolean apply(String animalName) {
        return animalName.startsWith("d");

It's a little hard to appreciate functional programming when it's so verbose; even the Guava documentation reads (paraphrasing) "if you think you need these functional constructs, consider first if you can do the same in a tenth of the lines of code by using a for-each loop". Ouch.

The same thing in Javascript or Python is a lot more concise; here's the python equivalent:

animals = ["dog", "cat", "dingo", "leopard"]
dAnimals = filter(lambda x: x.startswith("d"), animals)
# or using a list comprehension:
dAnimals = [animal for animal in animals if animal.startswith("d")]

First class functions are relatively common in Python (and are a requirement for getting anything done in Javascript these days), so I'm familiar with passing functions around as arguments.

And I had the obligatory course on Haskell way back at uni, where I learnt about function currying and partially applied functions.

What about Scala?

You're expecting Scala to be the savior, bringing functional conciseness to Java's world of verbosity. Surely I wouldn't have typed out those other code examples if it wasn't? Well, I'd hate to disappoint:

val animals = List("dog", "cat", "dingo", "leopard")
val dAnimals = animals.filter(animal => animal.startsWith("d"))
// or using the underscore shortcut
val dAnimals = animals.filter(_.startsWith("d"))

It looks like Python, but it's statically typed, runs on the JVM, and is actually transparently compiled to Java first!

I'll just wait here for you to finish clapping.

Done? Good, now let's talk a little about some neat features of Scala which are demo'd here.

  • val is an immutable reference, like a final variabl in Java.
  • Scala does its best to infer the type of variables. I could have written List[String]("dog", "cat", "dingo", "leopard"), but there was no need to, and I also didn't need to declare animals as a val animals: List[String].
  • List has a filter() method, which returns a new list (because a list created with List() is immutable) which contains only elements for which the given function returns true. You can map (transform), reduce, and fold out of the box as well.
  • Very concise syntax for functions: animal: String => animal.startsWith("d") is a function which takes a String and returns a boolean. (In the example, we didn't need to declare that animal has a type of String because it can be inferred from the context.)

And that doesn't even touch on the many other parts of Scala which are much nicer than Java: free standing functions, inline/nested functions (with closures), pairs/tuples, pattern matching (a way to avoid the mass of if else that regularly happens in Java), interfaces (traits) which can specify implementations too (which enables class mixins), support for lazy evaluation (i.e. a variable's value is only evaluated as it's required), support for call by name evaluation (e.g. you can write a logging function which only evaluates its arguments inside the function, rather than at the call site), and way more.

Scala in Practice

In practice, Scala is a cross between "what Java should have been if it hadn't taken a 5 year-long tea break" and "wouldn't it be neat if we could X in a type-safe way?".

Unfortunately, the enthusiasm of the latter camp seems to have led to a lot of scary features; for example, you can write code that writes that writes code that is statically checked at compile time for type safety. There's also a tendency to invent cryptic new operators (try and guess what /: does? It's not a right-associative division operator).

This seems familiar.

Still, I think it's very much worth using today; in my opinion, the benefits (conciseness & safety) outweigh the downsides (namely, that it's still a bit of an academic dumping ground). Right now, I believe much of the scary stuff can be worked around by having strong style guidelines.

Look for a more complete verdict after I've spent some time writing production code. I have some personal project ideas, and Atlassian is starting to use Scala in production, so that will probably be sooner rather than later.

Anyway, I highly recommend Odersky's Functional Programming Principles in Scala. I've already signed up for the next course in the series, Principles of Reactive Programming.

(With apologies to any functional programming or Scala guru readers for any inaccuracies; please feel free to contact me with corrections)

Pelican pages not showing?

I spent a bit of time playing with Pelican. Pelican is a static site generator written in Python, similar to Ruby's hugely popular Jekyll.

The Problem

One thing that isn't particularly well documented is that if you create a sample project, your articles (blog posts) will be picked up in the static site generation process from the content directory without any further configuration, but your pages in the content/pages directory won't get detected, and hence won't appear.

Of course, the first thing to do is to check the pelican settings and look for settings related to pages.. and when you do so, you find PAGE_DIR defaults to pages.

The key to the solution is in the description of PAGE_DIR, which is Directory to look at for pages, relative to PATH.

What's the default PATH? None. Right, that could be a problem.

The Fix

I had assumed that PATH was already be set to content because some of the articles that I had created were showing up, but it doesn't appear to be set in the settings file. Let's try setting it.

PATH = "content"

Voila, it works! Pages are now shown.

My guess is that even though Pelican is clever enough to default your content directory to content, it doesn't seem to do it by setting PATH.