Spiral Arm Logo

Richard's technical notes

Sunday, May 17, 2009

The Smileys in Scala

or: a visual interpretation of some of the syntax of the Scala programming language, with the aim of providing an aide memoire.

The Happy Walrus     : =>


(Yeah, I needed to include the : to make this work).

"The walrus prefers shallow shelf regions and forages on the sea bottom." And like the walrus, in Scala the symbol for by name parameters allows the shallow surface parameter to forage deep inside your code.

An example from Beginning Scala of a method that keeps appending the results of a block of code until the test is true:

def bmap[T](test: => Boolean)(block: => T): List[T] = {
val ret = new ListBuffer[T]
while (test) ret += block
ret.toList
}

Here we see two walruses, test and block, neither of which are evaluated as parameters until you dive into the body of the method.

The Scissors     >:


Putting a lower bounds on a type is rather like cutting off the option of passing a subclass. That is, anything to the left of the scissors in...

def doStuff[B >: T](p: B) = ...

... is snipped away if the type to the left is a subclass ("below") the type on the right. I find it helps to mentally rotate 90 degrees anti-clockwise to see the scissors cutting B if it's "below" T in the class hierarchy.

Chapter 19 of Programming in Scala has the details, as does 8.3 of Scala By Example (PDF).

Let's wield the scissors, by considering a manager who can't handle detail: they can think about fruit and apples, but nothing more specific...

scala> case class Fruit()
defined class Fruit

scala> case class Apple() extends Fruit
defined class Apple

scala> case class Pippin() extends Apple
defined class Pippin

scala> class Manager[A >: Apple]
defined class Manager

scala> new Manager[Fruit]
res2: Manager[Fruit] = Manager@3b2000a5

scala> new Manager[Apple]
res3: Manager[Apple] = Manager@276c9124

scala> new Manager[Pippin]
:11: error: type arguments [Pippin] do not conform to class
Manager's type parameter bounds [A >: Apple]

The Fussy Bird     <:


When it comes to...

A <: B

...the fussy bird knows that the morsel on the left is the special, tastier class when compared to the big old lump of class to the right. In Scala it signifies an upper bound, that A must be the same or a subtype (specialism) of B. And special means tasty, so no wonder she's got her beak and eyes pointing left towards the tasty class.


scala> case class Stuff()
defined class Stuff

scala> case class Food() extends Stuff
defined class Food

scala> case class TastyFood() extends Food
defined class TastyFood

scala> def peck[A <: Food](p: A) = println("yum")
peck: [A <: Food](A)Unit

scala> peck(Food())
yum

scala> peck(TastyFood())
yum

scala> peck(Stuff())
:10: error: inferred type arguments [Stuff] do not conform
to method peck's type parameter bounds [A <: Food]
peck(Stuff())


The bird is so fussy she'll even be happy with Nothing.

So that's the three I've had to work to remember. I hope I've got that far enough in the direction of correct to be useful. Someone will have to sort out stories for <% (view bounds) and the rest of the syntax you might see such as /:.

Sunday, May 10, 2009

Still loving the Scala

The other night I sat down to satisfy just one more quick-fix screen-scraping twitter-based itch. Of course I decided to scratch using Scala, and it really is an attractive language for "doing stuff". It also made me reflect on how Scala has already started to changed the way I write software.

Here's the itch: If you live in the centre of the pebble-beach city of Brighton, and have a dog, the tide times become really interesting, because at low tide sand is revealed, and sand is fantastic to play fetch in. You can get Brighton tide times from VisitBrighton.com, but of course I want the information at the point I'm going to use it. And for me, that means I want a tweet at 6:30 every morning. And so, @brightontide was born (I'm still chatting about copyright with the council, so it might have to disappear).

The problem could be boiled down to curl -> grep/sed/awk -> curl, but there's the added complication that tide times are in GMT, and I want to tweet them corrected for daylight saving.

So with that background out of the way, here we go...

In essence I want to get the tide times for a given date:

trait TideSource {
def lowsFor(day:LocalDate): List[Tide]
}

I'm using the Joda Time classes LocaDate and LocalTime to represent a date (without time) and a time (without a date). The Tide class is just a wrapper for the time and height of the tide, plus the method for converting the time into the right timezone:

case class Tide(when:LocalTime, height:Metre) {

override val toString = when.toString("HH:mm") +
" (" + height + ")"

def forZone(destZone:DateTimeZone) =
Tide( when.toDateTimeToday(DateTimeZone.forID("GMT")).
withZone(destZone).toLocalTime(), height)
}

Next I need to implement a TideSource and, until I find something less hacky, that will be an implementation that scrapes the data from VisitBrighton.com. I want to use it like this:

val tide_times = VisitBrightonScraper.lowsFor(today)

Here's the code to allow that:

object VisitBrightonScraper extends VisitBrightonScraper

class VisitBrightonScraper extends TideSource {

def page = Source.fromURL(
"http://www.visitbrighton.com/site/tourist-information/tide-timetables").mkString

override def lowsFor(day:LocalDate) = {

// We want the times that start with the date in this
// format: 10th May 2009
val date = day.ordinal +
DateTimeFormat.forPattern(" MMM yyyy").print(day);

val Pattern =
"""|(?sm).*<div class="TidalDataEntry"><h3>DATE</h3><table class="TidalData"><tr>
|<th> </th><th class="Time">Time</th><th class="Height">Height .m.</th></tr><tr>
|<td class="Tide">High</td><td class="Time">(.+?)</td><td class="Height">(.+?)</td></tr><tr>
|<td class="Tide">Low</td><td class="Time">(.+?)</td><td class="Height">(.+?)</td></tr>
|</table></div>.*""".stripMargin
.replaceAll("\n","").replaceFirst("DATE", date).r

try {
val Pattern(high_times,high_heights,low_times,low_heights) = page

// The times and heights are in separate columns;
// multiple values separated by "<br/>"
val tides = for ( (time_string,height) <-
low_times.split("<br/>") zip low_heights.split("<br/>") )
yield
Tide( time_string.toLocalTime, Metre(height.toDouble) )

tides.toList
}
catch {
case x:scala.MatchError => println(x)
Nil
}
}

(Yuck. I have to find a better way to format code on this blog)

There's nothing particularly exciting in any of this, but there are a couple of things that surprised me. The first is the zip function, which I distinctly remember reading about and thinking at the time: "nice, but there's no practical application I'll write that will ever need that" :-) But here we are, with a page layout where I have two columns of numbers that need to be paired up: exactly what zip does, and it saved me a couple of loops.

The regular expression is a bit hairy. You can figure out what's going on if you view the source of the page:

Screen Scraping Target

But that code is going to need a unit test. But how to "mock out" the HTTP request? In Java, I probably would have separated things more so I supply the content of the HTTP request to another method that processes it. Or maybe I'd have a factory to supply something that can fetch the HTML; or perhaps I'd use dependency injection.... but this is a quick script, to do a hacky job... but I do want to test it. It turned out to be so very easy:

object MockScraper extends VisitBrightonScraper {
override def page = Source.fromFile(
"src/test/resources/visitbrighton07052009.html",
"UTF-8").mkString
}

Nothing you can't do in Java, but here it feels so concise and easy that it something that becomes usable for a unit test.

Here's the corresponding test code that uses this test scraper:

object VisitBrightonScraperSpec extends Specification {

"Visit Brighton screen scraper" should {

"locate low tide in first day" in {

val tides = MockScraper.lowsFor(new LocalDate(2009, 5, 7))
tides.length must be_==(2)

val expected = List( Tide(new LocalTime(3,38), Metre(1.0)),
Tide(new LocalTime(15,59), Metre(1.0)) )

tides must be_==(expected)
}
// etc
}

Putting it together, you get:

object TideTweet {

def main(args:Array[String]) {

val today = new LocalDate

// Time tides are in GMT, but we will later convert to
// whatever timezone we're in:
val tz = DateTimeZone.getDefault

val gmt_tides = VisitBrightonScraper.lowsFor(today)

val tweet = gmt_tides match {
case Nil => "Gah! Failed to find tide times today.... Help!"
case tides => today.toString("'Low tides for 'EE d MMM': '") +
tides.map(_.forZone(tz)).mkString(", ")
}

println(tweet)

if (args contains "-dotweet")
send(tweet)

}

I've skipped some of the details: you can download a tar.gz of the source if you're interested.

This isn't exactly great code (error states returning Nil? Twitter passwords baked into the source?) but it was an absolute joy and pleasure to write it in Scala.

Thursday, February 05, 2009

Scala & Wicket London Meet Up

Features include...

I know next to nothing about the Wicket web framework, but I was intrigued by the jWeekend London Wicket User's Group meet up last night. The topic was "Scala and Wicket".

As far as Scala and web frameworks go, the main name is Lift. The problem with Lift... No, let me re-phrase that because it's not a problem with Lift. The great thing about Lift is that it's made for Scala, so it's going a great fit to the language. Learning Lift and Scala at the same time should mean you're bouncing the framework learning and the language learning off each other, which is going to teach you a lot.

Then again... if you're learning the language and trying to get your head around a framework, it seems to make the goal of "doing something useful with Scala" just that little bit further away. So how about this: Scala and Java work well together, so why not use a web framework you already know, but just use Scala instead of Java? Start gently, then, when ready, take a look a Lift.

I don't know which approach is best, but it seems there might be something in the gently-gently approach. There is one other compelling reason for looking at a existing (legacy? :-) web framework: you can dig into the publications on the topic, such as Wicket in Action, or Programming Struts, or Stripes etc. (Obviously this situation will change for Lift: I've already expressed my disappointment that none of the big publishers are looking at the practical aspects of Scala, but there is the start of a creative commons text).

Sure, you're not necessarily going to learn Scala idioms from a non-Scala framework, and you're going to run into head-scratching issues, but it seems somehow more manageable to at least try it. More so if you're inserting Scala into an existing project.

Of course, the whole argument falls apart for me in the case of this event, as I don't know Wicket :-) But I've tried Scala in a trivial way with a large existing Struts 1 application, and it was surprisingly painless.

But back to the event and the talks:
  • Daan van Etten gave a lovely "Basic Introduction to Scala With Wicket". The slides, handouts and code are available. One download and two commands to get the example app up and running was pleasing.
  • Dean Phersson-Chapman spoke about his "Experiences Converting an Existing Wicket Application To Scala". It seems there are some serialization issues between Wicket and Scala 2.7.3 which are being fixed.
  • Jan Kriesten showed examples of "Real World Scala and Wicket". If you'd not seen Scala before, this was probably pretty scary stuff in places. Jan clearly knows his Scala and his Wicket very well.
  • Finally, Alastair Maw spoke about the evils of abstraction, which had some fine points about when to avoid it (mostly) and when to embrace it (rarely).


It looks like slides appear over at londonwicket.org.

Thursday, January 22, 2009

Four Scala Books

In addition to the completed and shipping Programming in Scala book from Artima, there are three others in the works:

Great news.

If I'm going to be picky, I'd say I'm a little concerned all these look like they are language books, rather than "activity books". What I mean by that is there's no "Database Persistence using Scala" or "Building Fast Web Sites Quickly with Scala" or "Asynchronous Messaging with Scala"... you get the idea. Sure, you can just get on and use the Java way, but the idiomatic Scala approach is what I'm looking for.

Wednesday, January 14, 2009

The 'Hull City Problem' in Scala

Odd one out

Now that I have a physical copy of Programming in Scala next to me, the final excuse for me not spending more time with the language has gone. In the interest of dipping my toe in the language (well, it's more like my whole leg by now) I'm making a conscious effort to do all and any scripty things I have to do in Scala.

One such script-like thing is the 'Hull City problem'. It's actually a very simple problem, and now I think about it, it's more an assertion than a problem: Hull City football club is the only club name in the English league that cannot be coloured in.

By which I mean, if you look at a name like "Liverpool", it is made up of letters that are fully enclosed. If you were doodling you could colour in the "o"s, the "p"s and the top of the "e". You can't do that with "Hull City". Not in title case, not in upper case and not in lower case. There's no other club in the league that you can say that about.

Or can you?

Clubs come and go, and even change name, so it'd be handy to have a script that could test that the assertion is still true. Here's how I went about it in Scala, once I'd decided it was partly a regular expression problem. Yes, regular expressions are my hammer of choice.

Although not really necessary for this problem, I defined a class for the name of the club and the level in the league (e.g., 1 is the Premier League; 12 is Gloucester Northern Senior League), and a method for deciding if the name can be coloured in:

class Club(val name:String, val level:Int)
{
val failChars = "&ABDOPQRabdegopq"

// We're given team names in Title Case, so that's what we
// check here. The name can be coloured in if there exists
// a character in the name that is contained in the list
// of "fail characters".
def canBeColouredIn = name.exists(c => failChars.contains(c))
}

That seems OK:

$ scalac Club.scala
$ scala
Welcome to Scala version 2.7.2.final [...]
Type in expressions to have them evaluated.
Type :help for more information.

scala> val hull = new Club("Hull City", 1)
hull: Club = Club@bc9065

scala> hull.canBeColouredIn
res0: Boolean = false

scala> new Club("Liverpool", 1).canBeColouredIn
res1: Boolean = true

To try this on all clubs you need to export the list of teams from Wikipedia, which gives you an XML file that contains a list of club names in a Wiki markup text format. A line in that file looks like this: |[[Aveley F.C.|Aveley]]||[[Isthmian League]] [[Isthmian League Division One North|Division One North]] (Level 8)

And you also need some code to read the file and run a regular expression over it:

import scala.io.Source
object Club
{
def load(filename: String): Iterator[Club] =
{
// Helper to remove the crud from team names:
def clean(input:String) = input.replaceAll("&","&").
replaceAll("AFC","").replaceAll("A.F.C.","").
replaceAll("F.C.","").trim

val fileContents = Source.fromFile(filename, "UTF8").mkString

// Regular expression for extracting team name and level.
val clubInfo = """\|\[\[([^|\]]+).*\(Level (\d+)\)""".r

for(clubInfo(name,level) <- clubInfo findAllIn fileContents)
yield new Club(clean(name),level.toInt)
}
}

A couple of points here: reading a file into memory using Source and mkString is rather handy compared to what I'd usually do (i.e, using a buffer and a loop, or grabbing a library that has it defined already); creating a regular expression means calling .r() on a String which is almost as short as having regular expressions as part of the syntax of the language. But I think it's the for loop that stands out. The findAllIn call looks straight-forward enough, but the left hand side of that is cute. A regular expression in Scala, such as clubInfo, is also an "extractor" in the language, which for me is probably best explained by example as I don't have a full grip on all the concepts yet:

$ scala
Welcome to Scala version 2.7.2.final [...]
Type in expressions to have them evaluated.
Type :help for more information.

scala> val clubInfo = """\|\[\[([^|\]]+).*\(Level (\d+)\)""".r
clubInfo: scala.util.matching.Regex = \|\[\[([^|\]]+).*\(Level (\d+)\)

scala> val clubInfo(name,level) = "|[[Aveley F.C.|Aveley]]||[[Isthmian League]] [[Isthmian League Division One North|Division One North]] (Level 8)"
name: String = Aveley F.C.
level: String = 8

scala> println(name)
Aveley F.C.

scala> println(level)
8

Handy? I think so. Especially when you can use it in a for loop to split out the matches into the two groups.

Putting it all together (and you can download the source files if you want) we can print out all the clubs that cannot be coloured in:

$ scalac Club.scala
$ scala
Welcome to Scala version 2.7.2.final [...]
Type in expressions to have them evaluated.
Type :help for more information.

scala> val clubs = Club.load("footballclubs.xml")
clubs: Iterator[Club] = non-empty iterator

scala> for(club <- clubs; if !club.canBeColouredIn)
| println(club.name)
Hull City

So there we have it. It's true: only Hull City can not be coloured in. For now...

UPDATE: If you prefer just the script version, here it is condensed:

def clean(input:String) = input.replaceAll("A.F.C.","").trim

def isGood(name:String) =
!name.exists( c => "&ABDOPQRabdegopq".contains(c) )

val regexp = """\|\[\[([^|\]]+).*Level""".r

import scala.io.Source

for( line <- Source.fromFile("footballclubs.xml", "UTF8").getLines;
regexp(name) <- regexp findAllIn line;
if isGood(clean(name)) )
println(name)

Wednesday, December 10, 2008

Augmented Reality at £5 App Xmas Special

At £5 app

There's something right about £5 App running a demo/game/fun event as the Xmas special. I mean, how does this sound: I turn up at The Werks, icy cold, to be handed a lovely hot mulled wine by the Ribots, directed towards the mince pies, and then entertained with a range of funky technology stuff. Mmm.

This evening there were talks and demos covering: collaborative interactive fiction (a.k.a., group-generated text adventure), namely Spaceship; Emily demoing robot sumo; mobile phone Lightsaber dueling (no, not with the iPhone); eye controlled Pong, from Ben Rubinstein. All excellent.

Maybe it's because of the fun we had the previous day at the Flash Big Screen Bonanza, or perhaps it's because I'd been noodling with JavaFX, but I found Seb's Flash 3D and augmented reality demos great fun.

Seb augmented reality

I'd say, of the evening, the biggest wow! was when Seb showed us the augmented reality stuff; and the biggest round of applause of the night was when he said "hey, all I've done is download it and compile it" :-)

But you don't even need to do that. From that last link, if you print out the PDF symbol, click the start button (and right click and select "USB camera" in settings, but you may not have to do that), and hold up the symbol you can try it yourself. It's not bad, eh?

Papervision Augmented Reality

The underlying tech seems to be from the C ARToolKit, which is available under the GPL and other licenses. Given that the documentation touches on support for SGI/Irix and VRML I think it must have a pretty robust history.

There's some mention of a Java bindings for it, but after a bit of digging it turns out there's a pure Java 6 port, and from what I can understand a version from the same author for the Android platform. I've not tried them; I've no idea if they work.

If you're interested in this area, there's documentation on how ARToolKit works.

Thursday, November 13, 2008

Brighton Scala User Group

I left the book cover in the pub

Last night was the Brighton Scala User Group pub meet up. For the record the conversation covered:

  • The Scala Plugin for Eclipse, how it runs on the Mac, how it lives with Maven.
  • The power of Aspect Orientated Programming for fixing other people's code.
  • NIO.
  • The JVM language summit, and photos taken.
  • Neil Gafter, Closures In Java, what we think NG thinks of Scala.
  • Refinements, a.k.a. duck typing (see section 3.2.7 and specifically example 3.2.4 on page 22 of the Language Spec).
  • Invoke dynamic, tail recursion, catching multiple exceptions.
  • Is there a JSR for Java 7?
  • Java 7, do we care about Java language features now? We do care about the JVM.
  • Languages on the JVM with parity performance with Java: Scala, Clojure.
  • Idioms.
  • Mac keyboards as a way for Apple to lock in users.
  • What Scala language features Java developers like.
  • Using existing web frameworks other than Lift with Scala.
  • Focal length, depth of field.
  • Sun's war chest.
  • The Groovy/Grails/Spring Source deal.
  • Return early v. one and only one return.
  • Language features for nicely dealing with instanceOf tests.
  • Interesting corner-cases of the type system.
  • What *is* it about Scala that's so attractive? What problems is Scala solving?


The next event will be 17 December 2008.