Richard's technical notes

Scala London Coding Dojo

noreply@blogger.com (Richard) — Wed, 23 Jul 2008 06:06:00 +0000

Monday was the first Scala London User Group meeting that was held around a keyboard. Robert Rees arranged for a room, food, and drink at ThoughtWorks, Jamie Webb kindly gave an intro to Scala talk, and Aaron Roth provided print outs of a Scala for Java Programmers cheat sheet...and so we spent time working on the supermarket problem.

I counted 19 people (of which 3 were Brighton based), of mixed abilty (from functional people or Java people who had never touched Scala, all the way to expert Scala-heads), and we had working, unit tested, software by 10pm. Which—given that coding dojos sound a bit mad—is pretty good going.

Dojos are probably not going to be the way forward for this group, but I think it did a good job as an ice-breaker: an inclusive activity; a task that's not daunting, that anyone can have a crack at. I definitely learnt stuff: testing using Specs, and need to understand the Scala patterns, such as Cake for dependency injection, and I have to stop putting off looking at Lift. Perhaps the most important thing for me was to learn that the Eclipse Scala plugin is going to be able to do mixed compilation of Java and Scala source in a project.

There was video of Jamies talk. I'll add a link if it's posted any place. At the moment there's no web site for lsug but get in touch if you'd like the mailing list email address.

There are more photos on Flickr tagged with lsug.

Comparing closures in Java, Groovy and Scala

noreply@blogger.com (Richard) — Sat, 14 Jun 2008 10:09:00 +0000

On Paul's return from JavaOne this year, we spoke about Neal Gafter's Closures Cookbook talk. From what I understood, this was a look at the BGGA closures proposal, and contained an example that pushed hard on some of the tougher closure issues for Java. I thought it might be fun to look at the Java example from the talk, and covert it to Scala and Groovy.

Why those languages? Because they are the three JVM languages I'm most interested in. I suppose I could also have compared the closure support in Jython, JRuby or... well, there are a few to choose from, but this blog is going to be plenty long enough with just three.

Let's start with the Java example that was given, remembering that this is a proposed syntax, that may or may not make it to Java 7 or later. As I understood the example it was this: imagine you want to add the ability to time a block of code, and you wanted to do it in a way that would look almost like a new keyword has been added to the language; and you wanted to pass in a parameter to name what you were timing; and the block you're timing returns a result, or might throw an exception. So, quite an involved case.

Here's how the current Java proposal looks:

// Here's a method that uses the time call:
int f() throws MyException { 
  time("opName", {=> 
    // some statements that can throw MyException 
  }); 
  time("opName", {=> 
    return ...compute result...; 
  }); 
}

So we're timing a couple of operations, and we're doing this inside a method, f, that returns an integer. The implementation would be....

interface Block { 
  R execute() throws X; 
} 

public  R time( 
    String opName, Block block) throws X { 
  long startTime = System.nanoTime(); 
  boolean success = true; 
  try { 
    return block.execute(); 
  } catch (final Throwable ex) { 
    success = false; 
    throw ex; 
  } finally { 
    recordTiming( 
      "opName", System.nanoTime() - startTime, success); 
  } 
}

As you can see, time takes an arbitrary text label and a block of code, runs the block and tells you how long the block took to run and if it succeeded or not.

That's the example that was given at JavaOne. Now for the same thing in Groovy...

To make runnable code for Groovy (and for Scala), I had to decided to time something. So I'm timing a block of code that randomly throws an exception or returns something. And then timing a block of code that just returns a number. In Groovy that would be:

def time(opname, block)
{ 
 long start_time = System.nanoTime()
 boolean success = true
 try {
  return block()
 } catch (Throwable ex) {
  success = false;
  throw ex
 } finally {
  diff = System.nanoTime() - start_time
  println "$opname $diff $success"
 }
}

int f() throws Exception {
 time("a") { 
  Random r = new Random()
  if (r.nextInt(100) > 50)
   throw new IOException("Boom")
  else
   return 42
  }
 
 time("b") {
  return 7
 }
}

println f()

An example of running the code:

$ groovy time.groovy 
a 37116000 true
b 69000 true
7

$ groovy time.groovy 
a 39998000 false
Caught: java.io.IOException: Boom
 at time$_f_closure1.doCall(time.groovy:21)
 at time$_f_closure1.doCall(time.groovy)
 at time.time(time.groovy:6)
 at time.f(time.groovy:18)
 at time.run(time.groovy:31)
 at time.main(time.groovy)

Note that the Java example is typed in that it uses a generic type, R, for the return value which gives you some compile-time checks. That is, when you run time and use the result, the compiler will enforce that your declaration of the result has the same type as the return type of the block you're timing.

Although Groovy does support generics, I've not used them here, and as a result the Groovy example doesn't have that type-safety. I think that's the way one would typically write Groovy code.

UPDATE: as was pointed out to me in the comments on the Java Lobby version of this blog post, this isn't the same as the Java version. In the Java version a return in the closure returns out of the enclosing block.

Now a look at the same code in Scala:

def time[R](opname: String)(block: => R) = {
 val start_time = System.nanoTime()
 var success = true
 try {
  block
 } catch {
  case ex: Throwable => {
   success = false;
   throw ex
   }
 } finally {
  val diff = System.nanoTime() - start_time
  println(opname + " " + diff + " "+success)
 }
}

def f():Integer = {

 val answer = time("a") {
  val r = new Random()
  if (r.nextInt(100) > 50)
   throw new java.io.IOException("Boom")
  else
       "42" 
 }

 println ("the answer, "+answer+" is of type "+
      answer.getClass())

 val seven:Integer = time("b") {
  7
 }
 
 println ("seven is of type "+seven.getClass())
 
 return seven 
}

f()

I'm still not using Scala day-to-day, so this might be a little awkward: thank you to the London Scala User Group for helping me clean up my syntax, but all the mistakes are mine.

This code has the same properties as the Java code (type safety via the R generic type), but seems a little shorter and neater. Additionally, the thing I like about the Scala code (and the Groovy code) is that the languages return the value of the last statement in a block, and that the syntax allows a clean time("thing") { ... } format.

One observation: I've used the Integer class, which is deprecated, in order to be able to print out the class of the return type in the function f(). Without the :Integer declaration I was getting weird compile errors. As I said, my understanding of Scala and type inference isn't there yet.

The output from running the code:

a 913000 true
the answer, 42 is of type class java.lang.String
b 13000 true
seven is of type class java.lang.Integer

a 936000 false
java.io.IOException: Boom
 at Main$$anonfun$1.apply((virtual file):26)
 at Main$$anonfun$1.apply((virtual file):23)
 at Main$.time$1((virtual file):8)
 at Main$.f$1((virtual file):23)
 at Main$.main((virtual file):44)
 at Main.main((virtual file))
// rest of stack trace removed

There's no conclusions here. It's just an exercise in comparing closures code in three different languages. I've probably missed some of the nuances of the Java example, but hey... it's a starting point.

Mobile Monday: Monetisation through Advertising

noreply@blogger.com (Richard) — Tue, 13 May 2008 08:34:00 +0000

Last night I was at Mobile Monday London to catch four presentations. The topic: advertising on mobile. The summary: there's a need for better measurement.

Claire Valoti from Mindshare was up first, giving the agency view of mobile advertising. She split the appeal of mobile in a media plan in to two parts: as a delivery mechanic (to extended reach, getting to a winder audience in a different mode); and as a platform (for couponing, or video uploads). She then went on to describe some issues and themes... and standardization and measurement were pretty much top of the list. That is, there's currently no good standard measure of reach, sessions, traffic or users for mobile advertising. And even if there was, it needs to be integrated into existing web buying systems, rather than via a mobile-specific system.

Claire had a good observation on social networking, and the importance for mobile. The m:metrics numbers quoted showed that the 18-24 and 25-34 age groups were more likely to be using social networking from the mobile, compared to 13-17 age group. It's not just about a youth market.

Another good point was made that events drive handset usage, as does "the right handset". An example given: at the end of the football (soccer) season, there's a drop in usage; during the season people are out and about and will use the mobile internet to look up news. As for "the right device", just look at the figures for internet usage on the iPhone compared to anything else....

Obviously mobile advertising has to be used in the context of the whole campaign, and it has to be relevant content: don't go taking an ad made for TV and slapping it on a mobile. Use the right techniques to get reach (banners, SMS, bluetooth), use location information to make it relevant, use coupons and free content to make it useful to the customer. And the user experience needs to be more streamlined: don't issue mobile coupons without telling your retail staff how to handle them.

Aside from measurement, another barrier to growing advertising budgets is the number of people involved in setting up a mobile campaign. The buying experience needs to be more streamlined.

Mobile advertising challenges were summarized as: speed, reach, cost, ROI, standards, and measurement. As examples of reach issues: Claire was interested in QR codes, but only 10% of mobile devices have a reader installed; buying from Blyk or similar to get an audience was described as having "limited reach" at the moment; mobile search needs to scale up as, right now, it's hard to spend the budget.

Shan Henderson from Vodafone kicked off his presentation by saying that it's the metrics that matter. Click rate is not the full story, but it's often the headline. Also audience demographic, behaviour, session length, frequency and reach are all important because "money follows measurement in media". He showed the "measurement gap" graph. I'm sure the slides will be available soon, but for now I've done a shaky recreation:

(I'm guessing Edward Tufte would not approve). So, we all agree it seems: we need better standards for measurement and more efficency in the buying and selling of advertising in order for the market to grow.

To address this, Shan went on to describe the GSMA Metrics Study which aims to make it easy to plan and measure mobile media. Later this year it's going to produce (one or more of): guidelines, best practices, definitions, technical standards, responsible ad practices, content standards. This will all be via an as yet unnamed ABC-like trusted independent organization.

Russell Buckley took the stage to talk about Admob, a self-serve and full-service ad network, shifting 2.bn ads a month, in 160 countries via 3000 mobile web sites. They select mobile sites, put ads on them, and share the revenue with the site. He described mobile advertising as good for: bands; to promote mobile websites; for operators to monetize their sites; and content owners to have access to a new marketing channel.

There was then a page of logos of who's advertising, and it showed that the US is generally ahead in terms of adopting mobile advertising. In fact, ad requests by country (to March) showed 47% of ads were for the USA. Other figures were: India (9.8%); UK (6.7%); South Africa (5.3%); Indonesia (4.7%); Romania (1.9%); Canada (1.3%); Philippines (1.3%); France (1%); Israel (1%); RoW (20.1%). By handset, Nokia were the top, although they are dropping: something that's not yet reflected in their market share.

Russell presented some case studies. MTV wanted to drive traffic to an awards site, and using text adverts they had a 300% traffic boost and 400% increase in downloads. Land Rover USA saw 23% of users interacting with an advertised landing page, with 3% going on to click the link to make a call to a dealer. Adidas saw a CTR of "well above 3%". Coca-Cola saw a CTR of 1.31%, but 130% watched a mobile video that was being promoted (some people watched it more than once).

Ray Anderson from Bango described the basic model for using advertising as: select a channel, measure response, look at user purchases, analyse ROI, then... repeat.

He made some great observations on the issues of search. Depending on the combinations of search provider and operator, you may find you get very different results from the advertising spend (especially if the content is transcoded). But he had a graph that compared the revenue Bango customers had received over time from Orange and Vodafone: the doubling (or more) in the graphs when the operators introduced a search page was impressive.

After the talks there was a brief panel session. All good stuff. And held in a great building.

Photos: from Route79, from appelquist, from Alex Craxton

Video and audio: usual appear on the Mobile Monday web site after a little while, so check there.

Decrypting JetS3t Files

noreply@blogger.com (Richard) — Sun, 13 Apr 2008 11:48:00 +0000

This post is going to be a bit niche. The scenario is that you've used JetS3t to backup data to Amazon S3 via the synchronize tool, and in particular you've used the -c option to encrypt the data. But you've downloaded the file with another tool, such as ForkLift or S3 Browser. How do you decrypt the downloaded file?

The default encryption is PBEWithMD5AndDES, and with that knowledge you may be able to find a tool that can decrypt it for you. I went a different way, and just hooked straight into the encryption utilities inside JetS3t:

// Apache 2 license 
public static void main(final String... args) 
    throws GeneralSecurityException, IOException
    {
      
        if (args.length != 4 && args.length != 2)
        {
            System.err.println(
              "Usage: Decrypt [-alg algorithm] KEY FILE");
            System.exit(1);
        }
   
        final String alg;
        final String key;
        final File encryptedFile;
        
        if (args.length == 4)
        {
            alg = args[1];
            key = args[2];
            encryptedFile = new File(args[3]);
        }
        else // use defaults for algorithm:
        {
            alg = "PBEWithMD5AndDES";
            key = args[0];
            encryptedFile = new File(args[1]);
        }
        
        final EncryptionUtil eu = new EncryptionUtil(
                key, alg, EncryptionUtil.DEFAULT_VERSION);
        
        CipherInputStream decrypt=null;
        BufferedOutputStream outStream=null;
        try
        {
            decrypt = eu.decrypt(
                 new FileInputStream(encryptedFile));
            IOUtils.copy(decrypt, System.out);
        }
        finally
        {
            IOUtils.closeQuietly(decrypt);
            IOUtils.closeQuietly(System.out);
        }
               
    }

So with a bit of fiddling you can copy, paste and compile that (you'll need the JetS3t library and supporting JARs, plus Commons IO). Or you can download jets3t-decrypt-dist.zip, which contains the source and the reqired libraries. Once inside the ZIP you can run:

 java -jar Jets3tDecrypt.jar password encrypted.file 
                 > decrypted.file

The libraries and the Jets3tDecrypt.jar was packaged automatically by Netbeans 6.1, which is a lovely touch from an IDE.

QuickTime for Java

noreply@blogger.com (Richard) — Sun, 16 Mar 2008 16:18:00 +0000

A few months ago I was experimenting with QuickTime for Java. It's the binding between Apple's QuickTime "stuff" and the Java language, allowing a Java developer to invoke QuickTime on the Mac (and presumably also on Windows). The reason this appeals is that we do Java, and have accumulated a fair amount of Mac hardware.

To cut a long story short, my limited experienced shows that QuickTime, when compared to the standard Java image libraries, produced smaller images of higher quality, faster. It's unusual to get all three benefits together (smaller, faster, better). Too good to be true, even, which leads me to think I've screwed up someplace, but I've not spotted it yet.

The particular test that interested me was was taking a camera phone photo, resizing it and rotating it. Here's an example original:

Here's the rotation using tried-and-trusted AffineTransform plus ImageIO:

And here's the same transformation run through QuickTime for Java using the GraphicsImporter and Matrix objects:

Now, it's subtle but the QT4J image looks to have sharper colours, and seems to be a better representation of the original input. It's also 20k compared to the 76k using the Java 2D libraries, and the code runs 1.7 times faster. The downside: you need Apple hardware.

A few conclusions: it seems the Java 2D code doesn't just fall through to QuickTime on the Apple platform in any simple sense (which, I suppose I might have naively expected). Second, I suspect there's a lot of tuning that can be done in the Java 2D client usage to improve the quality, but QT4J seems to have good defaults.

(Before you ask, the dog's name is Jack.)

Getting Started with Scala

noreply@blogger.com (Richard) — Sun, 03 Feb 2008 14:44:00 +0000

Friday night was the first London Scala User Group meet-up, so it seems like the right time to say what I've learned so far about the language.

I'm not a functional programming person. My background, pre-Java and C, was dominated by POP-11 so it's fair to say I'm more comfortable with imperative coding than anything else. But that's kind of why Scala's a serious consideration for me: it doesn't force the functional stuff down your throat, but rather it's all there, object and functional, so you can pick and choose.

Other compelling aspects of the language are: it runs on the JVM, which has had a huge engineering investment to make it as fast as it is today; has Java-like features, but looks like it lets you get your work done with less lines of code; and it's strongly typed. In other words, it's comfy for a Java person, while putting you in a position to use the funky actors and functional stuff, and maybe make even better use of those multi-core machines.

Here's an example of less-lines-of-code thing. In Java, the type-safe idiom for iterating over a map is:

final Map<String, String> map 
    = new HashMap<String, String>();
map.put("dog", "woof");
map.put("cat", "meow");

for(Map.Entry<String, String> entry : map.entrySet())
{
  final String name = entry.getKey();
  final String value = entry.getValue();
  // do something with name and value here
}

The same code, just as strongly typed, in Scala is:

val map = Map[String,String]("dog" -> "woof", "cat" -> "meow")
for( (name,value) <- map)
{
 // do something with name and value here
}

Or, if you wanted that last part could be: map.foreach( (pair) => ...do something with pair._1 and pair._2 here.. ).

If you're interested in getting into this stuff, I found it useful to start with A Scala Tutorial for Java programmers PDF, then listen to Episode 62 of Software Engineering Radio, an interview with Martin Odersky on Scala (thanks to Dom for pointing that out to me).

After that, I think the best place to go is into your wallet and buy a copy of Programming in Scala: A comprehensive step-by-step guide. It's a pre-print, with quite a few errors, but I'm pretty sure they'll get cleaned up soon and it'll be a great text.

Tags: scala

UPDATE: oops, I forgot to escape the < and > tags in the Java example above, which hid the generic types of the map. Fixed now.

Android Developer Workshop

noreply@blogger.com (Richard) — Wed, 02 Jan 2008 09:16:00 +0000

Half-day workshops are great: you get away from distractions; get to ask questions and get quick answers to cover off those things that are bugging you; you're more-or-less forced to spend more time hands-on with a technology; you get a sense of the buzz around something and which bits are good and bad; and... it's only half a day gone if it doesn't turn out that way. For those and other reasons I attended the Android Developer Workshop at Google London on 17th December 2007, hosted by Dick Wall. It did a fine job of highlighting the current state of Android and which areas are the important ones to focus on.

The first thing that's very clear: you can get a lot done in Android in not a lot of code.

If you're a Mac user the other thing you notice is that you can develop on a Mac without jumping through hoops. It's almost like Google are treating the Mac as a first class citizen, and I can't tell you how convenient and wonderful that is.

I've been playing around with some of the APIs and finding it all well documented already, and quite fun. For my own reference, useful pages include: the optional APIs home page, the glossary, the class index, the application lifecycle, and the activity lifecycle.

It's new, so there are gaps in the documentation. For example, when putting together an LBS application, you discover that the emulator supports mock data sources, but you need to dig around the mailing list to find out how to register them. Say you have a KML file you want to use as a location provider called "brighton". Call the file "kml" then load that into the emulator from your shell with: $ adb push kml /data/misc/location/brighton/kml and then restart the emulator.

There's a lot that's not done yet or not figured out. One nice idea is that there may be support for an XMPP-like server push. That is, your Android app can listen out for server-generated Jabber-ish messages and do something. How will that work in practice across the networks? How will mobile clients be looked up and discovered? We can guess at these things, but we don't yet know the answers.

So, early days. For now we only have emulators, so we won't get much of a feel for how this will all pan out until there are at least a couple of handsets out there. But it's powerful, well thought-out stuff. If you're thinking about this space, I'd say it's at least worth spending a day working through the tutorial application.

The event was recorded so I'd expect the video to show up somewhere on YouTube one day.

Silverlight

noreply@blogger.com (Richard) — Fri, 07 Dec 2007 11:24:00 +0000

I'm impressed with what I've heard about Silverlight. Microsoft put on an evening event in Brighton, bought a few beers, and tried to show Silverlight. I say "tried" because it turned into an evening of question answering. Which was fine: I could have done with seeing more, but I guess I can find that online.

Judging by the numbers, there are a lot of people interested in learning about the technology. So what is it? We were told it's not a "Flash killer", but you know what... the differentiation between Silverlight and Flash is subtle, so comparisons to Flash are going to be difficult to avoid. It's probably fair to say it's not a Flash killer yet, as Flash will have the edge in experience and capabilities for a while. But beyond that, it's probably not a Flash killer because I suspect Silverlight doesn't care about Flash or competition from Flash.

As of 1.0, Silverlight has gone after the prominent use case of Flash: video. Microsoft are talking to content partners to create compelling stuff, which will lead to more downloads the plugin, which means a bigger market, which means more content produced, which means more downloads of the plugin, which means...spotted the pattern? Good.

With 2.0 (or is it 1.1?) Silverlight will be more capable, supporting development in the .NET languages. This is the only differentiator from Flash that I heard all night: Aral asked the question, and the answer was that Silverlight was going to appeal to the corporate developer, where Silverlight + .NET are going to be a natural fit for them.

It looks like Microsoft have done a great job on the tooling (I've not actually tried the tools myself). The workflow seems to be going from Illustrator or Expressions Design to XAML then adding behaviour via Expression Blend or Visual Studio and back to XAML for presentation. The demo of the tools, although brief, hit the mark. Sure you can only run them on Windows, but the point was made that you can fall back on editing the text (ouch). Or third parties could work on editors for other platforms.

This leads to some envy here, because over in the Java world we're still waiting to see the tooling, fast and light consumer VM, and plugins for JavaFX. The Java GUI people have been producing some great UI content—tellingly, the kind you don't normally associate with Java—and they've been making all this easier. And yes, the Java Media people are initially targeting (can you guess?) video playback. So that's all great, and will presumably all be open source, although it'll be interesting to see how that plays out with the various patent issues over video . Still, as our glorious pony-tailed leader puts it: "this was the original vision behind the Java platform". But it's frustrating watching a game of catch-up. Whatever gets delivered is going to have to be very good: Adobe and Microsoft have set the standard.

The overall impression is that Silverlight is well executed. "What's the catch?" was one question. I can only speculate at why Silverlight exists and what warning signs there might be:

Silverlight could be all about pushing Microsoft video formats. So, for example, if you want to do properly controlled streaming, you're going to need to pay and use the Windows streaming products. That's a shame as other streaming tools exist, other video formats exist, and would actually make Silverlight more compelling.
Silverlight is going to run on Vista, XP, Mac OS 10 and all the major Linux distributions via the Mono team's Moonlight project. That's a great intention, but be worried when you hear of any stories of delays to a release on a particular OS or spats between OS vendors because someone changed something, leading to finger pointing about why Silverlight doesn't work somewhere.
Content: this could be all about bringing RIAs and compelling content to Microsoft developers. In that case it's just breaking a Flash monopoly (no bad thing, and btw, it's not like it's a strong-armed kind of monopoly there: it's just that nothing else is as good as Flash at what it does).
Open source: Silverlight isn't, and that's important. Not because of cost, but because of transparency and trust and standards. It's easier to get screwed over when the product is directed and developed behind closed doors.

But that speculation aside, it's striking in what's been achieved: a technology to watch and noodle with. Hats off to Josh for organizing a great event.

Future of Mobile, 2007

noreply@blogger.com (Richard) — Thu, 22 Nov 2007 18:17:00 +0000

I was at a one-day conference last week, Future of Mobile, held at the IMAX at Waterloo, London. Great idea to have a conference in a cinema: huge screen, good sound, comfy seats, cup holders.

What follows are the remarks or thoughts I noted down...

Consumers

Mobile phone users are not consumers, but creators. Phone calls, text messages are made up of a create and consume pair. I've heard this a great deal lately, especially in the context of the iPhone. There's clearly some truth in it, but I wonder for how long: if my phone has a music player in it, that's more consumption that creation. If I'm browsing the web, that's consumption. If I'm updating a social networking site from my phone, or uploading a photo to Flickr, sure, that creating, but it's not unique to the phone. Email and IM can also be a create and consume pair, and that's something you can do on the desktop or the handset. I'm not sure the whole active v. passive consumers meme is terribly useful.
Mobile's combination of location plus attention is a unique advertising opportunity.

Content creation

"Vodafone UK and Novarra are breaking the mobile web". (See TechCrunch)
If you're anti One Web, Luca Passani's alternative to the W3C's Mobile Web Best Practices will be something you'll want to read: Global Authoring Practices for the Mobile Web
One Web is about thematic consistency: "Ensure that content provided by accessing a URI yields a thematically coherent experience when accessed from different devices."
WURFL has 3,000 developers, and WALL (Wireless abstraction library - a markup system to help automatically output the right kind of markup for different devices) is going into a new version: WNG. Slides are available for that announcement.
.Mobi exists to "help the mobile web grow". The Device Description Repository (DDR) API is due in Dec 2007/Jan 2008, but isn't based on WURFL. The reason is that .mobi want to "add trust" and "add industry" to the API, protecting companies against possible royalty payments, and to provide confidence via a brand.
Google mobile search is currently terrible.

Mobile AJAX

Daniel Appelquist: Mobile Ajax will replace a class of application currently written in Java, will reduce time to market, open up the space to more developers, and result in more apps.
Challenges for mobile Ajax: access to device hardware, consequences of privacy control, need for developer guidelines.
Charles McCathieNevile: ECMAScript 4 is a "proper programming language", which might help reduce the amount of battery power that Javascript consumes.

Android

There was lots of interest in Android.

It's a full-stack mobile solution, and open source. Based on Linux 2.6, plus a hardware spec, apps, and Java.
Android exists to promote openness, build a loyal developer base, grow Google experience, and make handsets cheaper (software is 10% of the handset BOM).
Each application is a separate Linux process, and all apps are equal to the operating system. It will include two-way synchronization (it'll be interesting to see how they've solved that problem).
Google are not worried about writing specs, and just wanted to write it and get the code out.
The architecture does allow for support of MIDP applications alongside Android applications. The Java implementation is Dalvik, plus a collection of useful Java packages.
The claim is that the richer APIs provided give more control compared to MIDP.
The system will be completely open source as soon as the first handset shipped. This isn't expected until late 2008.

Some of presentations have started to appear on SlideShare.

Watching Production Logs

noreply@blogger.com (Richard) — Mon, 22 Oct 2007 09:19:00 +0000

I've always believed that production application error logs should be empty. I've also implicitly assumed that any errors in live application logs are rare, freakish, occurrences, and on the whole of little relevance. But... what would it take to get an empty error log? Or at least an error log that only contained truly anomalous events?

You know what I mean, right? You sprinkle your code with log.error("a bad thing happened", exception), and during development and testing maybe you see them. On your production machines you might be lucky enough to have someone who knows and watches the logs, or perhaps you occasionally look at the logs yourself: but on the whole they are ignored unless someone starts jumping up and down—and then you take a look at the logs.

Well, that's going to change for me. I've now become addicted to having production application error messages sent to me by email (before you offer, no thanks: you can keep your own errors to yourself). It's quite instructive to see messages in your mailbox the moment something odd happens in production.

Here's an trivial example of what you might see:

From:    whoever
Subject:  [SMTPAppender] richard@toto error message
Date:  19 October 2007 19:43:10 BDT
To:    someone

[2007-10-19 19:43:10,290]

ERROR

Test

Caught an exception

java.lang.NumberFormatException: For input string: "Hello world"
 at java.lang.NumberFormatException.forInputString
   (NumberFormatException.java:48)
 at java.lang.Integer.parseInt(Integer.java:447)
 at java.lang.Integer.parseInt(Integer.java:497)
 at Test.main(Test.java:30)

The kinds of things we've seen so far are:

A couple of spiders hitting URLs that that don't exist, but must once have been incorrectly published.
A user calling an action but not passing any parameters. I'm thinking they've bookmarked something that ought to be bookmarkable, but isn't (what can I say: we didn't write all the code that we support).
A plain good-old straight-forward bug in some background task that no-one would ever know was failing, but is now going to get fixed.

I'm not telling you anything new here. I'm just saying that it's worthwhile trying.

Setting this up with log4j is pretty straightforward. There's a good article over at ONJava.com that explains it. All you need to do is configure the appender in log4j.xml (if you're using the .properties version, migrate to the .xml version first). Here's an example:

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration>
    <appender name="STDOUT" class="org.apache.log4j.ConsoleAppender">
        <param name="Target" value="System.out" />
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern"
              value="[%d{ISO8601}] %-5p %c %m %n" />
        </layout>
        <filter class="org.apache.log4j.varia.LevelRangeFilter">
            <param name="LevelMin" value="DEBUG"/>
            <param name="LevelMax" value="INFO"/>
        </filter>
    </appender>
    <appender name="STDERR"  class="org.apache.log4j.ConsoleAppender">
        <param name="Target" value="System.err" />
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern"
              value="[%d{ISO8601}] %-5p %c %m %n" />
        </layout>
        <filter class="org.apache.log4j.varia.LevelRangeFilter">
            <param name="LevelMin" value="ERROR"/>
            <param name="LevelMax" value="FATAL"/>
        </filter>
    </appender>

  <appender name="email"  class="org.apache.log4j.net.SMTPAppender">
        <param name="BufferSize" value="10" />
        <param name="SMTPHost" value="your host here" />
        <param name="SMTPUsername" value="optional username" />
        <param name="SMTPPassword" value="password if you need it" />
        <param name="From" value="whoever@blah" />
        <param name="To" value="someone@blah" />
        <param name="Subject" value="[SMTPAppender] 
             ${user.name}@${hostname} error message" />
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern"
              value="[%d{ISO8601}]%n%n%-5p%n%n%c%n%n%m%n%n" />
        </layout>
        <filter class="org.apache.log4j.varia.LevelRangeFilter">
            <param name="LevelMin" value="ERROR"/> 
            <param name="LevelMax" value="FATAL"/>
        </filter>
    </appender>

    <root>
        <level value="all" />
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="STDERR"/>
    <appender-ref ref="email"/>
 </root>
</log4j:configuration>

Yeah, yuck to look at, but it's pretty simple. The part you want is the SMTPAppender bit in the middle. Leave out the username and password param tags if your SMTP server doesn't require authentication.

The XML above is almost the exact example from the ONJava.com article, except for a couple of items I needed. The first is that I wanted the name of the user the application was running under, and also the hostname of the box the application was running on. To do that, I've used a feature the fine log4j people included: environment properties substitution. You'll see ${user.name} in the subject line, which is a standard Java system property. The ${hostname} is a property I provided to the application by starting it with -Dhostname=`uname -n`.

If you want to you can also dynamically modify the SMTP Appender at run time by iterating over the root logger, checking for an instance of SMTPAppender and then changing what you want. If you do that, though, the clue you need is to call appender.activateOptions() when you're done modifying the appender.

BarCamp Brighton 2007

noreply@blogger.com (Richard) — Sat, 15 Sep 2007 15:36:00 +0000

Tick... I've now attended a BarCamp. I absolutely loved it: many topics + small groups = a great opportunity to learn and chat about lots of stuff.

Some of the stuff I managed to sit in on, included:

CSS Selectivity Snap. Earlier in the day I'd jotted a reminder: "must learn more CSS properly", and then Jon Linklater-Johnson (pictured above, right) steps in with a fantastic crash course in selectivity. Huge fun.
Jay Caines-Gooby chatting about backups. Nice idea: grab a NAS drive, stick Linux on it, and then schedule syncing to S3.
QR (quick response) codes, from Tristan Roddis. I didn't know QR codes were an ISO standard or that there were different variants. I hope Tristan's laptop has recovered from the water spill :-(
An enlightening presentation called "Take my wallet but don't hurt my face: Teenagers & the social web" by Daniel Morris. Another presentation I learned a hell of a lot from: teenagers don't use mobile internet, don't send MMSs, and visit a bunch of web sites I've never heard of (of course).
"Building a recommendation engine from tagged content": it was great to see the inside of an application exposed in terms of how worldreviewer.com is handling user recommendations. They're taking the tag cloud for users and for activities, and computing a dot product to get a measure. The problem is that if you want to do this real-time the calculations get expensive quickly. The solution being explored is to reduce the space to a fixed-size vector (manually at the moment, but they're looking to do some form of proper feature extraction).
Matthew Somerville's Is Cornwall Part of England? talk was superb. Turns out Cornwall probably has a case for an independent government, but best not to mention it. Statutelaw.gov.uk looks a fun place to spend a few hours.
Behaviour driven development (BDD), which is "TDD done well". Appears to be a good way to structure and express unit tests, and I need to go and check out JDave.

I spoke about Mac OS X Server, which is something no-one seems to use. I'll maybe write up my presentation at some point.

The hosts and sponsors were enormously generous, and the organizers really pulled it all together well. Thank you, Paul, Matt...well, so many people seemed to be making it happen. Check out the blog and the wiki to get an idea of how many people were involved.

The one particular memory I have is, surprisingly, not of the Champagne jelly. It's of sitting in a presentation on a warm summer's night. Through the window drifted sounds of the young and beautiful people of Brighton starting their evening of partying, while inside a group of 10 or so geeks sat around discussing dating and how it could all be "fixed" with a large and elaborate XML schema.

Label: barcampbrighton

d.Construct 2007

noreply@blogger.com (Richard) — Wed, 12 Sep 2007 19:46:00 +0000

Fun and fulfilling is my quick review of d.Construct this year. No, wait. Cheap, fun and fulfilling. That's better. Sure, there are other ways to spend £85 in Brighton, but I doubt you'd get 8 hours of hot user experience action elsewhere.

We've heard the gist plenty of times: As far as the customer is concerned, the interface is the product. Sitting in a room and hearing it in a variety of different ways can only do good. And as it was less technical than the previous d.Construct I attended, I could sit back and soak up the speaker's words.

As far as specifics go, the highlights for me were:

Having a vision statement, or an experience vision, for a product is a good idea: it helps direct you, it helps measure progress, it helps guide you when having to make trade-offs.
Give up competing on features and technology, and instead compete on the experience. That's my reading of Peter Merholz's presentation.

Jane has a more complete write-up of the day, and you'll find all the photos in the usual place.

Labels: dconstruct07

A Comment on Static Typing

noreply@blogger.com (Richard) — Fri, 17 Aug 2007 08:29:00 +0000

There's a line of thought put forward by Neal Ford to help Java developers get over any hang-ups they may have with dynamic typing. If you like compile-type checking of your types, maybe you're worried "that utter chaos would reign if we discard typing". Neil's argument is that: (a) unit testing is good; (b) 100% code coverage is good; therefore static typing buys you nothing:

"If you have pervasive testing, static typing == more typing. The static typing is nothing but a requirement to type extraneous code to satisfy a compiler that isn't telling you anything interesting anymore."

It's a compelling argument ("[a]fter all, it's the tests that tell you if your code [is] correct or not"), but I think it hides more that it reveals. My main concern is that the argument assumes there's the same amount of testing in both the dynamic and static cases. That doesn't seem to be true: if the compiler is catching cases that you need to test for in a dynamic language, then that's more unit tests that need to be written.

Let me phrase this another way:

If you have static typing, dynamic typing == more testing. The dynamic typing is a requirement to type extraneous code to satisfy a code coverage tool that would otherwise have been caught by the compiler.

That's not to knock dynamic languages (I know first-hand how damn productive Perl is): I'll use dynamic languages some times, and static languages other times and admire them for what each can do for me in different situations. And it's not to support the idea that 100% code coverage is an achievable goal in practice. It's just that Neal Ford's in principle argument doesn't work for me.

Java Caching APIs

noreply@blogger.com (Richard) — Thu, 19 Jul 2007 11:11:00 +0000

Last month I attended a superb presentation on JBoss Cache by the project lead, Manik Surtani. Luckily only six people showed up, so were were able to chat to Manik, and really pick his brains on this stuff. Talking to Manik gave me a huge amount of confidence in the JBoss Cache product.

Some background to all of this: we've all probably kept something that's frequently used, and infrequently changing, in memory to save a trip to a database. That's one of the things you can use a cache like JBoss Cache for, but that are other benefits too: the preference for code that's been written elsewhere (many eyes etc); a smart way to do invalidation; a way to distribute a cache between servers; reporting tools... and so on.

JBoss Cache isn't the only solution out there. There are a bunch of APIs you can use to add caching into your application, and maybe one day the Java Temporary Caching API (JSR 107) standard will mean you can use one API across all the implementations. I was wondering... how do they compare. I decided to have quick look at performance for three of the popular ones.

I'm looking at:

JBoss Cache, version 2.0.0 RC3. LGPL license.
JCS: Java Caching System, version 1.3. Apache license.
Ehcache, version 1.3. Apache license.

I have a scenario in mind, where there are a lot of cache hits and only a few cache stores. In particular I wanted to know how the implementations perform under concurrent access, such as a heavily loaded web site: what happens to the cache performance as load increases.

To set the stage here's a graph showing the performance of a work load when using the ye olde synchronized Hashtable, the spanky new ConcurrentHashMap and a "noop" implementation that neither stores nor returns anything. The idea here is just to provide a sanity check of how my laptop performs under increasing load.

Here's what you're looking at. Along the x-axis is the number of threads concurrently accessing the cache. The y-axis is showing normalized throughput (so bigger is better on the y-axis), averaged over 5 runs. What I've done is create a worker that picks a random number, looks in the cache to see if it's there, and if it's not it associates a list against the number. There are only 400 numbers to pick from, and the worker tries this 40,000 times (weird numbers, but it's relevant to the application I have in mind). Each thread equates to one worker, and all the workers share the same cache instance. You can download the source code for this, if you want the details.

So focus on the orange curve in the middle first, which is the graph for my "noop" cache. It's doing pretty much what you'd expect. As the number of threads increases, the machine is able to perform more operations in the same amount of time, up to around 32 threads. After that, performs starts to fall off.

The purple line tells you that you don't want to use a Hashtable for concurrent access. There's a lot of synchronization and waiting happening (probably) as threads compete to access the shared object.

The green line shows you that the concurrent APIs in Java have come a long long way since Hashtable. The ConcurrentHashMap implementation is blazing fast and doesn't seem to suffer too badly under increasing load. In fact, it has a better throughput than my noop code (possibly because noop always causes both a get and a put on the cache).

So much for the baseline measurements. Now let's see what happens with the Java caching implementations:

The first thing I need to say is that these tests are for local, non-replicated, non-transactional caching. I.e., it's the simplest case. I've also only done the bare minimum configuration on the caches to make them work. I'm sure they are tweekable. Also note that the y-axis values are an order of magnitude lower than the baseline graph.

It looks like JCS have really tuned the single thread case, but after that there's not a lot to choose between that and EHcache in terms of performance. Neither seem to suffer under increasing thread load. On the whole, I prefer the JCS API to the EHcahce one.

Don't be overly put off by the JBoss performance curve. Sure it looks a lot lower, but we're talking a total of 300ms difference at 4 threads for a total of 160,000 cache requests: there are bigger considerations that those performance measures.

So the conclusion from all of this is... they're all pretty good. I was hoping one would run so badly I could rule it out, but that's not the case, and I'm going to have to look at features.

For the simple case, I'm going to opt for JCS based on these numbers; but beyond that the numbers tell you nothing of use. For example, if you need transactional support, or want an object cache, then it has to be JBoss as that's the only one with those features. Or you may prefer one API other the others in terms of style.

As a final comment, do not trust these curves for replicated or distributed caches: the issues there in measuring performance add a lot of complexity.

Comments for Logging

noreply@blogger.com (Richard) — Thu, 05 Jul 2007 11:43:00 +0000

Logging can become quite verbose. I found myself writing a method in which most of the lines were logging related, which really clutters the code:

if (source == null)
{
   if (log.isDebugEnabled())
   {
      log.debug("No copy required as no source provided (dest \
               was "+dest.getAbsolutePath()+")");
   }
   return;
}

Possibly I'm over-logging, to try to avoid ever running a debugger. But I was wondering if it'd be possible to do something so that I could turn on logging of comments for a given method, class, package or whatever. So if I'd written....

if (source == null)
{
    // No need to copy as no source directory provided when copying to dest
    return;
}

...and there was some way to say "show me the comments as you execute", that would be as useful as my four lines of debugging. You could even imagine this hypothetical tool could spot references in comments to variables in scope and do a toString on them.

This wouldn't replace logging (you'd still want error, info, fatal, etc), but it might be a useful piece of ease-of-use. From what I can tell Java doesn't preserve comments in the bytecode, so it's not something easy to do. But we have annotations and reflection and AOP and static analysis and assertions and all sorts of stuff, so there's probably some way to make it happen.

Google Developer Day 2007

noreply@blogger.com (Richard) — Tue, 19 Jun 2007 09:20:00 +0000

I was in London for the GD Day 2007. I sat in on some pretty interesting sessions:

Gears does indeed look impressive. I was on the edge of my seat waiting to hear how they'd solved online-to-offline synchronization, but they explicitly backed out of that one: "What we've chosen to do for this initial release is not tackle the synchronization problem [...] overtime maybe there will be something that works for the 80% case, and if that happens I think it's going to come from the community". I can understand that, and it's probably a great opportunity for anyone who can solve that problem. Worth catching the video. I'll also add that because it's open source, I wouldn't be surprised if the plugin was ported to mobile devices (Safari, after all, is already supported, so how far away can a mobile WebKit version be?).
Google have opened up their geolocation API. I think this is absolutely huge for the UK, but right now it returns an error message for any UK address. I learned that (a) it's going to be OK, it's just the post office dragging their feet over the legals; and (b) if you need the data, there's a hack to get it out of Google Local Search.
There was a presentation on porting a JSP site to GWT. The lesson seems to be that you start from scratch, but I did learn you can use Java to implement impressive Javascript behaviors (drag and drop, for example,) without writing any Javascript.

Go watch the videos.

Google London Open Source Jam 3

noreply@blogger.com (Richard) — Fri, 08 Jun 2007 09:03:00 +0000

A couple of weeks ago I was over at the Google London Open Source Jam 3, which was focused on Java. I wasn't sure what to expect, but it turned out to be a series of five minute talks. I'm not sure if that's normal or not for the Open Source Jam series.

If I had to pick out one thing, I'd say take a look at the closures talk. I'd thought closures for Java was on it's way to being sorted, but it turns out there are three proposals with pros and cons:

BGGA (Bracha, Gafter, Gosling, Ahe)

CICE (Concise Inner Class Expressions)

FCM (First Class Method)

So more stuff to read and think about, then.

A really good evening of listening and chatting to some good people, so thanks to those behind it. I think most people were weirded by attending an open source jam and signing an NDA :-/ But let's face it, no-one was going to not sign.

JavaOne 2007

noreply@blogger.com (Richard) — Sat, 19 May 2007 09:19:00 +0000

For the impatient: impressive Seam and JRuby/NetBeans demos; JavaFX doesn't look that crazy after all; performance improvements show promise; quality of the talks were consistently high and better than I recall from previous years; need to look at Groovy and Grails more.

I don't like to make too much fuss about these events, but when I go into the first keynote on the first day of JavaOne, look around and see the size of the place (space for just 15,000 attendees this year), look at the huge screens on stage, with DJ Anon doing her thing, and then the big numbers come out: 6 million developers, 2 billion mobile devices, 11 million TVs, Java on 91% of desktops. I have to admit to being somewhat awed by the event. They do a good show. It's not Steve Jobs, but it's pretty good.

The big announcement this year was JavaFX. This is a blanket term covering authoring tools for "content professionals", an environment for mobile and a scripting language. The authoring tools don't exist yet. JavaFX Script does, and used to be called F3—a technology I've previously described as "nuts". We first heard about F3 at the London Sun Tech Days, and we couldn't understand why it existed when SVG looked to do the same thing in a similar way.

This time, we had Chris Oliver, the author of the language, on hand to explain it. His talk kicked off by asking why Java applications looked so different to Flash applications. Java has the capability to produce all the effects you see in Flash, but they're harder to get at. Step in JavaFX Script which is component scripting of Java 2D and Swing.

And why not SVG? Well, there are lots of similarities between SVG and JavaFX Script (indeed, there's a converter), but the dynamic stuff in SVG is a pain. That's something I can confirm from experience.

In JavaFX Script you can do the dynamic stuff with Java (naturally), but the language also has a data binding feature (functional reactive programming, apparently). As I understood it, elements of the view are bound to elements of the model. This means there's no listening code to write: the model changes, the view updates. There are demos, downloads and documentation over at openjfx.org.

So maybe not so nuts after all. To get that into the browser, though, needs some work. Right now you need to add about 700k to an applet download. The Java Plugin needs some work to make this better (but see the demos over at swinglabs.org; I must get round to trying out Iris). Chris Oliver wasn't taking questions on this area. In fact, we had the impression that he had perhaps been under a bit of pressure recently: after all, the technology had a different name a month or so ago, was a side demo at London Tech Days, and now was pushed way up the conference agenda.

In related news, there is going to be a consumer JVM, presumably targeting RIA, scheduled for early 2008. The focus is on performance, both runtime and download time. There's also work going on to modularize Java (super packages in development terms; super JARs in deployment terms).

The JavaFX Mobile announcement is that Sun have a complete mobile stack which is being offered to OEMs: it's the SavaJe technology on top of Linux. We saw a demo running on a Neo1973, the SavaJe device, and a Nokia N800. Meh... we'll see what comes of it.

While on the subject of mobile, just two notes: I was disappointed to see only one prototype MSA device (SE Z50i), given that MSA has been talked about for a long time; but I had a look at the Nokia 6110 Navigator, and that has a great deal of promise as a phone for location, although no WLAN and only a single processor.

Anyway, JavaFX Mobile is being placed on the cusp between ME and SE:

Probably the other headline news was the support for Ruby and Rails via the JRuby implementation. If you're developing in Ruby (and/or using Rails), you'll want to check out the Toy Show Part 2 from the Friday keynote or take a look at one of the online demos. Either way, you should try out NetBeans 6.

Going into the conference I was wondering which scripting language I should be putting effort into: Ruby or Groovy. Both look to be very capable. After attending the Groovy/Grails One Meet-up, and gaining some confidence from hearing the people behind the language, I think that Groovy might be more useful to me. It has the edge on integration with Java right now. What I didn't get was an intro to Grails. Something I'll have to try out.

Another thing that demands attention is GWT. It has a reasonable widget set, fast client development, they've worked on cross-browser and performance issues and browser memory leak issues. Paul blazed the trail, and we managed to hack on a GWT app for a while, getting it all working, including client-to-server serialization. I've not seen anything that puts me off it yet. (Interesting to hear from Google that they think a page load time of 300ms is the threshold for users to be happy. That's the start up time they target.)

Gavin King's Seam demo was just perfect for getting a feel for what Seam is about. The demo nicely contrasted the assumption that EJB is heavy by having a containter that starts up in the IDE in less than a second. He then continued by developing a Seam web application without the need to restart the container once. Another technology worth checking out.

From other sessions...

The Hibernate integration with Lucene looks like a no-brainer if you need to add text search to an application.
The API for restful web services looks good. I'll take another look when they finish the spec.
I was at a session on deploying and scaling massive digital archive repositories. It turned out to be a vendor pitch to go buy a StorageTek 5800, but it was interesting to learn about the APIs they are building into the disk storage, with hints about this becoming an open standard. Forget files. (This could be related to Project Honeycomb.)
NASDAQ do 100k transactions/second on the Java platform. They're looking to use Java Real Time. Presumably for predictability.
Mark Harriman of Ericsson said they're basing their IMS products on Glassfish. Possibly to be called the Sun Java System Communication Application Server.
When running the Glassfish application server, you can write your site in Ruby, Javascript, PHP and Java. Probably other languages too, via the scripting plugin spec.
I understand that JChav was mentioned in one session. Fantastic.
The only session I was disappointed with was a Jini one. While the language has been changing and going forward, Jini seems (to an outsider like me) relatively unchanged since the first version. Right now to use Jini, you really need to want to use Jini.
The Neal Gafter presentation on closures for the languages was great to sit through. Each time I hear about the proposal I understand a little bit more about it.
The Effective Java Reloaded session (Josh Bloch again) was superb. Note to self: need to understand wildcard capture in generics, and remember that arrays and generics don't play so well together.

So: quite an emphasis on scripting, language changes and richer UIs.

Container Services

noreply@blogger.com (Richard) — Mon, 30 Apr 2007 11:19:00 +0000

You don't need to use all of Java EE to build an enterprise application. Partly because of that I've always been nervous of big EE application servers such as JBoss, Glassfish, OC4J, WebLogic, WebSphere, and instead I've preferred mixing and matching open source libraries and stand-alone components. Or maybe I just don't need the features, or I've not run into scalability or transaction control that containers help manage, or maybe I'm not convinced on the benefits of a "single solution". So although I have ended up using them for some projects, I'm not sold on the need and I've never really thought they make life easier.

But lately I've started to come round to handing some responsibilities off to containers. One responsibility is web user authentication (a.k.a. login). It's a great example of saving time and improving quality, by leaving those aspects to tried-and-tested implementations provided by the application server. Unfortunately, form based authentication in the EE spec isn't flexible enough. Most people I know end up rolling their own solutions, which is a shame. Maybe it'll be usable one day...

There is one good example I've found where you can leave something to the container. If you have an application to deploy, and you want to deploy it on a test machine and a production machine, it's very convenient to leave the database configuration to the container. That is: you build you application (say, a WAR file) and then you can drop it into a container on a QA machine and it'll pick up the connection settings for the QA database; drop it into the live environment and it'll pick up the live database settings. The configuration is in the container (once), not the application (each build), and so Mr Cock-Up is less likely to visit.

Here's how you do it with Tomcat 5.5 and Hibernate 3.0.x:

Copy your JDBC driver and your connection pool into $TOMCAT/common/lib/ e.g., mysql-connector-java-3.0.14-production-bin.jar and c3p0-0.9.0.4.jar for example.

In Tomcat's server.xml, in the <GlobalNamingResources> section add:

<Resource name="jdbc/MYPROJECT-database"
description="My Project Database Connection"
driverClass="com.mysql.jdbc.Driver"
maxPoolSize="25"
minPoolSize="2"
acquireIncrement="1"
auth="Container"
maxStatements="50"
idleConnectionTestPeriod="3600"
testConnectionOnCheckin="true"
automaticTestTable="connection_test"
maxIdleTime="21600"
factory="org.apache.naming.factory.BeanFactory"
type="com.mchange.v2.c3p0.ComboPooledDataSource"
jdbcUrl="jdbc:mysql://localhost:3306/MYPROJECT?useUnicode=true&
amp;characterEncoding=UTF-8&autoReconnect=true"
user="myproj" password="trustno1" />

Adjust parameters to taste. What's going on here, is that the Apache BeanFactory is creating an instance of ComboPooledDataSource and then trying to call setX methods
on the instance from the attributes in this XML snippet. So check the API for your connection pool implementation to see what you can set.

Create the webapp/META-INF/context.xml (or similar) with this content:

<Context>
<ResourceLink name="jdbc/MYPROJECT-database"
global="jdbc/MYPROJECT-database"
type="javax.sql.DataSource" />
<Context>

That tells Tomcat that this web application wants to access a global Tomcat resource called jdbc/MYPROJECT-database.
When Tomcat deploys the WAR this file is copied to TOMCAT/conf/Catalina/localhost/YOUR-WAR-NAME.xml.

If you're using Hibernate, modify the config file to ask the container for the connection:

<property name="dialect">
org.hibernate.dialect.MySQLDialect
</property>
<property name="hibernate.connection.datasource">
java:comp/env/jdbc/MYPROJECT-database
</property>

...ensuring that all the other database connection stuff (username etc) are removed.

That's it. Then when you run, if Hibernate needs a connection it asks Tomcat for jdbc/MYPROJECT-database and Tomcat gives back whatever the instance is set up to provide. Looks like it's doable in Jetty too.

Microformats

noreply@blogger.com (Richard) — Thu, 05 Apr 2007 07:29:00 +0000

Last night's Sussex Geek dinner featured Glenn Jones with a presentation on Microformats. I learned a lot in a short space of time.

My previous impression of microformats was that it was a kludgy way to piggyback contact and event details into HTML—clever and useful, admittedly, but a kludge nonetheless. But I was totally wrong. I recommend grabbing the PDF presentation and taking a look for yourself, but it turns out the attributes used to mark-up XHTML with contact details and the like are, for want of a better word, legitimate. For example, using the class attribute to identify a name of a person is fine: the XHTML spec describes class as good for pretty much any use: "The class attribute can be used for different purposes in XHTML, for instance as a style sheet selector (when an author wishes to assign style information to a set of elements), and for general purpose processing by user agents." Ah, so it's not just for CSS. Ok.

Adding to that are a couple more appealing features: the specs are short, look pretty straightforward to start using, and are based on existing standards such as vcard. The details are over at microformats.org. I'll add "hReview-ifying book write-ups" to my to do list...one day....

For me the most interesting part of the talk was idea of designing URLs so they look like API calls. The example given was along the lines of: http://somesite/tag/creative can be thought of as somesite.tags.getList("creative"). That works for me, and is probably a good rule of thumb for designing URLs. The tricky part comes with queries: how to incorporate do ANDs, ORs, ranges, etc. One option might be to use a builder pattern, such as /tags/creative/startingWith/c interpreted as tags.getList("creative").startingWith("c"). Another option would be convention where by the order of the elements of the URL have a meaning and you just need to know it. The example there would be http://traintimes.org.uk/brighton/london/08:30 means "list trains from Brighton to London, leaving Brighton at 08:30". Stuff to ponder.

Indeed, the evening left me with plenty of stuff to ponder: How does this relate to RSS? How does this tie up with RESTful web services. Stuff to ponder is a good sign, BTW.

Now we all need to go play with the Operator plugin for Firefox.

UPDATE: 19 April 2007, via Jeremy's twitterings, Picoformats look interesting.

Disclaimer: Glenn Jones is the Creative Director at Madgex, which is
Jane's new employer.

Sun Tech Days London 2007

noreply@blogger.com (Richard) — Fri, 16 Mar 2007 12:44:00 +0000

I caught the middle day of the three-day Sun Tech Days in London. Compared to last year, it was busier and felt to me like there was less in-depth technical content. There was too much "Hello London! Are you ready to rock?" and not enough to get rocking about.

Note to the organizers: yes, maybe we are a tougher crowd than the Japanese, but if your idea of crowd foreplay is just lobbing out a beach ball, you've got to try harder. Also: we need better coffee, make sure all the presentations actually match their titles and descriptions... I could go on... to gauge how unimpressed I was with the whole event I left before the free booze came out. And that's obviously a phrase I never thought I'd have to use.

But I like these events. Forget the coffee, forget the cheerleading, and forgive some of the content. What you get is time to think about the tech with like-minded people, and to look and hear about things you don't normally give yourself the change to think about. It's a good thing, and I'd recommend it to anyone.

For me "London's own" Simon Ritter was the star. I caught his presentation on Java 6 and 7 (PDF) which was useful for the explanation of the bounded wildcard stuff in generics.

His demo on music similarity was the best thing I saw all day. It looks to be the Search inside the Music project and the basic idea is to extract features from music so that you can cluster tracks by similarity. Reading from "A model-based approach to constructing music similarity functions" (which is a good review of the field) it looks like they extract features from MP3 data and then use a learning algorithm to classify a track by genre. What you get out is 10 genre likelihood numbers which you can plot by reducing from 10D to 3D (using MDS, which tries to reduce the dimensions in such as way that points that are close in 10D space are mostly close in 3D space too. I'd expect PCA to work too. In other words, don't ask what the labels on the axis are, because there's no easy way to say). Once plotted you can see the clusters of genre and sub-genre, and then take journeys from one genre to another. Cute.

What else...

Netbeans 6 looks like it's getting to be usable day-to-day soon. The profiler looks excellent: so thanks to the guy on the stand who demoed that to us.
I learned a little bit about Java DB (a.k.a. Derby, a.k.a Cloudscape): Mostly I learned (PDF) that it's a proper database.
I don't think I need to worry about BPEL and JBI.
The main features of Java 6 seem to be: Visa L&F; SystemTray stuff; access via Desktop to email, browser; splash screen support; scripting engine; ability to discover available disk space. All dull, but worthy.
Java 7 looks to be up for lots of ease-of-development changes: maybe closures, syntax for list and hash creation, properties. Stuff people have been banging on about for ages. Due mid 2008.
F3...looks nuts.

The event was held in at Central Hall Westminster. Nice building. It somehow seems right that events like this are held in places of worship.

Mac OS 10.4, Apache, Subversion

noreply@blogger.com (Richard) — Thu, 08 Feb 2007 12:42:00 +0000

I recently set up a Subversion server on my Mac OS 10.4 MacBook Pro. It was, amongst other things, a test-run for installing SVN on an XServe. It was somewhat involved, so here's my reminder on how to do this. Hopefully when 10.5 ships a lot of this will just go away.

The goal is to: install subversion, convert an existing CVS repository to SVN, set up Apache to access the repository, set up some access restrictions, and get it running over HTTPS. The starting point is that 10.4 ships with Apache 1.3, and I'd already installed a SVN client from a .dmg, but you need Apache 2 and to install Subversion from source to get the Apache SVN module compiled up, so that really doesn't matter (the Mac OS 10.4 server, BTW, ships with an Apache 2.0 instance for "evaluation purposes", but I'm going to ignore that as we want the latest and greatest Apache really).

My plan: install Apache in to ~/Applications/apache/2.2.4/ and SVN into ~/Applications/svn/1.4.3/ and to convert the CVS repository into a SVN one in ~/Developer/svn/BIG_PROJECT (obviously, adjust to your own tastes).

Here we go...

Step 0: You need the Apple Developer tools installed.

The next five steps are all concerned with cvs2svn.

Step 1: Install DarwinPorts (or is it MacPorts now?). Ports provides a way to download, compile and install open source projects into /opt. The DarwinPorts instructions are pretty clear: grab the Tiger .dmg installer and do the sudo port selfupdate thing. One gotcha here: the installer adds /opt/local/bin to your $PATH by adding a line to .profile. If you've got a .bash_profile already, you may find the .profile file being ignored, so adjust however you like so that /opt/local/bin:/opt/local/sbin ends up at the start of your $PATH. An echo $PATH will tell you if you're good or not after install and starting a new shell.

Step 2: Install Berkeley DB, which you need for cvs2svn later. If you want you can skip this step and discover later if you need it, but I needed it. The command to perform this install is: sudo port install db42. Two confessions at this point. DarwinPorts understands dependencies and will go and get any additional packages required by db42, but I don't know how many that will be for you because I'd already used Ports to install the latest version of Ruby which downloaded lots of packages. The second confession is that at the time I ran the above command, Ports wasn't responding to my proxy configurations and it couldn't download the package from the shell. To get around this I manually downloaded the package (db-4.2.52.tar.gz) in my browser and put it in /opt/local/var/db/dports/distfiles/db4/. If you have problems like this, running the port command with -v can help.

Step 3: Add Berkley DB support to Python, which is something required by cvs2svn as described by plasticfuture. The steps here are to grab the Python Berkeley DB bindings source (bsddb3-4.5.0.tar.gz for me), extract it and run:

$ python setup.py build --berkeley-db-libdir=/opt/local/lib/db42
      --berkeley-db-incdir=/opt/local/include/db42

$ python setup.py install --berkeley-db-libdir=/opt/local/lib/db42
      --berkeley-db-incdir=/opt/local/include/db42

Step 4: You're now set to download and set up cvs2svn. I had a tar.gz of an existing CVS repository, which I unpacked to ~/Developer/oldcvs/BIG_PROJECT. You need to create a cvs2svn.options file to set up the conversion. I was going to create a new SVN repository, and you can download the cvs2svn.options I used (more or less... search for [CHANGE ME] in that file and fix it up for your own usage).

Step 5: Run the conversion with: cvs2svn-1.5.1/cvs2svn --options PATH_TO/cvs2svn.options. If you have a "big project" this may take a while, so while it's running you might want to type the following to be told when it's done: say Conversion Complete. I had a handful of errors about files existing in the repository and also in the Attic. I rm-ed the Attic versions of the files and re-ran the command.

Now that we have a converted CVS repository as a SVN repository we can continue and set up the server. Before that, I just did a sanity check with svn co file:///Users/richard/Developer/svn/BIG_PROJECT into /tmp or wherever to ensure there were no problems with the conversion (remember: I already had the SVN client installed, but if you don't you'll be able to do a test checkout around step 7, below).

Step 6: Install Apache. There's a pretty nice Apple Developer article on Subversion and Apache which is worth a read. All I did in the end was to download Apache, unpack it, and run:

$ ./configure  --prefix=/Users/richard/Applications/apache/2.2.4 
     --enable-mods-shared=most --enable-ssl --with-mpm=worker
     --without-berkeley-db --enable-proxy

The --without-berkeley-db is ironic after the BDB hoops we've jumped through above. The usual make and sudo make install and you're done. Start apache with sudo ./apachectrl start from the ~/apache/2.2.4/bin directory and check that http://127.0.0.1/ is running and then stop Apache. (Warning: if you just run apachectrl you'll pick up the Apple-installed Apache 1.3 instance, so remember to use the ./ or use the full path to the apachectrl for the Apache you've installed).

I ran into a few segmentation faults (or bus errors, depending on which user you run as) when starting up Apache. It turns out there are a couple of modules that fail to load or other issues in the configuration file. One useful way to see how far Apache is getting is to validate the configuration with debugging on: sudo bin/httpd -t -e DEBUG.

We'll configure Apache later, but first...

Step 7: Install subversion. Again, the Apple Developer document above is helpful here. I downloaded the subversion source from subversion.tigris.org and ran...

$ ./configure --prefix=/Users/richard/Applications/svn/1.4.3  --with-ssl 
     --with-apxs=/Users/richard/Applications/apache/2.2.4/bin/apxs 
     --with-zlib --enable-swig-bindings=no --without-berkeley-db 
     --with-apr=/Users/richard/Applications/apache/2.2.4/ 
     --with-apr-util=/Users/richard/Applications/apache/2.2.4/

...followed by make and sudo make install. If you don't already have a Subversion client on your $PATH, you can add ~/Appliations/1.4.3/bin now.

If you're going to go on to install Continuum or anything else that might require http or https support from the Subversion client, you'll want to also download the Neon library as documented by The Punk Engineer and indeed also by the Subverison INSTALL text file.

Step 8: Configure Apache. Now comes the Apache voodoo to turn on SVN access. Edit ~/Applications/apache/2.2.4/conf/httpd.conf. (BTW, TextMate is quite nice for all of this, partly because it recognizes and colours Apache configuration, but also because from the shell you can run mate . to open all files in a nice folder view.) At the bottom of the Apache config add the following:

<IfModule dav_svn_module>
<Location /repos>
DAV svn
SVNPath /Users/richard/Developer/svn/BIG_PROJECT
</Location>
</IfModule>

Start Apache and in your browser you can hit http://127.0.0.1/repos to browse the source.

Step 10: Add authentication. Go back to httpd.conf and add in some form of authentication and access control. Here's what I did...

<IfModule dav_svn_module>
<Location /repos>
DAV svn
SVNPath /Users/richard/Developer/svn/BIG_PROJECT
AuthzSVNAccessFile conf/svn-access.conf

Require valid-user
AuthType Basic
AuthName "Subversion Access"
AuthUserFile /Users/richard/Developer/svn/svn-pwd-file
</Location>
</IfModule>

There are two changes here. The first is a svn-access.conf file that controls what users or groups can do to parts of the repository. This file looks like this:

[groups]
dev = paul, gareth, richard, andrew, jono

[/]
@dev = rw

It's saying there's a group of users call "dev" to have read+write access to the "/" path in the repository (i.e., the lot). I wasn't able to find a great deal of documentation on this, but thank you to Richard Tesh for spotting the archlinux.org guide.

The second part of the Apache configuration is the basic authentication part, which is straight-forward non-weird Apache stuff. The command you need is htpasswd: run it with -c the first time only to create the svn-pwd-file. Use this to create a username and password for each of your "dev" users.

Quit your browser, restart Apache, and try to hit the repository again: This time you'll be prompted to login.

Step 11: Set up permissions on SVN repository. A gotcha here is that the Apache user (daemon, group daemon) needs write access to the repository, so you need to decide what you're going to do there. One option is to chmod the repository, which is what I did.

Step 12: Finally you'll want all of this to run over HTTPS, and so you'll need to set up a certificate for Apache. I'm using a self-signed certificate. There's some great documentation for this, including the ever-useful macosxhints.com. I created a folder in my home directory called keys, and from there I ran the following:

$ /System/Library/OpenSSL/misc/CA.pl -newca
$ openssl genrsa -des3 -out webserver.key 1024
$ openssl rsa -in webserver.key -out webserver.nopass.key
$ openssl req -config /System/Library/OpenSSL/openssl.cnf -new 
     -key webserver.key -out newreq.pem -days 3650
$ /System/Library/OpenSSL/misc/CA.pl -signreq

You'll get prompted to enter a bunch of information. Use your hostname as the common name.

When those commands are done, you'll have a set of files and a subdirectory in ~/keys. Now to configure Apache, by adding the following the httpd.conf:

<IfModule mod_ssl.c>
Listen 80
Listen 443

<VirtualHost _default_:443>
SSLEngine on
ServerName 127.0.0.1
SSLRequireSSL

SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM: \ 
     +LOW:+SSLv2:+EXP:+eNULL
SSLCertificateFile /Users/richard/keys/newcert.pem
SSLCertificateKeyFile /Users/richard/keys/webserver.nopass.key
SSLCACertificateFile /Users/richard/keys/demoCA/cacert.pem
SSLCARevocationPath /Users/richard/keys/demoCA/crl
</VirtualHost>

</IfModule>

Restart Apache and you can now hit https://127.0.0.1/repos/BIG_PROJECT or enter that into whatever tool you use to access subversion, such as an Eclipse plugin. You'll be told the certificate isn't recognized, but just "continue" or "OK" or "accept" your way past that. BTW, if you try to hit the non-HTTPS URL, you'll be forbidden.

Aaaaaaaaand we're done. Apart from backups, more sensible security, better file naming, multiple repositories, automated builds....

Thank you: Paul, Pragmatic Version Control Using Subversion and all the HOW TOs I've not mentioned.

OpenStreetMap

noreply@blogger.com (Richard) — Sun, 21 Jan 2007 17:03:00 +0000

Mikel Maron's fireside-style talk at the Sussex Geek Dinner was a great introduction to OpenStreetMap: "providing free geographic data such as street maps to anyone who wants them". If you need to know anything about this area, Mikel is clearly the person to talk to.

Jane's blogged about the talk, so I'll skip that and just mention that the talk inspired me to take another look at the Bluetooth GPS unit we have. Turns out it's easy to get the GPS unit running with a MacBook via the terminal: pair with the device, and then in Bluetooth Preferences, select the device and check the serial ports. In my case, I didn't have to do anything, as the Mac had already set it up as a serial device. From the shell, you can then connect to the device...

$ screen /dev/tty.BT-GPS-3404F4-BT-GPSCOM-1

...and it starts spewing data:

$GPGSA,A,3,20,01,11,17,,,,,,,,,2.6,2.4,1.0*34
$GPRMC,164654.000,A,5049.4955,N,00008.8944,W,0.06,87.03,180107,,*2C
$GPGGA,164655.000,5049.4957,N,00008.8945,W,1,04,2.4,26.0,M,47.0,M,,0000*71
$GPGSA,A,3,20,01,11,17,,,,,,,,,2.6,2.4,1.0*34
$GPRMC,164655.000,A,5049.4957,N,00008.8945,W,0.09,24.77,180107,,*2B
$GPGGA,164656.000,5049.4959,N,00008.8946,W,1,04,2.4,26.3,M,47.0,M,,0000*7C
$GPGSA,A,3,20,01,11,17,,,,,,,,,2.6,2.4,1.0*34

These sentences aren't too scary, once you have the NMEA descriptions. And as it's Bluetooth and a serial connection, you can imagine it's not too bad to get that data live on a phone, without the need for JSR-179 or JSR-293. Great fun.

But back to the talk, I should mention that the big thing OpenStreetMap has is a great refresh rate. Compared to other mapping organizations, OpenStreetMap contributors can, for example, spot new streets in their area as they are being built, meaning the OpenStreetMaps maps can be years ahead of the other players. On the other hand, organizations like the Ordnance Survey have a great advantage in precision. But what you need depends on what you want to do, and for OpenStreetMap the focus (I think) is about navigation.

Links to follow up: New Popular Edition (OS maps out of copyright); Neogeography; Free The Postcode; KML; Geonames; OS information; GPSBabel; Yahoo! allows satellite imagery in OpenStreetMaps.

Generics in Comments

noreply@blogger.com (Richard) — Wed, 13 Dec 2006 19:03:00 +0000

For a project I'm involved in that is stuck using JDK 1.4, I've found doing this soothing:

List/*<Stuff>*/ getStuff();

It's documentation, and gives me the hope that one day I can remove the comments for this particular project.

Next Net

noreply@blogger.com (Richard) — Tue, 05 Dec 2006 20:10:00 +0000

Thanks to Ewan Silver and others for organizing the Next Net meet up. It's sort of a London Jini Group, but more general: a gathering of people interested in distributed (utility computing, grid computing, massively scalable, resilient) systems.

I unfortunately missed a large chunk of Dan Creswell's presentation on Utility Computing with Amazon EC2. What I could figure out rekindled my interest in these technologies.

For example: On my way over to the meeting I was wondering about how best to distribute database load somehow or other. When I walked into the meeting, the presentation was discussing doing-away with things like relational databases to obtain big scalability, and shifting mindsets away from Java EE. Ah. So think different.

Why would you bother? I think I'm finally starting to get utility computing. If you've worked in an environment where budgets for hardware involve a lot of hoop-jumping, it suddenly makes sense to be able to call on a dynamic network of resources at the moment you need it. What's more, it's looking like it's cheap. There's effort in building systems that take advantage of that kind of an environment, but if the cost savings are there, the pressure will come to build them that way.

It's interesting stuff, and now I have a larger-than-usual reading list as a result: MapReduce, Bigtable, Chubby, Google File System, Hadoop the
10th Jini Community Meeting papers and the Amazon Web Services Developer Connection docs.

The next Next Net will be Thur 18 Jan 2007 at Brunel.