Tag Archives: technology

On future web

Out of the tragedy of Aaron Swartz’s untimely departure earlier this year are some of his outputs that endure. One, RSS, through his contribution in the RSS-DEV Working Group, is a cornerstone of syndicated web content distribution, although with the demise of Google Reader, one wonders how long RSS or its successor, the Atom Syndication Format, will survive in a new model of published content. That’s perhaps for another post. Markdown, co-written with John Gruber is a format I’m using to write this text. And there are others.

A more recent output was the unfinished e-book The Programmable Web, available for free download from the publisher Morgan & Claypool’s site. In it, he lays out a brief but amusing history of the web, and, although it’s a first draft and was never updated, he also paints an attractive—if utopian—vision of what the web could be, and why the Semantic Web would be an important part of that.

Layer Cake

It reminds me of why I sometimes fiddle with semantic web technologies in my spare time. While it’s a sometimes impenetrable mass of specifications dealing with knowledge models and representations, it is a critical body of knowledge for implementing and traversing a machine-readable web. To implement this vision is to bootstrap the web to a totally new level of unimaginable utility, as the current Web 2.0 functionality would be to our pre-1990 selves. (I’m also cynical enough to believe it would also have unimaginable negative consequences, but then we’ve so far survived the nuclear age too, despite the odds.)

I’m not a futurist or misty-eyed sci-fi fanatic but the combination of knowledge models, open linked data, discoverable APIs, and agents leads to some predictable outcomes. The products can be as pedestrian as the Internet refrigerator that arranges its own restocking and will inevitably display an ad each time you open it. Or as provocative as an undetectable wearable computing fabric that “augments” its owner’s reality, and could change the very nature of “self” and “other”. I think it is the future. Laugh or cry at Google Glass, it’s a pointer to where we’re going. That’s both exciting and challenging, but inevitable. Assuming we survive our more pressing current and future geo-political challenges of course.

The image above comes from here.

Small steps

In a classic example of what goes wrong in bigger environments, I wanted to knock up a quick tool to solve a problem. I decided to use Clojure because the solution involves data transformation from XML to JSON, so a functional approach makes sense. I also want to improve my Clojure skills, which are on the amateur side.

Leiningen is the natural lifecycle tool. I created my project, updated my dependencies in project.clj to the latest version of the midje testing tool, and in fine TDD style wrote a quick sanity test, ran lein midje and got a green response. Good work.

After some reading up on XML zippers in Clojure, I then made the timeless error of overconfidence by taking a big leap forward and writing a simple functional test that required a number of implementation steps, including coding and updating components and tools. Pretty soon, I was in Leiningen hell, getting meaningless exception stack traces, thrashing around trying different versions of tools and libraries, and commenting out increasingly large pieces of code until I found my issue—in about the second line I’d written. Lots of time wasted, no value delivered.

Moral to the story: when you’re in unfamiliar territory, you move faster if you take small steps.

Semantic appendix

As an appendix to the Semantic Scratchings post, a couple of additional diagrams to illustrate the process.

 

The first shows a simple graph of a mythical contact. There are additional properties that can be added, but it shows the core split into organisation, vCard, and FOAF relationships.

 

0image

 

 

The second shows the workflow from entering contact details, to displaying query results on a web page. The icons are standard UML analysis boundary, control, and entity classes, and it also shows the domains in which each entity sits.

 

Image

 

I hope it gives some idea of the steps I’m using. Not optimised, certainly, but it’s reasonably decoupled across standard protocols and APIs.

 

Semantic scratchings

Over the last few weeks, while I’ve had some time to myself, I’ve been scratching an itch by going deeper into semantic web technologies with an exploratory project of sorts. I guess it’s paying off in that it’s raising as many questions as it answers, and it’s also giving me some in-my-own-head street cred by both getting down dirty with writing, building, and deploying code, and thinking about things like ontologies and formal knowledge capture. If that floats your boat, read on. It’s a long entry.

Background

I won’t go into the nature of the itch; the scratching is always more interesting. I’ve been using sem-web tools to get a handle on my address book. It’s always annoyed me how siloed that information is, and how much the data could be augmented and improved by linking it with other data, both public and private. The same problem on a much larger scale has provoked the rise of the Linked Data movement. I credit a large part of inspiration for my little project from Norm Walsh’s paper from 2002 and related blog posts.

It’s a journey, possibly without end, but I’ll describe a little about where I’ve been, with some technical detail where useful. I’ll throw in links for the less familiar. Ask me if you want more detail.

It goes without saying that the organisations that hold your contact information, like Google, LinkedIn, and Facebook are building this sort stuff anyway, whether it’s using semantic web or not. Even if I wanted to, there’s unlikely to be a web startup in this.

So Far

To date I’ve built three main building blocks that do the following:

  1. Perform an extract-transform-load from my OSX Contacts app into RDF triples.
  2. Load the triples into a cloud-based RDF triple store.
  3. Run queries on the triple store using a web front end.

None of this is in itself even remotely revolutionary or exciting in the regular IT world, and the sem-web people have been doing it for years too. A triple store is just a specialised graph database, one of the many NoSQL entrants. It’s just that I’ve just chosen to hand-build some of the pieces, learning as I go.

Implementation

First a word on implementation language. I really wanted to do this in Clojure because its elegance and scalability should complement RDF. Sadly, the tools weren’t as mature as I would have liked, and my Clojure skills weren’t up to developing them myself, although others have shown the way. I considered using Node.js but again, I didn’t feel that there wasn’t enough there to work with. Not so with my old friend Ruby, where there are a wealth of useful libraries, and it’s a language I’m competent (but not expert) in, and encourages fast development. I also get to use RSpec and Rake. For a worked example, see Jeni Tennison’s post on using 4store and RDF.rb.

ETL

The ETL step uses a simple AppleScript to extract my Contacts app database into a single vCard file. I could use a Ruby Cocoa library to access it, but this was quick and easy. Moreover, vCard is a standard, so anything source data I can get in vCard format should be consumable from this step on.

The question was then how to transform this information about people into RDF. FOAF as a vocabulary doesn’t cut it on its own. Luckily, the W3C has already addressed this in their draft paper, and have a reference model (see image below). It leverages heavily off some well-known vocabularies: FOAF, ORG, SKOS, and vCard in RDF.

People Example

There’s also a big version.

I wrote a Ruby class to iterate through the vCard entries in the exported file and assemble the graph shown above using the excellent RDF.rb library, generating nodes and edges on the fly. These were pulled into an RDF graph in memory and then dumped out to a file of triples in the Turtle format. This can take some time: well over a minute on my i5 MacBook Pro grinding through nearly 300 contacts. It currently generates around 7000 triples (an average of 23 per contact).

An entry for an example contact currently looks like this in Turtle:

@prefix : <http://alphajuliet.com/ns/contact#> .
@prefix db: <http://dbpedia.org/resource/> .
@prefix dbo: <http://dbpedia.org/ontology/> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix gldp: <http://www.w3.org/ns/people#> .
@prefix net: <http://alphajuliet.com/ns/ont/network#> .
@prefix org: <http://www.w3.org/ns/org#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
@prefix v: <http://www.w3.org/2006/vcard/ns#> .

:m254028 a org:Membership;
     org:member :person-6D9E0CBF-C599-4BEC-8C01-B1B699914D04;
     org:organization :org-example-corporation;
     org:role [ a org:Role;
         skos:prefLabel "CTO"] .

:org-example-corporation a org:Organization;
     skos:prefLabel "Example Corporation" .

:person-6D9E0CBF-C599-4BEC-8C01-B1B699914D04 a foaf:Person;
     net:workedAt [ a org:Organization;
         skos:prefLabel "Oracle Australia"];
     dbo:team db:Geelong_football_club;
     gldp:card [ a v:VCard;
         v:adr [ a v:work;
             v:country "Australia";
             v:locality "Sydney"];
         v:email [ a v:work;
             rdf:value "jane.smith@example.org"],
             [ a v:home;
             rdf:value "jane.smith12345@gmail.com"];
         v:fn "Jane Smith";
         v:note "Met at Oracle";
         v:tel [ a v:cell;
             rdf:value "+61 412 345 678"],
             [ a v:work;
             rdf:value "+61 2 9876 5432"]];
     foaf:account [ a foaf:OnlineAccount;
         foaf:accountName <http://www.linkedin.com/in/janesmith12345>],
         [ a foaf:OnlineAccount;
         foaf:accountName <http://twitter.com/janesmith12345>];
     foaf:homepage <http://www.example.org/>,
         <http://jane.smith.name/>,
         <http://www.example.org/janesmith/profile>;
     foaf:knows [ a foaf:Person;
         foaf:name "John Smith"],
         [ a foaf:Person;
         foaf:name "Marcus Smith"],
         [ a foaf:Person;
         foaf:name "Alice Jones"];
     foaf:name "Jane Smith" .

The UUID in the foaf:Person node is generated and retained by the Contacts app, so I have a guaranteed ID over the lifetime of the contact.

Because of the stream processing of the vCard entries, there is no way of setting up inferred relationships between existing items of data on the first pass, such as identifying explicitly that I know all my contacts. Fortunately, that’s what SPARQL is good at, so I use the following slightly awkward query:

CONSTRUCT {
    ?a foaf:knows ?b .
} 
WHERE {
    ?a a foaf:Person .
    ?a foaf:name "Andrew Joyner" ; 
        gldp:card ?c1 .
    ?b a foaf:Person ; 
        gldp:card ?c2 .
}

This generates a set of inferred triples that I can add into the main graph.

Import

I was using a local version of 4store up until very recently. It’s a no-fuss, solid, and open-source triple store with REST and SPARQL endpoints. However, I am doing development across different computers via Dropbox, and I wanted to centralise the data. One option would have been to set up an instance of 4store or maybe stardog on Amazon’s magic cloud. Fortunately, there is a new cloud triple store called Dydra that keeps it very simple, and I was kindly given a private beta account with some free storage.

Currently I’m manually clearing and adding the entire graph each time it gets updated, but that’s ok while it’s in development. Eventually, this will be scripted through the Dydra Ruby API gem.

Web Query

The core of querying an RDF store is SPARQL, which is as SQL is to relational databases. It even looks similar. I’ve set up a localhost front end using Sinatra, Markaby and a Ruby SPARQL client to apply queries, some with user inputs, to return “interesting facts”, like:

  • List all the organisations and how many contacts work for them, shown as a line chart using the D3 Javascript visualisation library.
  • List who knows who across the graph.
  • Display a D3 “force” graph of my network based on foaf:knows relationships (that’s me at the centre of the universe)

foaf:knows graph

  • Who are all the people in a person’s circle, i.e. the subject and object of a foaf:knows predicate
  • Who works at a given company and their mobile number
  • Who don’t I have an email address for?

As I said, fascinating. It’s very basic but it’s a start. I want to start visualising more information with D3, such as the broader social networks.

As a matter of style, I’m applying actual SPARQL queries rather than using a pure Ruby approach that sparql_client encourages. It just seems to replace one format with another without adding any useful abstraction.

Deployment

I’m managing the code under Git, and I’ve deployed the code base onto Heroku, both as a cloud education exercise, and so I can access it everywhere. However, because it contains personal contact data, I can’t make it public.

Data model

Being an organised person, I’ve filled in a number of the relationships in Contacts to reflect spouses, friends, children, pets, and so on. These all get mapped using a triple such as ajc:person-xxxx foaf:knows [ a foaf:Person; foaf:name "John Smith" ] .. The square brackets result in a blank node that can to be linked up later to the actual person based on inference rules. I don’t want to assume at this stage that there is only one “John Smith” in my address book. I know three Steve Wilsons for example.

Along the lines of Norm Walsh’s approach, I’ve also added “custom” relationships such as foaf:knows and net:workedAt, which get mapped into a set of triples during the transform process.

I’ve also played with adding my own RDF triples explicitly as notes in my contract entries, to give maximum flexibility. I use the format rdf: { ... } to enclose a series of Turtle triples, using my standard RDF prefixes, and transform them into real triples.

I’ve started an ontology to capture entities and relationships that don’t seem to exist elsewhere or are too complex for my needs. One example is to capture where someone worked before, using a net:workedAt property to map from foaf:Person to org:Organization. It highlights a major question in my mind around provenance and versioning of information (see next section).

Provenance and versioning

Clearly, one of the potential shortcomings of the system so far is that the quality of the data is determined by my efforts to keep my address book accurate and up to date. I did take some steps to pull in data from my LinkedIn connections, but quickly hit the API transaction limit in trying to pull down all the info on my nearly 600 connections, so it’s on hold for now. I do fantasise about LinkedIn (and Facebook) having a SPARQL endpoint on their data, but I suspect they would rather be at the centre of my social network, and not my little triple store.

Assuming that I did import contact data from LinkedIn and Facebook, where people manage their own contact details. I’d want to capture the source or provenance of that information, so I could decide the level of trust I should place in it, and resolve conflicts. Of course, there’s a W3C Provenance vocabulary for expressing that. The bigger question is how to capture the dynamic nature of the data over time. A person works for this company this month, and that company next month; how do I best capture both bits of information? The Provenance ontology provides a basis for capturing that in terms of a validity duration of a triple, but not necessarily at a single point in time, like a snapshot. I’d like to say, for example: “this triple is valid right now”, and then change it in the future, and say the same thing at a later time. It’s not as precise as a duration, but I may neither have start and end date, nor care.

Updating triples

Another question is around the mechanics of updating triples. At the moment I clear the store and do a full ETL from Contacts each time, but that’s clearly not workable longer-term. If something changes, I want to be able to insert a new triple and appropriately handle the old one by deleting it, changing it, or adding additional triples to resolve any conflict. That requires me to know what triples are already there. I can see an involved solution requiring me to query the existing store for relevant triples, determine the steps to do the update, and then apply the updates to the store. The SPARQL Update spec provides for conditional deletes but I need to work it through to see how to do it. There’s a parallel here to the problem of maintaining referential integrity in a relational database.

Of course, these are all very answerable questions, I just haven’t got there yet or seen existing solutions. Updates in later posts.

Future work

It’s still developing, and a long way off being useful. There’s also a bunch of related technologies I want to play with over time. Amongst the backlog, in no particular order…

  • Add some more useful queries and visualisations
  • Include hyperlinks in the returned data so I can start browsing it
  • Link to information in public stores such as DBpedia, i.e., real Linked Data
  • Set up a continuous deployment chain using VMs and Puppet, maybe on AWS for fun
  • Import LinkedIn connection data
  • Add provenance metadata with [PROV][]
  • Add more specific contact relationship information with the REL vocabulary
  • Leverage other ontologies such as schema.org and NEPOMUK
  • Look at using reasoners or inference engines, such as Pellet or EulerSharp

References

Apart from all the links, a few good pointers for learning more.

On the cloud

cloud

The IT world is in a period of dramatic transition. That may be trite to say when it’s been in transition all its life, but things are shifting more rapidly. The evolution and adoption of new technologies is also raising business’s expectations on being able to execute on new business models faster, reduce investment costs, and attract and keep customers. 

The usual suspects are at play here, the ones you keep hearing about: cloud, data analytics, mobile, social networking, etc.

The globalisation of IT through XaaS cloud services (X being whatever you want these days) is  forcing commoditisation and a necessary maturing of the market to deliver the necessary features and service levels at acceptable cost. That’s not a bad thing for either cloud buyer or seller. If you consider the vertical technology stack of (from the bottom) data centre, hardware, networking, infrastructure, platform, middleware, applications, business process, and business service, it’s clear there are huge opportunities for all sorts of vertical and horizontal integration plays, niche players, value-added services, and consolidation. It’s easy to believe that most enterprise services for all but the largest organisations will be outsourced within five years. How that plays for large shrink-wrapped software vendors is a major question, but I can’t see the advantages of buying large suites of software on capex over a pay-as-you-go opex model surviving the market shift to increasingly trusted (and secure) clouds.

It’s a classic disruptive play. Clouds may have been immature once, but they will increase share as they grow upwards, pushing the big players into alternative models or premium niches. Companies  don’t want servers, they want a business enabler.

Cloud will also therefore change the way business is done. The pace of business will increase. Markets will evolve faster. Three-year or even financial-year strategic plans won’t survive sustained contact with the market. The pace of change will only be constrained by the rate that we ourselves can digest change, and perhaps technology will have a solution for that too.

But I’m jumping ahead. “Cloud” right now is increasingly about service delivery, ITIL-style. SLAs will become the mantra of those who pay for services, cloud or otherwise. The challenge has already been articulated elsewhere: how do IT departments maintain their relevance as their company’s legacy systems migrate into the ether, and the business starts buying their own IT per user per month?

 

 

Enhanced by Zemanta

Enterprise APIs

Below are some highlights from articles on enterprise API trends from O’Reilly, Programmable Web, and ZDNet.

  • Enterprise APIs are apparently becoming mainstream as organisations open their silos of data for internal consumption.
  • Enterprise APIs need to align with the business strategy. Most enterprise APIs are now “owned” by the business, not IT.
  • There are increasing numbers of third-party API providers, such as Datafiniti, whose success depends on fostering a developer community around their API, and offering other value-added services.
  • The load on enterprise APIs is unpredictable, so the service implementation needs to be elastic.
  • REST and JSON are already the majority, with SOAP and XML declining.
  • OAuth, both 1.0 and 2.0, is the default for securing APIs, especially for so-called “three-legged” authentication scenarios. Where it competes, OpenID is on the way out.
  • One quick win for implementing internal enterprise APIs is analytics, including the tactical sort I talked about before.
  • SOA, cloud, and enterprise APIs will effectively merge as concepts, and become “the way we do things”.

My thoughts on some of this:

Externally-accessible enterprise APIs make customers do the work, avoiding second-guessing the functionality customers need, and any subsequent delay in deployment. By so doing companies also reduce the cost of doing business, and increase their transparency. More strategically, it can encourage customer investment in building to their API and increasing “stickiness”. Monitoring the use of those APIs (via analytics) can provide a significant source of aggregate and individual customer information.

Among the tradeoffs of opening up enterprise data is of course data security. Another risk is business model security if, for example, substantial IP is visible through the API design and data structures.

Stickiness from implementation investment implies some amount of coupling. SOA and enterprise APIs still require developers to design with them, and generally do some amount of coding. Crucially, they require developers to bind to the API-defined data at design time unless the API is either carefully designed or very simple.

Ideally, an enterprise API should be standardised, with consistent request and response protocols that can be hard-coded across alternate providers, or dynamically discoverable by software agents, either at build or run-time. Even with standard REST approaches such as OData, dynamic or late binding without human intervention requires a level of discoverable semantic knowledge beyond a WSDL-like syntactic description. This is one of the reasons that the Semantic Web was developed, but it seems that mainstream developers are finding it overly complicated. Perhaps. However, for looser coupling and more agile use of enterprise data, automated selection and use of APIs will require a semantic “understanding”, and the significant existing semantic web work will be used and extended.

By example, a CSV file of tabular data, even if expressed in JSON or XML as a structured key-value map, can have machine-comprehensible metadata about the meaning of each column attached. The semantic web already offers the ability to describe each data fields not only in terms of a meaning defined in a common ontology such as UMBEL, but also expected data formats and relationships and dependencies between the fields, using a combination of RDF, Turtle, and OWL. It does require a more formal definition of an enterprise API, but a lot of it could be auto-generated.

I’m exploring. Feel free to agree, comment, or worse.

On Data 2.0

I’ve been thinking lately about what is—for want of a better term—sometimes called Data 2.0. My thoughts have been triggered by internal discussions at my workplace Unico about the direction of our Designer Analytics™ solutions. Caveat: I’m not a content, data, or even database specialist or architect.

Thinking about what could be termed tactical analytics leads to a bunch of follow-on thinking about where that data comes from, what it is, how useful it is, how much there is, how to present it, how to trust it, and so on.

Taking a cue from Michael Porter, my mental model of analytics is about building a value chain, with the following steps:

  1. Capture from data sources such as sensors, logs, feeds, and events
  2. Aggregation; involving filtering, transformation, compressing, often lossy
  3. Storage and indexing into large repositories such as data warehouses, relational databases, key-value stores, and content management systems
  4. Query and retrieval
  5. Analysis, perhaps with statistics, clustering
  6. Presentation of the output; sorting, categorising, summarising, filtering
  7. Visualisation

It occurs to me that tactical is a key word here. As one of my colleagues puts it, our analytics solutions are about “late binding” of the captured un- or semi-structured data, as opposed to the very early binding of structured data in traditional (Data 1.0?) BI, data warehouse, ETL-type solutions, where hundreds of pre-written, pre-ordained management reports are the norm.

By comparison, data 2.0 concerns sets, often large, of unstructured or semi-structured data. Late binding requires that as little data as possible should be thrown away or interpreted, and the downstream activity of query and retrieval is dominated by (often text-based) search, as a more agile approach to extracting sense and meaning from all the data. And because it's tactical, the analytics solution can be a framework for measuring RoI for a particular change project. Baseline at the start, monitor along the way, measure the final improvement, then focus attention elsewhere.

Late binding of data implies loose coupling of systems. SOA is already about looser coupling than pre-existing point-to-point approaches, but there is scope for looser coupling still in things like mashups, using published or enterprise APIs, as tactical responses to getting coherent meaning from disparate data sources. This area is being opened up by approaches like REST, standards such as the OpenData Protocol, nifty products such as ifttt, and ultimately, the Semantic Web.

There’s a lot more to think about here.

Goodbye Android

<p>I&#39;ve made the decision to give up on my Android phone and have ordered an iPhone 4S. My HTC Desire is 18 months old. It is a nicely-engineered phone, with decent battery life, good screen, and is fast enough, but I&#39;m tired of its limitations. The last OS update was to Fro-Yo a year ago, and there are no prospects of it ever being upgraded again. The limited onboard RAM means I can&#39;t install any more apps, and each incremental update to the existing ones only eats more memory, often forcing me to remove yet another app from my ever-decreasing list, despite the ability to store many (but not all) apps on the SD card. There are also intermittent issues with the 3G data connectivity that make it frustratingly unresponsive at critical times. </p> <p><img src="http://cdn.cbsi.com.au/story_media/339301153/htc-desire_1.jpg" alt="HTC Desire" /></p> <p>Android isn&#39;t bad either; it&#39;s fully-featured, and has good sharing options and a decent app base, but the UI is a little under-baked, it encourages an inconsistent user experience in apps, and the OS leaks through too often. Google integration, while good, isn&#39;t so compelling any more, in a large part due to the diminishing gap in innovation in its cloud-centric apps.</p> <p>Compare that to my 12-month old iPad 1 that I type this on. It still gets updated OS versions, and even with 16GB has enough RAM to buy new apps. In contrast, no point my getting interested in new Android apps. </p> <p>In iOS land Siri isn&#39;t yet a must-have but it will certainly improve, and iCloud will start replacing an amount of the Google cloud. The iPhone will play nicely with all my other Apple gear, and although I worry about lock-in to the Apple ecosystem, it&#39;s the best technology in town and the aesthetic works. Sorry, Google and HTC. You need to sweat the small stuff more.</p> <p><br /></p>

Agile explorations

While settling into my new city and looking for a job, an amount of my reading lately has been in the field of Agile. That is, of course, with a capital A, because this adjective is way too fashionable to require a noun.<p /> Although Agile (and XP before it) has always resonated with me, I have a history of working, even successfully, in rather un-agile companies. Although I have introduced both XP and Scrum in previous organisations, I&#39;m still wary of buying a one-way ticket on the Agile train, to coin a dodgy metaphor. While it seems pretty clear that Agile — as defined in the motherhood words of the Agile Manifesto — has become the most effective way of delivering quick returns on software development, it is not universally clear that the Agile approach has automatic application to other business activities. Agile evolved out of software development because of most modern software&#39;s recognised properties of complexity, market dynamism, requirements uncertainty, and so on. Do these same properties exist in other fields at a level at which Agile will bring the same benefits? Is there indeed a more universal Agile mindset or analytical lens that can be used for other areas of value creation?<p /> Trivially, the answer to the first question is yes if you consider fields like design, advertising, and music, where collaboration and rapid feedback definitely take a front seat to hierarchy and rigid process. I don&#39;t know if it&#39;s true, but maybe it&#39;s possible that some of the Agile concepts were inspired by successful practices from those fields.<p /> To the second question of a broader Agile mindset, there is certainly movement in this direction. Three books on Agile that I&#39;ve been reading lately are:<br /><ul><li><a href="http://www.amazon.com/Management-3-0-Developers-Developing-Addison-Wesley/dp/0321712471/">Management 3.0</a>, by Jurgen Appelo</li> <li><a href="http://www.amazon.com/Agile-Project-Management-Creating-Innovative/dp/0321658396/">Agile Project Management (2nd Ed)</a>, by Jim Highsmith</li><li><a href="http://www.amazon.com/Succeeding-Agile-Software-Development-Using/dp/0321579364/">Succeeding with Agile: Software Development with Scrum</a>, by Mike Cohn</li> </ul>My crude mental model now is something like this.<p /><div><div class='p_embed p_image_embed'>

Image

Each of the three books essentially focuses on one of the above areas. From a first pass on each, Mike Cohn's book is a very practical manual of insights and best practices in forming and maintaining successful Scrum teams to deliver software. Clearly, a lot of practical experience of success and failure has gone into this, and we can at least avoid falling into some of the same holes. Although, human nature being what it is, we still will.

Jim Highsmith's book is more dense; it embeds a generic Agile implementation approach in a complementary project management framework, appropriate to a pre-existing mature business context. It can therefore be used to interface and buffer hierarchical command-and-control business managers on one side to agile implementation and delivery folk on the other. Electrical impedance matching between the two mindsets, if you will. Importantly, it includes good coverage of agile governance, scaling agile, and portfolio management, which are essential in larger organisations.

Jurgen Appelo's book is more about applied psychology and complexity theory than software as such, and works well as a complement to the other books. He frames Agile as a logical response to managing complex adaptive systems, such as team software development, and dicsusses how Agile values can be expressed at a management level. It's a wide-ranging and very readable text and updates an amount of traditional management and leadership thinking for the Agile Enterprise.

For me, even though I'm less than half way through it, Jurgen's book makes clear that Agile's scope is not limited to software development. Anything sufficiently complex, malleable, and people-centric can benefit from an Agile approach, whether that is delivering consulting services, a marketing campaign, starting a (non-software) business, restructuring a company, launching a product, or, yes, just building a web site. I've even used it in my job hunting. 

Speaking of which, I'm currently available, and itching to put more of this into practice. Let's do coffee.