Back in the day The Well had a text based conference system, you used dial in, then telnet and later ssh to their server and interacted with other members through a text system called PicoSpan. Eventually things moved to the web and it became a lot more forum like. The thing that I really loved was that in the web version of the forums there was a command line. You could type many of the same commands into the web CLI as you would into the Unix one and have the same effects. Posting, searching, jumping through conferences. It was the web with the CLI power for those who wanted it.

The browser is more and more our interface to all things online and frankly it sux a bit, I want the CLI speed for accessing the Web sites that I like. I’ve created a PHP system I called cmd ages ago that simply routed a command like “guk greenwich” to the Google UK search engine with results restricted to those from the UK. There are of course various online tools that does the same but I found that their ‘book’ keyword would search Amazon US while I wanted UK so I just did one that I can tweak to my liking.

Recently thanks to Googles widely hated changes to their Search UI simply redirecting to Google searches with keywords filled in just was not enough anymore. I want web search back the way it was before they made it suck. So I do what hackers do and wrote a Ruby based pluggable search system. You can see a screenshot of it here showing a Google search.

What you’re seeing here is the oldskool-gcse plugin in action. It uses the Google JSON API to query a Google Custom Search Engine and format the results in a way that does not suck. The Custom Search Engines are quite nice as you can customize all sorts of things in them like which sites to exclude, which to favor, limit results to certain countries or languages allowing you to really customize your search experience. The only down side to the GCSE approach is that Google limits API calls to 100 a day, for me that’s enough for searching but ymmv.

Using this method of searching can have some privacy wins, Google recently announced merging all their online accounts into one and will have all your online activity influence your searches. I wasn’t too worried since by then I had already written Oldskool and will simply use a different Google Account to access their search API than the one I use to read my work mail for example. Simple effective win.

My default search in oldskool is a GCSE that resembles a normal Google search but I can also search for “puppet exec” and oldskool will route that request to a specific GCSE that bumps the official Puppet Labs docs to the top, exclude some annoying things etc. So oldskool is a single entry frontend to many different GCSE backends is quite powerful.

As I said it’s plugable and I’ve written one other plugin that uses my Passmakr gem to generate random passwords. I can just search for pass 10 to get a 10 character password:

Writing your own plugins is very easy and I hope to see ones that queries Redmine instances or other internal databases that you might have using the Oldskool framework to display all the data in one handy place.

It retains the most basic feature of simple keyword base redirects, so I can search for book reamde to get Amazon UK book results instantly.

Config is through a simple YAML file:

---
:google_api_key: your.key
:username: http_auth_user
:password: http_auth_pass
:keywords:
- :type: :gcse
  :cx: you_gcse
  :keywords:
  - :default
- :type: :gcse
  :cx: your_gcse
  :keywords:
  - puppet
- :type: :url
  :url: http://amazon.co.uk/exec/obidos/search-handle-url/index=books-uk&field-keywords=%Q%
  :keywords:
  - book
  - books
- :type: :password
  :keywords: pass

This sets up 2 GCSE searches – one marked as my default search – and the mentioned book search and one that uses the password plugin I’ve shown above.

It needs no writable access to the webserver it runs on and it’s all managed by Bundler and Sinatra – perfect for hosting on the free Heroku tier.

As this is effectively my Web CLI I want it integrated in as many places as possible. I use a lot of desktops – 3 regularly – so the browser is my unified UI to all of this. Your instance will publish OpenSearch meta data which will make it seamlessly integrate into Firefox, Chrome, IE, Gnome DO, Gnome Shell and many many other places.

Here’s Firefox search box the first time you browse to a new instance:

And here is Chrome, you do not even have to add it just start typing the URL to your instance and press tab, the URL bar transforms into a Oldskool search box magically. You can add it permanently and make it default by right clicking on the URL bar and choosing Edit Search Engines….

The code is in my GitHub – Oldskool, Oldskool GCSE and Oldskool Password. I will blog again tomorrow or on another day about creating your own plugins etc.

I am often surprised by the sort of questions asked in the forums or on irc around open source projects - it just feels as if people are going out of their way to inflict pain and suffering upon themselves by trying to find the most awkward and most complicated way to do things. So how can we better help these people ? We dont need to save them or anything as drastic like that, its just a case of being able to show or explain that there might be a better way.

The first thing that I've started now doing, when asked a strange question is ask the person 'What are you really trying to achieve?'. You might be amazed how many times the answer has nothing to do with the question being asked. Try to establish what the end goal is, and in many cases its clear that the person has been lead astray by random posts on the internet, some of which are perfectly fine in their own context, but can be quite a kludge outside that context.

Establishing, clearly what the goal is before advice or opinion is thrown at people will always result in a better overall experience. And to the people spending their time in the irc channels, web forums and mailing lists helping others out : must respect. You guys are the ones making the idea of Communities and Open Source work.

- KB

Looks like I will need to get a Visa again to visit Belgium for Fosdem 2012. This is starting to get a bit irritating now, six times I've been to Fosdem and every time they have asked me to come in for an interview before they give me a visa; once again ? Surely by now it should be possible to get onto the visa-by-post process.

- KB

In what turned out to be a productive holiday hacking with languages I’d not used before, I got round to writing some coffeescript on node.js. This was more to do with scratching a personal itch that pure experimentation. I had a play with Janky (Github’s Jenkins/Hubot mashup) but found it a little opinionated on the Jenkins side, but the campfire integration is excellent. Looking at the Jenkins commands in hubot-scripts though I found those even more opinionated.

The magic of open source though is you can just fix things, then ask nice people if they like what you’ve done. I set about writing a few more general commands and lo, the’ve been quickly merged upstream.

These add:

• A command to list all your Jenkins jobs and the current state
• A command to trigger a normal build
• A command to trigger a build with a list of parameters

This was made much easier by first looking at the previous Jenkins commands, and then looking at other scripts in the hubot-scripts repository. The best way of learning a new language/framework is still on the shoulders of others.

I’ve got a few other good ideas for Jenkins related commands. I want to add a filter command to the jobs list, both by name and by current state. For longer running jobs I also want to report whether a build is currently running. And then maybe get information about a specific job, like the last few runs or similar. Any other requests or ideas most welcome.

For running ad-hoc commands across a small number of servers you really can’t beat Fabric. It requires nothing other than ssh installed on the servers, is generally just a one-line install and requires next to no syntaxtic fluff between you and the commands you want running. It’s much more of a swiss army knife to Capistranos bread knife.

I’ve found myself doing more and more EC2 work of late and have finally gotten around to making my life easier when using Fabric with Amazon instances. The result of a bit of hacking is Cloth (also available on PyPi). It contains some utility functions and a few handy tasks for loading host details from the EC2 API and using them in your Fabric tasks. No more static lists of host names that constantly need updating in your fabfile.

Specifically, with a fabfile that looks like:

#! /usr/bin/env python
from cloth.tasks import *

You can run:

fab all list

And get something like:

instance-name-1 (xx.xxx.xxx.xx, xxx.xx.xx.xx)
instance-name-2 (xx.xxx.xxx.xx, xxx.xx.xx.xx)
instance-name-3 (xx.xxx.xxx.xx, xxx.xx.xx.xx)
instance-name-4 (xx.xxx.xxx.xx, xxx.xx.xx.xx)
instance-name-5 (xx.xxx.xxx.xx, xxx.xx.xx.xx)
instance-name-6 (xx.xxx.xxx.xx, xxx.xx.xx.xx)
instance-name-7 (xx.xxx.xxx.xx, xxx.xx.xx.xx)
instance-name-8 (xx.xxx.xxx.xx, xxx.xx.xx.xx)
...

And then you could run:

fab -P all uptime

And you’d happily get the load averages and uptime for all your EC2 instances.

A few more tricks are documented in the GitHub README, including filtering the list by a regex and some convention based mapping to Fabric roles. I’ll hopefully add a few more features as I need them and generally tidy up a few things but I’m pretty happy with it so far.

• Jay from TDDium got in touch – there’s another outsourced CI service in town [link]
• Bamboo 3.4 is released, with Windows EC2 support. [link]
• Cloudbees released their Jenkins survey [link]

Happy new year!

Vendor news, December 2011 is a post from: The Build Doctor. Sponsored by AnthillPro, the build and deployment automation server that lets you release with confidence.

My annual End of Year update. See previous years: 2010, 2009 and 2008.

Last year I said I’d:

• Run another marathon.
• Write some more.
• Enjoy my new job and life in SF.

Well, I’ll score that a two out of three. Yes, I ran the marathon, but I stopped halfway (oh, the hills!); I didn’t really blog any more; but yes I enjoyed life in SF. Sometimes life makes things difficult, but my amazing wife and I got through it together.

We started the year in an empty apartment, sleeping on the floor, not knowing anyone in the city. By the end, I feel we’ve settled in and started to make friends.

This year I have:

• Had cataract surgery. (again)
• Passed my Driving Test. (again)
• Been visited by both mothers and both brothers.
• Launched a Follow Button and Web Intents.
• Tinkered with various aspects of twitter.com.
• Seen Buzz Aldrin, Snoop Dogg, Lady Gaga, and various others come in to the office to see where we work, and thank us.
• Been given free pretzels, trainers, earplugs, ice-cream, online music and books, just because I work at Twitter.
• Not been in a plane.
• Not been to England.
• Had but a single weekend away, on the other side of the bay.
• Become a dad.

Next year I plan to:

• Take it easy.
• Enjoy my new life and job new job and life in SF.

And I think that’s enough for 2012.

I nearly always try and grab some time over Christmas to try something new and this year I’d been planning on spending some time with Clojure. I have several friends who are big fans, but dipping in and out of a book hadn’t really worked. What I needed was an itch to scratch.

I stuck with a domain I’m pretty familiar with for this first project, namely building a little web application. It renders a web page, makes HTTP requests, parses JSON into native data structures and does a bit of data juggling. Nothing fancy or overly ambitious, I was mainly interested in picking up the syntax, understanding common libraries and getting something built. Here’s what I’ve got:

Jenkins has various API endpoints, but most dashboards I’ve seen concentrate on showing you the current status of all the builds. This is hugely useful when it comes to the simple case of continuous integration, but I’ve also been using Jenkins for other automation tasks, and making extensive use of parameterized builds. What the dashboard above concentrates on is showing recent builds for a specific job, along with the parameters used to run them.

Overall it was a fun project. Clojure made much more sense to me building this application than it had from simple examples. The Noir web framework is excellent and proved easy enough to jump into and simple enough that I could read the source code if I was interested in how something worked. The Leiningen build tool made getting started, downloading and managing dependencies and running tests and the application itself easy.

What I didn’t find particularly appealing was the lack of a strong standard library coupled with the difficulty of tracking down suitable libraries. JSON parsing, dealing with HTTP requests and date handing are very common activities in web programming and all needed me to jump around looking at the best way of dealing with the common case. I settled on clj-http, chesire and using Java interop for date formatting. clj-http suffered from having lots of forks on GitHub to navigate through. I started with clojure-json before discovering it had been deprecated. And neither clj-time or date-clj appeared to support unix timestamps as far as I could tell from the source. Throw in some uncertainty over the status of clojure-contrib that isn’t well documented on the official site and it needs some effort to get started.

The working code for this is already up on GitHub and I’d be interested in any Clojure experts showing me the error of my ways.

Happy $EVENT is a post from: The Build Doctor. Sponsored by AnthillPro, the build and deployment automation server that lets you release with confidence.

(Kushal Pisavidia has been working on enhancements to XFD – it’s only fair that he gets to do the release announcement)

The beginnings of XFD
A little over a year ago on the 4th of May 2010, XFD was born through the work of Chris and Julian. Since that date it’s gone from a minimally viable status indicator, to a tool which numerous companies use with a number of CI servers to keep their build status visible. The current supported list includes:

• Hudson
• Jenkins
• CI Joe
• CruiseControl
• Team City (via a plugin)
• CCNet, cc.rb, Go (via a plugin)

We’re always on the lookout for new servers to add.

Move to CoffeeScript

For those not familiar with it,

CoffeeScript

is a little language that compiles to JavaScript. It takes the “good parts” of JavaScript and exposes it through a cleaner syntax. Making it easier to stay away from the bad.

However CoffeeScript isn’t all about syntax sugar. It’s about being more confident that the code you write is correct the first time, while knowing that tests are less brittle and can be changed in a matter of keystrokes. This has made developing XFD a lot simpler.

HTTP Auth
HTTP Auth is by far one of the most requested features. In this release we’ve added support for both Hudson and Jenkins with more support planned.

Better feedback
In making XFD we sought to make a highly visible build status indicator. Related to the concept of visibility is feedback. This is best illustrated by an analogy to what everyday life would be like without it. Imagine trying to play a guitar, slice bread using a knife, or write using a pen if none of the actions produced any effect for several seconds.

XFD seeks to make this feedback step clearer for build jobs, but didn’t make it obvious what the application was doing at every step. Now, it’s incredibly easy to follow along and get clear information about what XFD is doing as you use it.

Increased accessibility
The emphasis on Red/Green for build status meant that colourblind users may have difficulty viewing the status of their builds. To counter this we’ve added universal symbols in addition to the colours to make this clearer for everyone. The different symbols are used to indicate different levels of importance or urgency with no need to vary their colours.

We hope you enjoy this new release, and let us know what you think!

XFD 0.2.16 Released is a post from: The Build Doctor. Sponsored by AnthillPro, the build and deployment automation server that lets you release with confidence.