I love my job. Fact.

But some days are easier than others. If you’ve been reading my tweets this week, you’ll have sensed a certain frustration.

Don’t worry. None of this is your fault.

But here’s what happened.

Over the weekend it had occurred to me, while I was asleep at about 6am – seriously, this happens – it occurred to me that I’d forgotten to submit my latest test fixes before I merged my branch. This would screw up the branch for the weekend, for the whole team, and delay the deploy on Monday. Bad news.

As soon as I woke up, I popped open the laptop, preemptively apologised to all and sundry – as is the English way – and proceeded the fix the problem. It turned out my fixes only fixed half the problem. And it’s the weekend.

So, I cheated. I knew this code was in a hidden section, and wouldn’t be live when deployed. I could just comment out the tests, ensuring the branch would pass even if the code didn’t work properly.

First thing Monday morning, I returned to the problem. I looked back over the logs to find the faulty test and dutifully kicked off a test run.

And this was the start of my problem.

It passed.

I was a little surprised. There were failures last week, I could see there were failures. Maybe somebody else fixed it. I check the logs. Nope. I run the tests again.

It passed.

I should mention that running the tests isn’t instantaneous. It’s a complex test suite we’re running here. We’re currently migrating our server code away from Ruby (we’re always migrating to faster systems). In the interim, we’re running tests against both systems. The various compilation and startup sequences, combined with the real browser used, and the queues of other developers waiting on the system, gives us an average runtime of about 15 minutes.

It’s optimised to run a suite of tests, rather than individual tests, so both cases take around the same amount of time to run. In fact, we have some excellently comprehensive test suites with thousands of tests, and they can all be run in about 15 mins. Running single tests is still slow, because the task can’t be distributed.

From this point on, I’ll indicate the test timings like this: (15).

I can of course run a test locally. Again, this can take time. Usually less time – let’s say 5 minutes or so. It takes less time because I already have the servers running locally.

So my tests are passing. Usually, a cause for joy. In this case, a worry. Why did they fail last week? I dig out the code from last week, and run the tests (15).

It fails, as expected.

Well, that’s something. I diff the code. And here I win the Goof of the Week award. Prize chump right here. The tests are passing because I commented them out.

Duh.

I go to lunch and try to apologise to my own brain cells for letting them down.

I kick off a real test run (15). It fails. Good. But weird – I swear these were working once. I test the Ruby code instead (15). It passes.

Interesting. So I’ve found a difference. This is weird because I haven’t actually changed any of the server code. It shouldn’t be possible to be different here.

What’s worse is that we can’t easily test the new code locally. It takes longer. I run them (10). They pass. I run the Ruby tests (5). They pass. I run the new tests again, to be sure it’s not random. Pass (10). Pass (10).

I spike the code full of debugger statements. Run the tests (15). More debugging (15). Add screenshots (15). Everything is the same between systems, except that the new tests are failing.

” is not true”. Thanks, error messages.

Hmm. I look back at the screenshots. While the systems show the same screens, there’s something I didn’t expect. When you’re a new user on Twitter, we show a little helper module at the top of the page. I’d forgotten we did that. It also loads asynchronously.

Aha. Here it is. The new code runs so much faster that this module loads earlier. This explains the difference. I go home, happy that I’ve established the problem.

Next day, I make allowances for the helper module. I pull out the debugging code, happily run the tests (15) one last time, and prep the branch for shipping.

It fails.

Bollocks. I unprep the branch and put the debugging code back in. I run the tests (15).

It fails. Same debugging result as before.

That’s weird. I was so convinced the module was the problem. I hack the code and remove the module entirely. Another run (15).

It passes.

Ok, so it’s the module. No question. I just got the allowances wrong. I tweak them, and run (15). Fail. I add debugging (15). Nothing. And again (15). No.

This is crazypants. I am so not happy.

I try to reproduce locally. Passes (10) every (10) time (10). No amount of debugging and breakpoints will help if I can’t reproduce the error.

I look back to the screenshots. Aha. No images in the module.

I knew the error was caused by an bad height calculation. And the missing images are (whips out an onscreen ruler), yes, the images are 137px – exactly the difference.

Eureka.

Our images don’t load on the testing machines because they’re insulated against the outside network. That makes sense. Locally, we can access the images, which must be why they pass. It’s perfect.

Furthermore, our testing machines are using Firefox, which won’t render broken images, while I’m using Chrome locally – which does. This would account for the height difference.

I curse and laugh, since this quirk of Firefox has bitten me many times in the distant past.

To fix the test, I simply break all the images – to be consistent. I prep the branch for shipping, throw it out to the test cluster one last time, and go and get a fifteenth cup of coffee.

It fails (15).

What. The. Fuck.
What. Thefuck.
Whatthefuck.
Fuck.

How can this be? I had this nailed.

Worse than that, the tests is now passing occasionally. When I run the tests on the test cluster, I run them about sixty times in parallel. This ensures we don’t get “flaky” tests, which sometimes pass and sometimes don’t. Flaky tests are notoriously disruptive and hard to fix.

Sometimes, all you can do is try it yourself. I open a browser. I downgrade it to the exact same version as our test cluster. I’ve done this before, but I’m doing it again. I run the tests (10). I click around. I fiddle with the code. I break the images, to match. Refresh, refresh, refresh.

Hey, what was that?

I refresh again. And again. There it is.

Firefox won’t render a broken image – it hides it away completely. I knew that. But what I’m seeing here is a flash of broken image. There’s a flash of a broken image in this helper module, before Firefox hides it away.

If my code is making calculations just at that moment, it’ll get a different result than, say, half a second later.

I don’t get my hopes up again. I tweak the test. There’s actually a little css property:
-moz-force-broken-image-icon:1;
This will prevent Firefox from hiding the broken images. I run the tests. All the tests (15).

Pass.

I prep. I ship. I smile.
I move on to the next thing.

(btw, if testing tools is your kind of thing, and you want to help me make things faster, we’re hiring and I will buy you cookies)

I throw another log onto the fire. I’m not sure we really need it – it’ll be warm enough all night long. Out here in Palm Springs, it’s only bearable when the sun sets anyway. The fire simply gives us something to watch while we wait.

Bill Gates is due for dinner. Most of the other guests have already arrived, so he should be next. Or the one after.

I poke the fire. Sparks fly. It’s been a few good hours now, and I still haven’t decided.

Do I shake his hand, or kick him in the fork?

—

This was about ten years ago. I’d never suggest or condone violence in any form these days. Hell, even then I wouldn’t really. So let’s moderate the language.

Do I shake his hand heartily, welcome him like an old friend, thank him for all he’s done for my career, and propose my next big idea? Or do I ignore him, snub him, turn the other cheek?

I should clarify my position. I don’t have dinner with billionaires. That doesn’t happen. I’m really not that kind of guy.

I’m here on holiday. Vacation, if you will. I’m visiting an old friend from London, who is originally from Perth, who is currently working as a hot air balloon pilot in Palm Desert. And in the evenings, he works as a parking valet at The Reserve, an exclusive golf club in Palm Springs. And, since I’ve got nothing else to do, I’m here to help.

I say “parking valet” to emphasise that I’m using the American meaning of “valet”. Someone who parks your car. Not the English meaning, which you might be familiar with from Downton Abbey, of someone who dresses you in the morning.

To make this slightly more bizarre, I can’t actually drive. No license. Never even sat behind the wheel. I’m opening the car door, welcoming them, opening the restaurant door, that kind of thing. Oh, and I get to drive the golf cart, ferrying my fellow valets back from the car park. This is a nice place – you expect your valet to be fast. This is a classy place – even the valets have a little nook with a fireplace.

When I arrived, they told me Bill Gates was booked for dinner. Yes, that Bill Gates. That one.

Obviously I was somewhat in awe. He was, I believe, the richest man in the world at the time. And I’d been working with his software for years. My job, my life, was founded upon it.

And yet, that brought so much trauma. I was out here for a much-needed break. I’d been fighting poorly implemented APIs, buggy applications, incomprehensible error codes, absent or irrelevant documentation, and that blasted comedy paperclip. In no small way did I hate this software as much as I loved it. I should snub this man for all the suffering he put me through.

But what a wasted opportunity. I should be pitching an idea. I’ve got maybe 20 seconds with the most powerful man in technology. I’ve got to plant a thought in his head, one that will distract him over dinner, so that he’ll emerge with a warm smile and an offer of partnership.

All I needed now was the idea.

Ideas are funny things. After a few pints, or several sleepless hours, your mind is full of amazing, wonderful, life-changing, world-changing, revolutionary ideas. When you’re put on the spot, nothing.

Fast forward to the future, and hindsight brings the answers. I could’ve pitched Twitter. Facebook. Myspace. The cloud. Groupon. Well, maybe not Groupon. Instagram. The iPhone. I could’ve begged him to continue development of IE.

If you were put on the spot, right here, right now – could you pitch? What’s your idea? Are you ready?

I wasn’t ready, but that was ok. Bill never showed up.

The other guys were fine with it. Apparently he’s not a big tipper.

Since then of course, he’s donated most of his wealth to charity. Seems like a generous tip to me. I’d heartily shake his hand. Or fist bump. Or whatever he’d prefer.

And he tweets. I build the software he uses every day. That’s pretty awesome.

Maybe it’s a good thing he didn’t show.

While Puppet may get all the glory, Facter, the hard working information gathering library that can, seldom gets much exciting new functionality. However with the release of Facter 1.7 Puppetlabs have standardised and included a couple of useful facter enhancements that make it easier than ever to add custom facts to your puppet runs.

These two improvements come under the banner of 'External Facts'. The first allows you to surface your own facts from a static file, either plain text key value pairs or a specific YAML / JSON format. These static files should be placed under /etc/facter/facts.d

$ sudo mkdir -p /etc/facter/facts.d

# note - the .txt file extension
$ echo 'external_fact=yes' | sudo tee /etc/facter/facts.d/external_test.txt
external_fact=worked

$ facter external_fact
worked

At its simplest this is a way to surface basic, static, details from system provisioning and other similar large events but it's also an easy way to include details from other daemon and cronjobs. One of my first use cases for this was to create 'last_backup_time' and 'last_backup_status' facts that are written at the conclusion of my backup cronjob. Having the values inserted from out of band is a much nicer prospect that writing a custom fact that parses the cron logs.

If that's a little too static for you then the second usage might be what you're looking for. Any executable scripts dropped in the same directory that produce the same output formats as allowed above will be executed by facter when it's invoked.

# scripts must be executable!
$ sudo chmod a+rx /etc/facter/facts.d/process_count

$ cat /etc/facter/facts.d/process_count
#!/bin/bash

count=$(ps -efwww | wc -l | tr -s ' ')
echo "process_count=$count"

$ facter process_count
209

The ability to run scripts that provide facts and values makes customisation easier in situations where ruby isn't the best language for the job. It's also a nice way to reuse existing tools or for including information from further afield - such as the current binary log in use by MySQL or Postgres or the hosts current state in the load balancer.

While there have been third party extensions that provided this functionality for a while it's great to see these enhancements get included in core facter.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

My blogging break has been so long, I feel like a vampire emerging from the grave in a Hammer Horror film.

I’m interrupting my relentless working day to announce that I’ll be at:

• ASWEC 2013 – I’ll be presenting the Neo4j Tutorial on the 4th of June
• Agile Australia – I’ll be doing a talk that mixes up Cloud, PaaS, Devops and NoSQL.  Bingo!
• I’ve proposed a talk for DevOps Down Under in July
• I’ll also be at Graph Connect San Francisco

Back to flogging Ruby code.

Update: for various reasons, both the struck out appearances are cancelled.  Back to flogging ruby code.  (I do actually use flog).

Conference appearances, 2013 is a post from: The Build Doctor. Sponsored by AnthillPro, the build and deployment automation server that lets you release with confidence.

Earlier this month, at the Cloud 2020 summit, in Las Vegas, a group of service providers, vendors, buyers and pundits met to discuss the future of cloud infrastructure. One of the tracks was on the economics of infrastructure services, and included John Cowan from 6Fusion, whose cloud metering technology is a key piece of the puzzle if cloud is to be treated as a commodity, James Mitchell, founder and CEO of Strategic Blue, and former Morgan Stanley commodities trader, and Joe Weinman, author of the fascinating book Cloudonomics.

Out of this track emerged some interesting debate around the viability of the whole notion of a cloud futures market. Jonathan Murray, EVP & Chief Technology Officer at Warner Music Group, followed up with an article presenting his perspective, from the skeptical end of the spectrum. This article presents a view from the optimistic side.

I’m indebted to John Woodley, investor and board advisor at Strategic Blue, for his insights and comments. John is a former Managing Director at Morgan Stanley, where he was co-Head of non-oil commodities for EMEA. He was the first person to be hired by a Wall Street bank from a power utility, becoming a founder member of the Morgan Stanley electricity trading desk. As one of the world’s foremost authorities on commodity market development and the pricing of non-standard pseudo-commodity products, his perspective is fascinating.

The challenge

Jonathan explicitly self-identifies as one who is unconvinced about the prospect of a futures market for cloud. His article is fairly long, and his argument well made. The core point he puts forward is essentially: “There is no volatility or friction so we don’t need intermediation.”

Is there friction in the cloud market?

Jonathan doesn’t explicitly define friction, but I think he means that in today’s world, primary buyers can easily find primary sellers, and as such there’s no need for an intermediary. This is a fair point. In the old world, this was impossible. In a market without intermediaries, in order to get the best price, you need each of the buyers to make as many calls as are there are sellers. Let the number of buyers in the market be ‘b’ and the number of sellers be ‘s’. In a tiny market, with 10 buyers and 10 sellers, you need b*s calls in order to achieve the best price, or 100 calls in total.

However, in a world in which buyers are also resellers, the number of calls needed in order to secure the best price becomes much larger - somewhere around the factorial of the number of buyers plus the number of sellers - (b+s)!. In our tiny market, (10+10)! is 2432902008176640000. If one entity in the middle takes calls and keeps note you only need b+s calls and not (b+s) factorial. This is a conventional argument for the existence of a market - it offers huge savings. On these grounds, it’s hard to imagine Jonathan thinks there isn’t a market, a forum for price comparison, but certainly in today’s world the web is that intermediary near as dammit is to swearing. Jonathan correctly describes markets as places where supply is matched to demand, and indeed, we use markets because they have proven better than manual or -so far- computational planners in correctly matching supply and demand efficiently.

There is, of course, very significant friction in the cloud market, per a different definition. Look at the way cloud providers sell their services. They want long-term commitment, in their own currency, by credit card. By contrast, the cloud user wants minimal commitment, the best flexibility to move should a better deal emerge, or a better technology offering appear. They’d rather not have to keep a credit card on file, and in many cases, for example in India, currency fluctuations make a huge impact, and they’d greatly prefer to pay in their own local currency. In this situation, an intermediary who buys from the providers on terms which suit the provider, and sells to the buyers on terms which suit the buyers is very attractive.

Is there a need for intermediation?

Let’s examine whether there’s a need for intermediation against the assertion that there is no volatility. I think that assertion is false. It strikes me as similar to the assertion that there was no petrol price volatility in the US in 1979 or more recently in the Northeastern US after Hurricane Sandy, because the posted price at the pump did not change. The reality of course is that the underlying value changed so radically that people would queue for hours to get the scarce commodity and even threaten each other with violence to get precedence.

To be sure we have not seen that in cloud yet but we have seen scarcities. John Woodley cited an exemplary case with which he was particularly familiar. The Morgan Stanley IT department was approached to lend space at a datacenter because a major movie producer could not get enough capability on the open market to render a movie before the all-important Christmas release date. Had the IT department called a Fixed Income trader before agreeing to the deal I am fairly sure the producer would have had to pay more in cash or kind than a few T Shirts for the deal.

So, I have to ask: Is there no state of the world in which a sudden demand for many thousands or millions of instances might occur? Are there that many sitting idle? No, a far more productive and interesting line of thought is how the demand surge might occur and how to be in the right place at the right time to benefit. For an interesting example, consider the events surrounding Silver Thursday, in 1980, in which the Hunt Brothers had hoarded a third of the world’s supply of silver, inflating the price by a whopping 700%. At the same time, anecdotally, a certain bank held for some strange reason a large part of the silver smelting capability, and thus were able to charge greatly increased prices for people wanting to melt down the family silver, to sell as ingots. In John Woodley’s view, it is tremendously likely that we will see scarcity and volatility in the cloud markets, and a wealth of opportunity and demand to intermediate.

What about contractual preference?

I touched on this when discussing the idea of friction in the cloud market. A second but no less important driver for intermediation was not addressed in Jonathon’s article. In certain markets, the contractual preferences of buyers and sellers diverge dramatically. In such cases the insertion of a financial intermediary is very helpful and both parties are prepared to pay the rather minimal cost such intermediation causes. An example that is blindingly obvious can be found in electricity. Power plants are only used when the demand is sufficiently high to cover the operating expenses of the power plant. Power plants are built on borrowed money. A power plant owner needs to finance the debts used to build the plant, and so wants a fixed payment per month from their customers. By contrast, a retailer of electricity simply wants to be charged on a per use basis, and wants nothing to do with fuel cost pass-through, or a fixed monthly fee. Think about the way datacenters are funded. It’s exactly the same model - why do you think cloud providers offer such a huge discount for a commitment to usage? It’s the same! In the electricity markets, intermediation provides a welcome service, in which the intermediary is the buyer to every seller and the seller to every buyer, ironing out the contractual wrinkles, and benefiting the market as a whole. There’s already evidence of demand for this kind of service in the cloud market today.

Summary

The fundamental point Jonathan makes in his argument is that because there is no volatility or friction, there’s no need for intermediation. However, there is already evidence of price fluctuation and scarcity, and in terms of the delta between the way buyers want to buy and sellers want to sell, there is clearly friction. The cloud market behaves like and looks like a blend of the electricity and coal market, both of which are highly intermediated and heavily traded. Yes, it’s early days, but it seems very likely indeed that we’ll see the same kind of behavior in the cloud.

Over time parts of your puppet manifests will become unneeded. You might move a cronjob or a users in to a package or no longer need a service to be enabled after a given release. I've recently had this use case and had two options - either rely on comments in the Puppet code and write an out of band tool to scan the code base and present a report or add them to the puppet resources themselves. I chose the latter.

Below you'll find a simple metaparameter (a parameter that works with any resource type) that adds this feature to puppet. As this is an early prototype I've hacked it directly in to my local puppet fork. Below you'll see a sample resource that declares a deprecation date and message, the code that implements it and a simple command line test you can run to confirm it works.

# sample puppet resource using :deprecation

  file { '/ec/cron.d/remove_foos':
    ensure      => 'file',
    source      => 'puppet:///modules/foo/foo.cron',
    deprecation => '20130425:Release 6 removes the need for the foo cronjob',
  }


  $ sudo vi puppet-3.1.1/lib/puppet/type.rb

  newmetaparam(:deprecation) do
    desc "
      Add a deprecation warning to resources.

      file { '/etc/foo':
        content     => 'Bar',
        deprecation => '20130425:We no longer need the foo'
      }

      The deprecation comes in two parts, separated by a :
      The date is in format YYYYMMDD and the message is a free form string.
    "

      munge do |deprecation|
        date, message = deprecation.split(':')

        # YYY MM DD - one true timestamp
        now = Time.now.strftime('%Y%m%d')

        if (now >= date)
          rsrc = "#{@resource.type.capitalize}[#{@resource.name}]"

          Puppet.warning "#{rsrc} expired on #{date}: #{message}"
        end
      end
    end

# command line test


$ puppet apply -e 'file { "/tmp/dep": content => "foo\n", deprecation =>
"20120425:We can remove this file after release 4" }' 
Warning: File[/tmp/dep] expired on 20120425: We can remove this file after release 4
Notice: Finished catalog run in 0.06 seconds

Using the metaparameter is easy enough, just specify 'deprecation' as a property on a resource and provide a string that contains the date to start flagging the deprecation on (in YYYYMMDD format) and the message puppet should show. I don't currently fail the run on an expired resource but this is an option.

The are some other aspects of this to consider - Richard Clamp raised the idea of having a native type that could indicate this for an entire class (I'd rather use a function, but only because they are much easier to write) and Trevor Vaughan suggested a Puppet face that could present a report of the expired, and soon to be expired, code.

I don't know how widely useful this is but it made a nice change to write some puppet code. The small size of the example will hopefully show how easy it is to extend nearly every part of puppet - including more 'complicated' aspects like metaparameters. Although not the relationship ones, those are horrible ;) I've submitted the idea to the upstream development list so we'll see what happens.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Hi,

In the last 45 days, Vault.centos.org's 2 public facing machines delivered just under 66 TiB of data. So while we try and spread this load a bit ( its growing at 25 - 35% month on month ), we've had to make a few changes.

Firstly, isos are no longer directly downloadable from vault.centos.org, you will need to go the torrent route if you want older, deprecated release isos

Secondly, we've turned off multi range requests ( httpd will still accept upto 5 range's, and then block after that )

Over the next few days, we are going to recycle some of the larger disk mirror.centos.org nodes into vault.centos.org; If someone wants to contribute to this effort, please come find us on irc.freenode.net in channel #centos-devel or #centos-mirror or tweet us @centos or email us the address mentioned at http://wiki.centos.org/Donate - but keep in mind that need machines with more than 1 TiB of usable space, and more than 300mbps of network capacity, and since we will consume that bandwidth high density hosting facilities with high contention on the links wont work.

- KB

Reviewing the second edition of Cisco Routers for the Desperate was quite hard for me as I have very little to add to the Cisco Routers for the Desperate 1st edition review I posted a few years ago. After reading through this update pretty much all those comments still stand. It's an excellent, useful, well written book and the author still has a -distinct- written tone.

I enjoyed the book; I must have considering I bought the second edition! The material has been updated where needed and it's still lacking a section on ACLs so I'll stick to my score of 8/10 for people purchasing this book for the first time and look forward to another refresh in a couple of years time. If you already own the first edition then your choice is a little harder - this book is still an excellent stepping on point for the cost but don't expect much beyond a refresh on the same content.

Disclaimer: Part of my previous review is quoted in the marketing blurb at the front of the book. I did however pay for this book myself.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Hi,

The first ever CentOS Dojo, a one day training and socalising day dedicated to CentOS and how people use it, will be held at Antwerp, Belgium on the 8th of Apr.

You can see the great speaker lineup on the events page at : http://wiki.centos.org/Events/Dojo/Antwerp2013 - we have tried to cover all the major conversation areas around CentOS these days. Ranging from provisioning, management, app deployments, system and virtualisation tuning, virtual infrastructure and more.

Its going to be a great day, register up, and see you all there. And remember, there is an exclusive CentOS Dojo Tshirt for everyone who attends ( plus, there might be more goddies too ).

Jump directly to the registration page : http://centosdojoantwerp2013.eventbrite.com/

- KB

It started off by being a place that everyone could chat and talk about things that were happening in the QA cycles inside CentOS. But things have changed quite a lot - our QA cycles are a lot shorter, there is a lot more automation and there is almost no real security exposure to users.

And I think we can do this better. We can create a better end user experience that gives them direct access, easily, to the state of play within the testing. And we should be able to automate more to get better coverage.

To that aim, qaweb.dev.centos.org is now going away. And we are working on some alternatives. Starting with having a nightly QA cycle, that considers point releases and all updates upto that point. And adding more external tests as well, like the ltp content ( http://ltp.sourceforge.net/ ). If you wish to join in that effort, drop in on the centos-devel list ( http://lists.centos.org/ ) and jump right in. Ref threads: http://lists.centos.org/pipermail/centos-devel/2013-March/009098.html and http://lists.centos.org/pipermail/centos-devel/2013-March/009099.html

Here is a link to the official announcement that just went out : http://lists.centos.org/pipermail/centos-announce/2013-March/019649.html

See you there,

- KB

Go with the grain. When you’re planing wood, you have to follow the grain of the wood. The grain has a direction, which depends on the cut of the wood. (which is the way it grew). Do that and you’ll end up with thin shavings and a smooth finish.. Go against the grain, and you’ll rip out chunks.

I’ve worked with Ant builds that did the same thing. Instead of using the features of the tool the way they grew, people have fought against them. The result is usually expensive and wasteful.

Many of our tools (in my opinion, the best ones) are optimized to do one thing well: that’s their grain. Go with it, or choose another.

Go with the grain is a post from: The Build Doctor. Sponsored by AnthillPro, the build and deployment automation server that lets you release with confidence.

Share:

One of the most under utilised things around in any CentOS sysadmins toolset is the rpm-metadata repositories. Some part of it comes from the fact that most people consider it to be yum specific, and while thats true to quite an extent - its not the only use for the rpm-metadata info. The bottom line is that rpm-metadata is a collection of information extract from the rpms contained in the repository and contains a lot of content, state and depenencies.

Over the course of next week, starting on the 26th September I'm going to run through a week of rpm-metadata tricks. One new cool thing to do with the info contained in the repo metadata's every day.

Some of these things are perhaps good candidates for yum plugins, but I'll try and stay clear of any dependancy on yum and also try and cover as many languages as possible : atleast ruby, python, php, bash and lua! Stay tuned.

- KB

I have a need for shared storage of around 300GB worth of 200×200 image files. These files are written once, then resized and stored. Once stored they never change again – they might get deleted.

They get served up to 10 Squid machines and the cache times are huge, like years. This is a very low IO setup in other words, very few writes, reasonably few reads and the data isn’t that big just a lot of files – around 2 million.

In the past I used a DRBD + Linux-HA + NFS setup to host this but I felt there’s a bit too much magic involved with this and I also felt it would be nice to be able to use 2 nodes a time rather than active-passive.

I considered many alternatives in the end I settled for GlusterFS based on the following:

• It stores just files, each storage brick has just a lot of files on ext3 or whatever, you can still safely perform reads on these files on the bricks. In the event of a FS failure or event your existing tool set for dealing with filesystems all apply still.
• It seems very simple – use a FUSE driver, store some xattr data with each file and let the client sort out replication, all seems simple
• I had concerns about FUSE but I felt my low IO overhead would not be a problem as the Gluster authors are very insistent – almost insultingly so when asked on IRC about this – that FUSE issues are just FUD.
• It has a lot of flexibility in how you can construct data, you can build all of the basic RAID style setups just using machines of reasonable price as storage bricks
• There is no metadata server, most cluster filesystems need a metadata server on dedicated hardware kept resiliant using DRDB and Linux-HA. Exactly the setup I wish to avoid and those are overkill if all I have is need for a 2 node cluster.

Going in I had a few concerns:

• There is no way to know the state of your storage in a replicated setup. The clients take care of data syncing not the servers, so there’s no healthy indicator anywhere.
• To re-sync your data after a maintenance event you need to run ls -lR to read each file, this will validate the validity of each file syncing out any strange ones. This seemed very weird for me and in the end my fears of this was well founded.
• The documentation is poor, extremely poor and lacking. What there is applies to older versions and the code has had a massive refactor in version 3.

I built a few test setups, first on EC2 then on some of my own VMs, tried to break in various ways, tried to corrupt data and come up with a scenario where the wrong file would be synced etc and found it overall to be sound. I went through the docs and identified any documented shortfalls and verified if these still existed in 3.0 and mostly found they didn’t apply anymore.

We eventually ordered kit, I built the replicas using their suggested tool, set it up and copied all my data onto the system. Immediately I saw that small files is totally going to kill this setup. Doing a rsync of 150GB took many days over a Gigabit network. IRC suggested that if I am worried about the initial build being slow I can use rsync to prep the machines directly individually and then start the FS layer and then sync it with ls -lR.

I tested this theory out and it worked, files copied onto my machines quickly and the ls -lR at the end found little to change according to write traffic to the disks and network, both bricks were in sync.

We cut over 12 client nodes to the storage and at first it was great. Load averages was higher which I expected since it would be a bit slower to respond on IO but nothing to worry about. A few hours into running it all client IO just stopped. Doing a ls, or a stat on a specific file, both would just take 2 or 3 minutes to respond. Predictably for a web app this is completely unbearable.

A quick bit of investigation suggested that the client machines were all doing lots of data syncing – very odd since all the data was in sync to start with so what gives? It seemed that with 12 machines all doing resyncs of data the storage bricks just couldn’t cope, they were showing very high CPU. We shut the 2nd brick in the replica and IO performance recovered and we were able to run but now without a 2nd host active.

I asked on the IRC channel for options on debugging this and roughly got the following options:

• Recompile the code and enable debugging, shut down everything and deploy the new code which would perform worse, but at least you can find out whats happening.
• Make various changes to the cluster setup files – tweaking caches etc, these at least didnt require recompiles or total downtime so I was able to test a few of these options.
• Get the storage back in sync by firewalling the bulk of my clients off the 2nd brick leaving just one – say a dev machine – start the 2nd brick and ls -lR fix the replica, then enable all the nodes. I was able to test this but even with one node doing file syncs all the IO on all the connected clients failed. Eventhough my bricks werent overloaded IO or CPU wise.

I posted to the mailing list hoping to hear from the authors who don’t seem to hang out on IRC much and this was met with zero responses.

At this point I decided to ditch GlusterFS. I don’t have a lot of data about what actually happened or caused it, I can’t say with certainty what events were happening that was killing all the IO – and that really is part of the problem, it is too hard to debug issues in a GlusterFS cluster as you need to recompile and take it all down.

Debugging complex systems is all about data, it’s all about being able to get debug information when needed, it’s about being able to graph metrics, it’s about being able to instrument the problem software. This is not possible or too disruptive with GlusterFS. Even if the issues can be overcome getting to that point is simply too disruptive to operations because the software is not easily managed.

Had the problem been something else – not replication related – I might have been better off as I could enable debug on one of the bricks but as at that point I had just one brick that had valid data and any attempt to sync the second node would result in IO dying it means in order to run debug code I had to unmount all connected clients and rebuild/restart my only viable storage server.

The bottom line is that while GlusterFS seems simple and elegant it is too hard/impossible to debug it should you run into problems. A HA file system should not require a complete shutdown to try out a lot of suggested tweaks, recompiles etc. Going down that route might mean days or even weeks of regular service interruption and that is something that is not suitable to the modern web world. Technically it might be sound and elegant, from an operations point of view it is not suited.

One small side note, as GlusterFS stores a lot of is magic data in x-attributes of the files I found that my GlusterFS based storage was about 15 to 20% bigger than my non GlusterFS ones, that seems a huge amount of waste. Not a problem these days with cheap disks but worth noting.