Real Tests, Real Simple with Minitest, Vagrant, and Toft

I have put a lot of work into my Ark cookbook lately and it has grown to such complexity that it my existing set of workflow and tools were not enough. I needed some way to sanity check my code. I needed tests.

You can't refactor with confidence without tests. I have fully rewritten ark several times already. After making serious changes, I then had to spend roughly 20 minutes manually testing the cookbook to ensure that  it still worked the way I intended it to. This manual testing was getting tedious, really tedious.

Looking at my various options, I first took a look at Cucumber. There are some nice tools out there like simple_cuke, cuken, and ironfan-ci. I made some attempts at using simple_cuke but they failed fairly miserably. That's because I thought I could use Cucumber without having to actually learn it. Cucumber is  a powerful and somewhat complex tool. To benefit from using Cucumber, I was really going to have to read the The Cucumber Book, something I don't have time for right now. I needed the simplest thing that could possibly work. Further, the real benefit of Cucumber appears to be the communication it facilitates between customer and engineer. In this case I am the client and the extra layer of communication is unnecessary.

Enter MiniTest

Luckily for me, BTM and Andrew Crump have been doing great work adapting David Calavera's chef-minitest-handler https://github.com/calavera/minitest-chef-handler. MiniTest is the simplest thing that could possibly work. It works by gathering all the files that match the path  files/default/tests/minitest/*_test.rb for the cookbooks in your run_list into /var/chef/minitest/. Once your regular chef run completes, the minitest-handler cookbook executes each of those tests and displays the results.

Note: As of 30 April, 2012 there is a known bug in minitest-handler that requires you run chef-client twice in a row for the tests to actually execute.
Update:  Set the network type to bridged and you won't have this problem config.vm.network :bridged  

However, running in bridged mode may require sudo.

My ark cookbook is really just an LWRP. The default recipe does nothing more than install the unzip package on those *nix distributions that don't have it by default. I created an ark::test recipe that exercises all of the features of the ark resource. Here is a snippet of it.

ark/recipes/test.rb

Next,  I have my actual MiniTest tests classes, scoped according to my recipe "test"

ark/files/default/tests/minitest/ark_test.rb

To run these tests I have to first add chef_handler and minitest-handler to the beginning of my run_list. Note that the minitest-handler-cookbook doesn't yet support chef-solo, but I patched it over the weekend so that it does.

Here is what the output of the tests looks like

I originally wrote my tests using MiniTest::Unit test cases. However, just a few days ago I discovered the awesomeness that Andrew Crump has built into MiniTest::Spec and a whole set of helper modules that really simplify the code you have to write.

Here is a snippet to test whether a file exists and has the correct owner


I especially love all of the resource matchers that let you connect to the actual resource objects created during your Chef run and to check whether their configuration matches the actual system state.

For some detailed examples, check out this example.

The Not so Simple Part

Ok, that was the easy part. The harder part is to run your tests on a clean environment. You need a relatively fresh VM to ensure that the tests are actually testing changes made by your cookbook and not some change made eons ago for another purpose. The simplest tool for this is Vagrant. Here is the Vagrantfile I use to test the ark cookbook.


Here are the commands you run to provision your new server using vagrant



Note when using Vagrant that bridging your VM's network to your wireless network connection is not a straightforward process, at least not on my linux laptop. If you run linux as your primary OS, I recommend you spare yourself the pain of trying to bridge your VM to wlan0 and just stick an Ethernet cable into your laptop.

While Vagrant is an awesome tool it is not the best tool for all situations. One issue I have is that my corporate workstation is a very locked down Windows Vista installation. I do virtually all of my work within an Ubuntu virtual machine running on top of Virtualbox. Vagrant uses virtualbox to create new virtual machines and there is no mechanism to run virtualbox within virtualbox. Earlier this year I attempted to get ruby and vagrant running on Windows Vista but after 4 hours I didn't even have ruby running on Vista.

Another issue with Virtualbox is that it's flexibility comes at a cost. Virtualbox runs entirely in userspace which means it is less performant than other virtualization tools such as KVM and linux containers (LXC). This performance lag could become very noticeable once you try to spin up a 4-node postgresql cluster. Of the three virtualization technologies mentioned here, LXC adds the least overhead as each container shares the same copy of the running kernel. There may be other optimizations that LXC provides which I am not aware of.

Enter LXC and Toft

As mentioned earlier, LXC has the least overhead of the other virtualization technologies available. Toft is a ruby library for testing chef and puppet cookbooks using LXC containers. Toft has a beautiful interface for working with containers as you can see in the following code.


I can easily see spinning up a multi-node cluster with toft, running my tests, then destroying the whole setup.

While toft has an excellent interface it is a young project with some rough edges, some of which have more to do with LXC than the toft library itself. To get started with toft, there are some nice guides on the  project wiki

I very unpleasantly discovered that the version of the cgroup-lite package for Ubuntu 11.10 installed with LXC will make your system unbootable. The solution is to install cgroup-lite from the oneiric-proposed repository.



Another pitfall is that the bind9 and dhcpd servers installed along with LXC will report erroneous information after initial installation. The fix is easy. Run the `lxc-prepare-host` command and you will be in business.

Finally, at this time Toft only works with Ruby 1.8.7 and does not support Ruby 1.9.*

As noted in the Vagrant section, I highly recommend connecting your laptop to Ethernet if your primary OS is Linux.

Summary

Testing is an exciting and increasing critical part of developing infrastructure. While the existing tools need some polish, you can get a lot done with them. The linkbait title of this post is a bit disingenous. Testing w/ minitest is a breeze, but using vagrant and toft are not straightforward affairs. Toft, in particular, is extremely promising and worthy of your attention. It is extremely well-suited to many of the usecases that Vagrant is not.

Unpacking and Installing from Archives with the Ark Resource

Updated 14 April 2012: The ark API has changed! 

The ark_put, ark_dump, ark_cherry_pick have been rolled into the main ark library as the actions :put, :dump, and :cherry_pick. Please see the updated README

There are many wheels that sysadmins reinvent but the one I see reinvented most frequently is the unpacking of a tarball and placing its contents in an arbitrary location. This task seems trivial but there are many slight variations but critical variations to handle. Most chef users resorts to a combination of remote_file, bash, and link resources. This gets the job done but it is a lot of boilerplate for an essentially simple task and it is ridiculously easy to inject errors. I first wrote the java_ark Lightweight Resource Provider to handle this. Then I discovered Infochimps excellent install_from_release LWRP.

Unfortunately, install_from_release did not do everything I need so I had to extend it further. In the process of extending install_from_release, I mangled Infochimps beautiful code into an unworkable mess. I have now refactored that mess into something that is more maintainable and extensible. Please meet the ark resource. Think of ark as an archive but Kewler. You can thank Joshua Timberman for the name as he didn't care for my original name for it, `cpr` for Crappy Package Resource. CPR was a play on the medical term CardioPulmonary Resuscitation.  Ark does not yet handle all of the functionality that install_from_release does. Install_from_release not not only unpacks tarballs but can build with make, ant, and more. Ark does not currently support ant, make, or setup.py but it wouldn't be too much effort to add that support.

Let's get back to the problem that ark solves. For many of us, the packages available from your given *nix distribution are too old for our needs. This is especially true in the Java ecosystem. The latest stable version of Tomcat is 7.0.26 yet only 7.0.21 is available on ubuntu 11.10. Tomcat 7 isn't even in the main RHEL 6 channels. My customers insist on having the latest security fixes, so installation from rpm/deb isn't an option. Installing the Oracle JDK is an even thornier issue as it is no longer available in Ubuntu nor RHEL software channels.

Here is an example of how I would install the Oracle JDK in the bad old days.



Here is how I accomplish the same with ark.



Wow! That's a lot easier, and frankly more intuitive. Pretty soon I found myself repeating several variations on the ark resource. At first I accomplished this by adding more attributes to the ark resource. This made the resource more verbose, intuitive, and made the code harder to maintain. A few weeks ago I read Nikolay Sturm's blog post on Cookbook Reusability and then interviewed him for the Food Fight Show.  Through Nikolay, I discovered the Interface Segregation Principle.  Rather than adding more options to the ark resource, I totally refactored ark from an LWRP into a Provider and Resource classes that subclass Chef::Provider::ArkBase and Chef::Resource::ArkBase.

Here is the ark_put resource which handles the simplest case:


ark_put simply unpacks ivy to the default location /usr/local/ivy, it does not create any symbolic links
I often have to download arbitrary collections of dependencies and place them in a lib/ directory. I do this frequently for tomcat-based applications. I just need to "dump" the dependencies in a specified directory that likely already holds a number of library files. This situation is a little more complicated as I need a way to determine whether or not the archive has previously been unpacked. With the ark and ark_put resources I simply check if the destination directory is empty. This logic won't work for a directory like tomcat/lib that holds library files from the initial tomcat installation. For this reason, I have to specify a "creates" attribute which specifies the file whose presence indicates that the ark has already been unpacked. Additionally, I need to discard any file hierarchy inside the archive.


ark_dump will place all the *.jar files in the archive into /usr/local/tomcat/lib/. Note: ark_dump currently only supports *.zip archives.

My friends at Oracle seem bent on making me work for my salary. I have encountered on several occasions the need to extract a single file from a tarball and place it in a arbitrary location. In this case that single file will be specified by the "creates" attribute. In this case I extract the JDBC connector for MySQL.



That's the state of the ark resource at the moment. If you are interested to use it and/or extend it, check it out on github. You should note that it is not a collection of LWRPs but actual Provider and Resource classes.  LWRPs are great to get started with but don't provide the same facility for code reuse that pure Ruby does. Thanks to Nikolay Sturm for inspiring me to venture beyond the LWRP DSL and to MrFlip (Philip Kromer) for creating the awesome install_from cookbook. I hope that ark will eventually support all of the functionality that install_from_release does.

FOSDEM Summary

I thoroughly enjoyed FOSDEM. The weather sucked, the beer was great, and the people were wonderful. Thanks to the FOSDEM organizers for putting on such a great conference. What follows is a very rough summary of my experience.

I spent most of my time in the Virtualization room. There were a number of interesting talks there. I didn't spend much time in the Configuration Management Devroom, that is not because the content there was lacking but because I felt more knowledegeable about the subjects there than in the Network/IO,
virtualization, and hypervisor talks where I was pretty much ignorant. From what I could tell, the
organizers did a fine job organizing the Systems and Configuration Management Devroom.

One of the best parts of going to a conference like this is meeting in person that you have collaborated with via e-mail and IRC. I met up w/ Cheffolk Andrea Campi, Juan Jesus Croissier (juanje), Nicholas
Szalay (nico), Stefan Jourdan, and Greg Karekinian (sp?). I have interacted with these guys quite extensively over the last four months, so it was awesome to finally meet them. Juanje has a very different use case for Chef than most users. He manages thousands of desktop machines running GNOME. He must take care not to overwrite the users custom settings, a challenge I have never had to deal with. For this reason he created an LWRP to manage portions of a file. Nico has written a nagios check that will report failing chef runs. I need to start using that.

The best moment of FOSDEM for me was a very personal one. I ran into Sulochan Acharya, a friend of mine from Nepal. We worked together to build the One Laptop Per Child project in Nepal. I didn't even know he was in Europe nor still working in IT. Now he works at Rackspace on Openstack-related stuff. It is a small, small open-source world.

Best talks I attended were Anil Madhavapeddy's "Wild West of UNIX I/O" and Charles Nutter's JRuby talk. Anil talked about the unpredictable behavior that we see in UNIX I/O in virtualized environments. In a
multicore world, hypervisors don't know enough about the underlying architecture to make intelligent decisions about I/O between CPU cores. This leads to very unpredictable I/O results. According to Anil, this requires a better I/O abstraction that is aware of the underlying microarchitecture--CPU layout, interconnects, NUMA--and can make intelligent choices for the developer. Anil calls his next-generation I/O framework FABLE, which is still in very early development. Charles Nutter is a great speaker though his
talk was a bit over my head. He focused on the "impossible" challenges of implementing Ruby on the JVM and how his team overcame them.

I missed the ZIO and CERN talks. Both talked about next-generation I/O frameworks. Also sadly missed Jonathan Ellis's talk on "Overcoming JVM limitations in Cassandra". I spent no time w/ the Mozilla track,  which is a shame as I love how they manage their community and the kinds of projects they work on.

I was struck by the large number of projects that featured web frontends written in Ruby on Rails. Besides Horizon, the Openstack web UI, it seemed that virtually every other web UI I saw was Rails. Ironically, there were no talks specifically about Ruby nor Rails at FOSDEM.

Red Hat is doing a lot of activity in the DevOps space. oVirt a web UI for managing VMs. Matahari a system administration API which looks interesting but seems like it is still in early development. I had the pleasure of talking w/ Ohad Levy, a very sharp fellow who develops Foreman, a front-end to Puppet and is employed by Red Hat.

I found out about the meta-project Katello from Red Hat that contains many components to automate system administration. To my limited knowledge, it includes Matahari for low-level administration, Puppet for configuration management, Foreman as a nice GUI to manage your configurations and server provisioning, oVirt to manage your KVM hosts a la VMware's vSphere, and pacemaker-cloud for high-availability across different cloud providers. I believe the combined stack is at alpha level. I would
love to know if Chef could be used in place of Puppet.

Met the very smart Marek Goldmann who is doing a lot of work packaging JBOSS for Fedora in addition to his boxgrinder project. It seems like there is a fair amount amount of overlap between what he and are trying to do in this regard. I need to spend some time hanging out on #fedora-java to see how we can avoid duplicating efforts.

My greatest regret is that I myself didn't give a presentation. I didn't think I had anything to say that others would be interested, but that view was wholly wrong. There was a lot of interest in Chef but unfortunately not much content.

I met a couple of people who had listened to my podcast, Food Fight, but they told me it was "unlistenable" due to the recording quality. Guess I will have to improve the audio quality :(

I was very lucky to have dinner with a group that included the DevOps gurus Kris Buytaert and  Patrick DeBois. After following both of them on Twitter for a number of months it was really interesting to see what they are orking on. These guys really are the "Belgian Braintrust" of DevOps not to mention that Patrick actually invented the term. If I recall correctly, both Patrick and Kris are spending a lot of time
lately working on monitoring and logging. Not only are Kris and Patrick DevOps heroes but they are also genuinely nice guys.

Beer

Belgians make the best beer in the world and they enjoy it at all hours and in all places. Quite a number of people enjoys fine brews during presentations and meetups. The Opensuse stand actually gave away opensuse beer. Note to self, avoid the 10 % alchohol beer when trying to stay alert. Stick to the delicious 4% blonde beers.


Final Thoughts

Openstack is the real deal, not just a cool idea. It will really be ready for production use this year. I had my doubts about an open-source project led by an industry consortium but what I saw made me put them
aside. Particularly impressive was Ryan Lane's explanation of how Wikipedia is using Openstack for its infrastructure.

There is awesome stuff happening with KVM, both in terms of performance and tooling.

There was a lot of interest in Chef but I didn't meet that many people who had actually used it. This must be remedied. I really should have given a talk and put more effort into planning a Chef meetup.

I will definitely be back next year and hopefully as a presenter.

Announcing the Food Fight Podcast

I just published the inaugural episode of Food Fight, the Chef community podcast that I host together with Matt Ray. The podcast will cover not only the developments within the Chef community but also larger world of DevOps. Enjoy!

Chef in 2012

image by Simon Griffee

Note: Petey King has written a great response to this post. Definitely worth a look.

I discovered Chef in the last last quarter of 2011 and turned my world upside down, work-wise at least, in a really awesome way. Over the holiday season I had some time to reflect about where Chef as a piece of software and as a community of people might go in 2012.

Packaging Cookbooks


Cookbooks are starting to more and more look like packages, with complex dependencies between them. I am not aware of any cookbooks that only work with Ruby 1.9.* but I expect we will see a number of them in 2012. I personally believe that we need to use an existing package management system rather than creating yet another one. RPM, DEB, dmg are all out of the question because they are operating system specific. Ruby Gems seems to be the best choice though I am not very knowledgeable about ruby gems. If you think this a terrible idea, please do let me know what problems ruby gems pose. I envision the scenario where your cookbook_path in knife.rb


As you can see from my example, you still would be use cookbooks as they currently are, but also allow chef to look for additional cookbooks in your GEM_PATH. Cookbook gems would have to be prefixed with something like "chef-ckbk-" to indicate to knife which gems are cookbooks.

Using ruby gems does present a couple challenges, particularly with rvm, which completely compartmentalizes gems from one ruby version to another. This means that changing your ruby interpreter  requires you install anew a large portion of your cookbooks. Using gems may also require a lot of people to change the structure of their cookbooks. Perhaps, moving moving all the directories to under gem_name/lib/ subdirectory.

Testing 1, 2, 3

There still isn't a broad consensus on how to automate the testing of chef cookbooks. There was some nice progress during 2011 with foodcritic (lint for cookbooks), chef-minitest (unit-testing?), and cucumber-chef but those tools are all still early days. Testing is a particular pain point as testing a minor change to a cookbook may require testing against 5 different linux distributions. I really hope we see more progress on this front though I can't point to any particular initiatives. I personally hope to spend some quality time with minitest in the next couple weeks.

Other Miscellaneous Technical Needs

Selinux support is currently very limited. Some people feel that selinux shouldn't be used in the first place, however, many sysadmins don't have the luxury to make this choice themselves but are required by an external corporate or legal body to use selinux. As Andrea Campi has pointed out on the Chef mailing list, there are a number of consulting firms, such as ZephirWorks, that have the capacity to implement better selinux support if someone steps forward with funding.

Image by Simon Griffee

The java application stack also needs a lot of love. The current set of tomcat, jetty, maven cookbooks only support minimal configurations. I am reorganizing these cookbooks around the java_ark LWRP. In a nutshell, the java_ark LWRP extracts a tarball, adds any included executables in the path, and runs update-alternatives. Along with the jboss cookbook I have already started, I expect to put a lot of work into the java stack. If this is something you are interested in, suggestions, code criticism, and patches are most welcome.

While chef-client and chef-server generally perform at acceptable levels, there are some pain points. For chef-client, memory usage can sometimes be an issue due to the node object ballooning in size during chef-client runs. If I recall correctly, Opscode is working hard on reducing memory usage so we should see some progress this year. On the server side, we can expect a lot of performance improvements as chef-server moves from ruby + couchdb to erlang + mysql. This may happen sooner than one might normally expect as Opscode has already converted part of Hosted Chef to Erlang and MySQL, it just hasn't open-sourced that code yet.

Growing Community

2012 should be the year that the myths "Chef is for programmers" and "Chef is hard" are demolished. These myths sadly linger and inhibit adoption in my opinion. Perhaps part of the reason that some sysadmins think Chef is hard is that it is pure Ruby. Now Ruby is not a difficult language but it is unfamiliar to most sysadmins. Advocating Chef (and Puppet to a lesser extent) also means advocating ruby amongst sysadmins. How to do that? Here are a couple suggestions.

1. Get ruby 1.9 into the default server installs for RHEL, Ubuntu, ArchLinux, etc. This could be quite a challenge as according to this blog post, there is almost nobody in the Ubuntu world doing any Ruby work. With the exception of the awesome Sergio Rubio, the same could be said for RHEL/CentOS/SL
2. More blog articles on using Ruby to solve system administration problems and that sysadmins should consider using Ruby as their default scripting language

All that talk about Ruby advocacy aside, Sysadmins will need to know less Ruby in 2012 in order to write Chef recipes. This is because we will see more and more LWRPs pop up. I love LWRPs. I think they are the best thing since dynamically loadable kernel modules.

Chef is growing by leaps and bounds because it solves an important problem in configuration management. The greatest constraint to that growth is the supply of sysadmins willing to make the investment to learn it. At this point, the majority of the community are self-taught linux geeks who found Chef on their own. At the next stage of growth, Chef needs to be attractive to the group I call "the Professionals". Professionals are talented engineers who make system administration their job but not their life. They don't read blogs during their lunch hour and they don't hang out on IRC in the evenings. The Professionals will typically learn Chef because their bosses tell them to. We need to make that learning process as smooth as possible.

The Chef wiki is an incredible reference but we need a guided tour to accompany it. We really need the Chef equivalent to Pro Git and the Git Community Book site. These resources hold your hand and walk you through all the important parts, whispering comforting words when you get frustrated. I know there is work being done a Definitive Guide to Chef from a major publisher, but I don't know its status.

Lastly on the subject of community, we need some mechanisms to summarize community activity.
The Chef project has a high velocity and I find it increasingly hard to keep up with.  The Opscode monthly newsletter is a good step in this direction. Additionally, I am starting a Chef podcast together with Opscode Technical Evangelist Matt Ray, focused on summarizing community activity and exploring important topics.

Dangers in 2012

I sometimes worry that Chef could become a victim of its own success. As I mentioned earlier, the greatest constraint to the growth of Chef is the supply of sysadmins willing and able to learn it.  If enough well-meaning software architects and aggressive CIOs force Chef on resistant ops teams, we will start hearing about implementations gone horribly wrong. Top-down adoption can only work if the ground below is fertile. That's my two cents.

In Summary

Chef will be a success when the default installation instructions for application X point to a Chef cookbook. I think we will make solid progress towards this goal in 2012 but to really get there, a lot of  things have to happen first. I couldn't be more excited. Thanks to everyone in the Chef community for giving sysadmins like me such a great toy to play with. Happy Hacking!

PS: Let me know if you like the Chef icons. My good buddy Simon Griffee created them after a brainstorming session fueled by Sagrantino Di Montefalco.

Emacs for Sysadmins

I am not going to tell you that emacs is superior to vi, because it is not. For me, vi is far better for basic text editing. However, emacs has such powerful modes that I find myself using it more often than vi these days. Emacs has historically been favored by programmers and vi by sysadmins. I have written in the past  that sysadmins are becoming less like ninjas and more like pirates because of this crazy thing called devops.

As a professional system administrator you can't choose between vi and emacs; you have to learn vi. There are just too many times you will be working on an arbitrary *nix that only has vi installed. You can choose to learn emacs in addition to vi. Vi lets you jump in and around files quickly. Nothing can match the speed of hjkl, g, G, etc. for movement or v + y for copying text. However, we sysadmins are moving from managing individual servers to managing configurations. In the last several couple months I have spent at least 50% of my work time entirely within my chef-repo/ directory working on data bags, roles, and cookbooks. Inspired by the devotion of a number of Chef community members to emacs--notably Matt Ray, Stephen Nelson-Smith, Josh Timberman, among others--I decided to give emacs a try. On the whole, it has been a very positive experience. That is why I recommend sysadmins try emacs.

Ninja	Pirate
Bill Joy Vi Creator Hand placement conceals poison dart	Richard Stallman Emacs Creator Note beard

from philosecurity blog ninjas or pirates, emacs or vi


Why Emacs?

Emacs has some truly awesome modes. The default ruby, python, bash, xml, html seem to do more than their vim counterparts. I think this is large part because emacs' extension language (elisp) is more powerful than vimscript. I made a stab at vimscript some years ago but came away quite frustrated. While I am not a lisp convert, it is a full-featured language that doesn't hold you back. Working with multiple buffers (10+) is extremely easy. Org-mode though, is what got me hooked.


Why Not Emacs?

Using vi is like being in a relationship, you are involved. Using Emacs is like a marriage, you are committed, in large part because there is so much to learn. This is one of the reasons I don't recommend it for everyone.

Further, vi is built for touch typists. You rarely, if ever, have to take your fingers off the home row keys. You can keep your fingers on the home row with emacs but that often means completing a number of key chords entirely with your left hand. Take for example opening a file, writing its contents to another file, and then closing emacs.

C-x C-f  + C-x C-w + C-x C-k

If you keep your fingers on the home row, you will only use your left-hand for the above commands, except for the final "k". You could start to feel some discomfort in your left hand if you did that key sequence several times in a row. For this reason a number of emacs addicts will hit the Control key with their right hand, the "x" with their left, Control again with the right, "f" with the left, and so on. This leads to a more "hunt and peck" typing style but saves your hands.

First, Learn How to be a Pirate


Before you even start playing with emacs you should make sure to swap your Caps Lock and left Control key. Emacs commands use the Control key heavily and leaving it in the default position will soon leave you angry, crippled, and ready to kill Richard Stallman.

Next, I recommend you organize your dotfiles on github or another source code repository. Being a pirate means you have booty and treasure to carry around. You need a ship to carry around all that treasure and for me that ship is github. The core of my dotfiles repo is the Rakefile, copied from Ryan Bates' master, that makes it dead simple to install your dotfiles on a new machine. I don't install my dotfiles on every machine I work on, only the 2-3 I use most frequently.

1.  create your own github/bitbucket/private github repo called dotfiles
2. copy your existing dotfiles -- .bashrc, .vimrc, .emacs, etc -- to ~/dotfiles but without the leading "." !
3. download Ryan Bates Rakefile and put it in ~/dotfiles
4. to test, rename ~/.bashrc to to ~/.bashrc_bak then run $ cd ~/dotfiles/ && rake install.  Your .bashrc should be back
5. Push to your repository

Whenever you make a change to your .emacs or .bashrc  on your ubuntu laptop at home, remember to push your changes and then run "rake install" on your work desktop.

Learning Emacs

OK, finally emacs. I recommend you buy Peepcode's "Meet Emacs" tutorial. It is only $12 and it is the best and most convincing emacs tutorial. It is well worth the money. If this tutorial doesn't convince you that emacs is worth your time stop, turn around, and go back to vi. One of the first things covered in "Meet Emacs" is how to install emacs 24 and the emacs-starter-kit. The emacs starter kit will set of bunch of defaults that help you avoid many of emacs' warts and show off some awesomeness such as ido-mode.

Org-Mode, Emacs Killer Feature

Ok, I have to admit that I have been working with emacs for longer than a few months. I have been using emacs' Org-Mode to manage my life for the last several years, ever since I happened upon Abhijeet Chavan's great article in Linux Journal. Soon after that, I came across Scott Jaderholm's excellent screencast for Org-mode. At the time I read the article, I was the Chief Technology Officer for a non-profit technology startup in Kathmandu, Nepal. I was completely overwhelmed by the sheer volume of different tasks I had to accomplish and all the attendant details, especially the details. I no longer have all that stress now that I am a regular sysadmin, but I do have a large number of very detailed tasks. Org-mode is in many ways my memory. It is also how I manage my attention span.

Many people use Org-mode to implement the Getting Things Done methodology, pioneered by David Allen. I have bastardized it into my own system.

Today is a list of general things that I want to get done today. This week, a list of tasks to complete this week. And of course, right now is a list of specific next actions, that I will complete starting at 1.

When you are finished with a task, don't delete it. Archive it so you have a record of when you are done. Org-mode handily records the timestamp of when you completed the item.

C-c $       # archives a subtree to your archive file,

If I finish reading the O'Reilly Emacs Manual and then archive it, here is what the entry looks like.

I love that I can move items in a numbered list almost effortlessly. M-<up> moves an item up and adjusts the numbering and M-<down> moves an item down in the list.

I typically keep the following org-mode files and their associated archive files.

• mygtd.org -- my todo list, for today, this week, this month, and right now!
• meetings.org -- all meeting notes and preparation
• notes.org  -- this is where I keep notes while I work on a problem

The notes.org file deserves special attention. This is my running work log. As I diagnose a problem, I write up my possible hypotheses, then tests for these hypotheses. Finally, I record the results of testing those hypotheses, pass or fail.

The true beauty of using emacs to manage your todo list is that you can grep through it to find a specific piece of information. This has saved my ass more times than I can count.

Org-mode protip: put your *.org files in a Dropbox folder so you have access to them on your various machines. It may be tempting to put them on github but you should realize that those org files will quickly fill up with personal and corporate information.

Working with Buffers

I never found it that easy to work with multiple buffers or tabs in vim without doing extensive key rebinding. Buffers are something that emacs really gets right, especially when you have ido-mode turned on. I have found working with multiple  buffers really effective when working with several chef cookbooks at once.

Let's say I am working in the sudo/recipes/default.rb recipe but want to jump back to the previous recipe I was working on, in this case iptables/recipes/default.rb. I type C-x b and ido-mode shows me a list of open buffers. To get the last buffer I was in before this, simply hit [RET] or I can start typing the name of the file and ido-mode will dynamically filter that list.

C-x b  [RET]  to get to the last visited buffer

Dired-Mode

I have always heard that Midnight Commander is an awesome but I have never taken the time to learn it. I believe that Dired-Mode offers many of the features of Midnight Commander but lets you leverage the emacs skills you have picked up.

Here are commands that I love

Key Binding	Command
v	View a file or directory in read-only mode
d	mark a file or directory for deletion
C	Copy a file
x	execute changes marked, i.e. delete files marked for deletion
q	close an open dired buffer
[RET]	when u hit [RET] on a file, emacs opens it for editing
R	Rename or move a file or directory
M-x find-grep-dired	Grep a directory for a regex and then view the matching files in a Dired Buffer
Q	Find and replace a regex across multiple files

While Dired-Mode is very useful for managing files I find it most useful for browsing code. You can hit "v" to view as read-only then q to close that buffer when you are done. Viewing files in emacs means you get all that juicy syntax highlighting. This helps me make sense of those butt-ugly XML configuration files I encounter on jboss and tomcat installations.

Miscellaneous Stuff I find Useful in Emacs

• Shell-Mode - This is very handy for moving text back and forth from a shell and code file.  M-x shell-command-region or M-! lets you execute a shell command on a region of selected text
• Inf-Ruby-mode - lets you select a bunch of ruby code and immediately execute it in an irb session. I would love to see this work with shef
• Modes for Languages you rarely touch - I rarely code in java but occasionally I have to read java code. java-mode helps me parse java without having to deal with Eclipse. Have to change a config file written in Erlang? There's a mode for that.
• Properly indenting text - select a region then C-M-\
• Completion - M-/ will guess what you are trying to type based on open buffers
• Commenting out multiple lines of code -- select a region, M-;

Emacs Shortcuts are Everywhere

In case you haven't already noticed, virtually every unix shell and programming REPL (irb, python, perl) supports Emacs keyboard shortcuts. This is because all those shells are built on top of libreadline, which implements a subset of emacs shortcuts. Even REPL's for newer JVM languages like scala, groovy, and clojure use a clone of libreadline, jline, that implements those same shortcuts.

Emacs Stuff I Haven't Learned Yet but Want to

• Tramp-Mode -- editing files over SSH, this would be pretty sweet for configuring a bunch of Virtual Box VM's running on my desktop
• Calendar Integration with gmail - just haven't gotten around to this yet
• ERC - I debate whether to move from Xchat to irrsi or ERC (Emacs irc client). jury is still out
• gist - This is a mode that allows you to select a region and create a github gist from it

Wrapping Up

Emacs, despite conventional wisdom, can be very useful for systems administrators and not solely the domain of Lisp Hackers and not Exclusively for Middle-Aged Computer Scientists. While it may be One True Program for some people, it is not for everyone. The highlights of Emacs for me are:

1. Org-mode -- time management for busy sysadmins
2. Dired Mode 
3. Buffer Management 

If you have some tricks that you find especially useful for sysadmin work, please share them! Here are some additional resources you may want to look at:

• Why Emacs? Borizhdar Batisov's excellent, more developer-focussed ode to emacs
• Hack Emacs Screencasts
• Mastering Emacs blog
• Emacswiki
• #emacs on freenode.net

Chef for Developers

Last Friday I gave a presentation to a group of developers on how they can get started writing Chef cookbooks. I put about 5 hours into writing this presentation. To my chagrin, Warwick Poole released a superior presentation that same afternoon that does a better job of explaining the basic concepts of Chef. I strongly recommend checking out Warwick's "LearnChef" if you haven't already. It is the single best introduction to Chef that I have seen so far. In more good news, Warwick informs me that he intends to create a 5 part series building on "LearnChef." I very much look forward to it.

My Chef presentation doesn't do as good a job explaining the core concepts of Chef but it does walk developers through some scenarios on how they can work with chef on a project. You can check it out below and even reuse my slide material by downloading the .ppt file here. My slides borrow heavily from Joshua Timberman's excellent Chef 101 presentation. The slides are under the Creative Commons 3.0 Unported CC-BY-SA. You can find the full list of publicly available Chef presentations on the Chef wiki.

Chef, Devops, and You

View more presentations from Bryan Berry.