First and foremost this is a biased article. These are all my opinions, and come from my working experience.

Bash(or Posix shell)

Importance 10/10

If you’re working with *nix and can’t toss together a simple init.d script in 5 minutes, you haven’t done enough bash. It’s everywhere, and it should still be your first automation choice. It has a simple syntax, and is designed specifically to execute programs in a non-interactive manner. You’ll be annoyed that it lacks unit tests, and complex error handling, but its purpose built to automate administrative tasks.

Perl

Importance 9/10

This is the language that you will run into if you work in operations. There will be backup scripts, nagios tests, and a large collection of digital duck tape written by co-workers, that do very important jobs. Its syntax is ugly, and you may find yourself writing a eval to handle exceptions, but its everywhere. CPAN makes it fairly easy to get things done, and you can’t beat this for string handling.

C/C++

Importance 5/10

This is the latin of the *nix world, and is basically portable assembly language. I refrain from writing C whenever possible, since I rarely need the raw performance, and the security and stability consequences are pretty severe. You should understand the syntax(its ALGO right), and be able to read a simple application. It would be great if you could submit a patch to a open-source project, but I would never turn down an ops hire because they didn’t know C well enough.

PHP

Importance 7/10

PHP more important than C?! Yep. Like perl its everywhere, people use it for prototyped webapps, and full blow production systems. Its another ALGO syntax language, except you can put together a simple web page in 2-3 minutes; its almost as magical as the Twilio API. You’ll find yourself poking at it on more than one occasion, so you might as well know what you’re doing.

Ruby

Importance 6/10

Doing something with puppet or chef? You probably should know some ruby, and in fact it probably more important to know ruby than chef of puppet. Its relatively easy to pick up, and so many of the automation tools people love are written int it. As an extra bonus, you could write rails and sinatra apps. It’s good to have in your back pocket.

Python

Importance 4/10

People love to love python, but the truth is that its a bit of a diva. Its a language that favors reading over writing, and has a very bloated standard library with lots of broken components(which is the right http library to use?). It wants to be a simpler perl, but I never find it as useful, and it always takes longer. I know a lot of companies say they want to use it as their “scripting” language, but in practice I’ve not seen the value(i stil want to rewrite everyones code).

Chef/Puppet

Importance 2/10

These are DSLs for configuration management. They are supposed to be simple to learn, and if you can’t figure them out with a web browser and a few minutes, they are failing.

Java

Importance 6/10

More ALGO syntax, and more prevalent in high scale web applications. Minimally you should be able to read this language, but its useful to be able to pound out a few lines of Java. It has many rich frameworks, and you’ll likely find it sneaking into your stack where you need something done fast. Also, it is really useful when it comes time to tune the JVM.

Haskel

Importance 0/10

When I’ve run into it running someplace serious I’ll update its score.

Javascript

Importance 8/10

I hate this language, but I can’t deny its growing importance. Its more common to see in a web browser, but its starting to creep into the backend with things like node.js. If you can understand javascript, you can help resolve whether the issue is a frontend or backend problem; you will have total stack awareness.

SQL

Importance 10/10

You have to know SQL. You will work with SQL databases, and you will want to move things in and out of them. You may want to know a dialect like MySQL very well, but you should understand the basics, and at a minimum be able to join a few tables, or create an aggregate query.

I got asked a question regarding filename expansion in bash the other day, and was stumped. It turns out to be something I should have considered a long time ago, and will always keep in mind when writing a script.

Question 1:

What does the following script do if there is a file abc in the current directory?

#!/bin/bash
for i in a*
do
  echo $i
done

Answer:

This a* matches abc and expands to abc, and the script outputs:
abc

Question 2:

What if you run the same script in a directory without any files?

Answer:

The script outputs:

a*

Why?

According to The Bash Reference Manual:

Bash scans each word for the characters ‘*’, ‘?’, and ‘[’. If one of these characters appears, then the word is regarded as a pattern, and replaced with an alphabetically sorted list of file names matching the pattern. If no matching file names are found, and the shell option nullglob is disabled, the word is left unchanged.

So bash will output ‘a*’, because that is how filename expansion works.

Question 3:

What if you run the following script and in a directory with no filename beginning with a:

#!/bin/bash
for i in a*
do
  echo /usr/bin/$i
done

Answer:

The script outputs:


/usr/bin/a2p /usr/bin/a2p5.10.0 /usr/bin/a2p5.8.9 /usr/bin/aaf_install /usr/bin/aclocal /usr/bin/aclocal-1.10 /usr/bin/addftinfo /usr/bin/afconvert /usr/bin/afinfo /usr/bin/afmtodit /usr/bin/afplay /usr/bin/afscexpand /usr/bin/agvtool /usr/bin/alias /usr/bin/allmemory /usr/bin/amavisd /usr/bin/amavisd-agent /usr/bin/amavisd-nanny /usr/bin/amavisd-release /usr/bin/amlint /usr/bin/ant /usr/bin/applesingle /usr/bin/appletviewer /usr/bin/apply /usr/bin/apr-1-config /usr/bin/apropos /usr/bin/apt /usr/bin/apu-1-config /usr/bin/ar /usr/bin/arch /usr/bin/as /usr/bin/asa /usr/bin/at /usr/bin/atos /usr/bin/atq /usr/bin/atrm /usr/bin/atsutil /usr/bin/autoconf /usr/bin/autoheader /usr/bin/autom4te /usr/bin/automake /usr/bin/automake-1.10 /usr/bin/automator /usr/bin/autoreconf /usr/bin/autoscan /usr/bin/autospec /usr/bin/autoupdate /usr/bin/auval /usr/bin/auvaltool /usr/bin/awk


Why?

Because you’re re-evaluating ‘/usr/bin/$i’ which is now ‘/usr/bin/a*’, which expands to the order list above due to shell filename expansion rules. If you want to avoid this you need to protect your variables using quotes. Here is the safe version of the script:

#!/bin/bash
for i in a*
do
  echo /usr/bin/"$i"
done

Just something simple to think about when writing your bash scripts. Expect to enter loops on globs that don't match anything, always protect your variables, and consider setting the failglob option in your scripts.

If you work in operations, you probably have used a ticketing system or two. They are common across the industry, and every organization has its own particular workflow. In my younger days I loathed them, since they seemed to be an impediment to me doing my job. Today, I’d describe myself as a reluctant fan. My goal in their use is to meet a few simple operational needs, without introducing undue bourdon on you development and ops teams.

Change Control

The first and most import use of your ticket system should be to track changes. Do you need to tweak a router configuration, or perhaps move a install 48 additional GB of ram in you DB server? There should be a ticket.

These types of tickets help you construct what happens when something changed. In a small environment (less than 30 servers, and fewer than 2 team members), it is easy to keep track of changes, since the volume of activity is lower than in large environment. But, once you add a few team members, and a couple hundred servers it becomes very difficult to track down what happened with a few email, particularly if there are a few time zones involved.

These types of tickets should be concise; they should contain the purpose of the change, when it was made, and any expected side effects. Ideally you should have a link to you revision control system (your commit message should include the ticket id), with the current and previous versions listed incase someone needs to revert your change.

Post-Mortem Evidence

When a critical production issue happens, step one should be filing a ticket. The initial ticket should be brief, for example DB Master Down, but be able to convey the issue. The goal is to start a clock that can be used in determining response time, and help you reconstruct events when running a post-mortem on the production failure. These tickets should be created for all unexpected production events. They should be periodically updated, and should contain the resolution to the issue.

It feels wrong to file a ticket when “Rome is burning”, but keep in mind you have commitments you’ve made. Most service level agreements (SLAs) contain language requiring that a ticket be created within X minutes of an incident. It also gives your support folks or users a way to communicate the scope of the problem, without interrupting work on a resolution. Once things are recovered, you should have the timeline of events for your post-mortem.

Making Sure It Happens

If your team is anything like most operations or devops groups, you have too much work to get done. Users may interrupt you, developers may want you to change the production environment, but not all of it is critical. Some work just has to be put off until there someone has enough time to do it properly.

Tasks that are longer than 5 minutes, and that you aren’t going to do immediately should be ticketed. The issue should have a priority, and you may want to assign a due date if needed (things like replace SSL cert, can be very low priority but become critical once the due date has passed). Give a brief outline of what you expect needs to be done, so if you’re lucky enough to have a colleague or two, they might be able to take the ticket from you.

If your team can manage, you should triage these types of tickets once per day, ideally at the start of the day. This should allow you to come up with a plan of attack for the day, and help you shift expectations on projects that are being derailed by interrupt driven or higher priority work.

Things Tickets Aren’t For

As soon as people start using issue-tracking software, there is a desire to use it to determine employee effectiveness. While tempting, this should be avoided since most tickets assigned to operations groups have different levels of difficulty.

A co-worker of mine tells a story of working at a large ISP, and being asked to generate this type of report. The newest, and most junior employees were completing the largest numbers of tickets, since they handled reverse DNS requests, and the most senior employees had the fewest since their tickets often took several days to solve. Management was on the verge of reprimanding their entire senior network staff, until they realized the difference in the pattern.

Similarly you shouldn’t hold a task open forever in your ticketing environment. While you may tell yourself this is being done to “track” the issue you are creating noise. These types of issues should be closed won’t fix, or scheduled for competition within a quarter.

Conclusion

Trouble ticket software makes sense, and you should use it. Try to think of the reason you’re creating the ticket, and see if you can keep the content of the tickets aligned with the purpose of the tickets.

Configuration management is a hot topic these days. Chef is one of the more popular choices, and does a fairly good job helping you maintain consistent configuration across your environment. That said it isn’t fool proof. I’ve outlined two common scenarios in which you might introduce a configuration issue.

Removing a File, Package, User, or and Chef Managed Resource

There are a few cases when using Chef where you will end up with an unintentionally installed package, user, file, or other resource. Typically this will happen when modifying a recipe to remove a resource. Lets say you have a recipe that installs three packages:

package "a" do
  action :install
end

package "b" do
  action :install
end

package “c” do
  action :install
end

You may want to remove “package b”, so you might remove it from the recipe:

package "a" do
  action :install
end

package "c" do
  action :install
end

This however will leave you with “package b” installed, and unmanaged across all the nodes running this recipe. Chef is no longer responsible for “package b”, and won’t take any action once its been removed from the recipe. In cloud instances, new instances and old instances will now have mismatched configurations, and you may see issues with dependencies across instances.

The proper way to remove a previously chef managed packages is to do the following:

package "a" do
  action :install
end

package "b" do
  action :remove
end

package "c" do
  action :install
end

If you want to remove the “package b” code from your recipe, wait until you have confirmed all nodes have removed the desired package, and then delete the lines from you recipe.

IMPORTANT:

If you are using chef to manage users, make sure chef removes your users for you, otherwise they will continue to have access. The same goes for any chef managed resource(cron jobs, files, etc…), once chef is in control, let chef remove/uninstall the resource.

Resources Definitions in Loops

I see people use loops to create resources in recipes. Most of the time these are being done for file creation, or execution of an external process. I came across something a few weeks back that was strange:

servers = %w{ "server-a", "server-b"}
servers.each { |server|
 execute "server-command-add" do
   not_if "/usr/bin/add-server-to-something exists #{server}"
   command "/usr/bin/add-server-to-something add #{server}"
  end
}

Chef will do something fairly unexpected here; the second command will not execute because a not_if condition is met on the second resource always. This is because the execute resource for “server-b” has two not_if conditions, (“/usr/bin/add-server-to-something exists server-a”, “/usr/bin/add-server-to-something exists server-b”). Chef copies attributes from the first execute resource defined, and concatenates the additional not_if conditional into and array. Because not_if and only_if are defined as arrays, ruby copies an array reference from the first resource to the second resource.

It is unclear whether this is intentional, but you should be aware of this issue when writing chef recipes. The best way to execute this pattern is to give each resource a unique name, like so:

servers = %w{ "server-a", "server-b"}

servers.each { |server|
  execute "server-command-add-#{server}" do
    not_if "/usr/bin/add-server-to-something exists #{server}"
    command "/usr/bin/add-server-to-something add #{server}"
  end
}

Conclusion

These are just two examples, and I’m sure there are plenty others. When using automation tools remember to check to see if it achieved the results you expected; never blindly trust the tool.