Category Archives: Linux

DevOps Linux Uncategorized

Just Enough Ops of Devs

A few weeks ago I was reading through the chef documentaion and I came across the page “Just Enough Ruby for Chef”. This inspired me to put together a quick article, on how much linux a developer needs to know. I’m going to be doing this as a series, and putting out one of these a week.
 

How to use and generate SSH keys

I’ve covered how to create them here, but you should know how to create distrubute, and change ssh keys. This will make it easier to discuss access to production servers with your ops team, and likely make it easier when you use things like Github.
 

How to use |

If you’ve used unix for sometime, you might be familar wih this. The pipe or | can be used to send the output from one process to another. Here’s a good example of its usage:

 

How to use tar

Tar is one of those basic unix commands that you need to know. Its the universal archiving tool for *nix systems(similar to Zip for windows). You should know how to create an archive and expand an archive. I’m only covering this with compression enabled, if you don’t have gzip, or don’t want it ommit the z option.


 

The file command

File is magic. It will look at a file and give you its best guess as to what it is. Usage is:

 

The strings command

Ever want to read the strings from a binary file? The strings command will do this for you. Just run “strings ” and you’ll get a dump of all the strings from that file. This is particularly useful when looking for strings in old PCAP files, or if a binary file has been tampered with.

 

How to use grep

Grep can be used to extract a lines of text from a file or stream matching a particular pattern. This is a really rich command, and should have a whole article dedicated to it. Here are some very simple use cases.

To match a pattern:

To pull all lines not matching a pattern:

 

How to count lines in a file

The wc commands will count the lines, words, and bytes in a file. The default options will return all three, if you ony want to count the lines in a file, use the -l option that will output only the lines in a file. Here is an example:

 

Count the unique occurrences of something

It might seem like its out of the reach of bash, but you can do this with a simple one liner. You just need to type:

This counts all the unique line occurrences, and then sorts them numerically.

 

Following the output of a file with tail

Tail is a very useful command; it will output the last 10 lines of a file by default. But sometimes you need to want to continiously watch a file. Fortunately tail can solve this for you. The -f option will print new lines as they’re added. Example:

 

I’ll follow this up a week from now, with more linux for devs. Hopefully you found this useful.

cron DevOps Linux

What I Wish Some Had Told Me About Writing Cron Jobs

Much like Doc Brown and Marty McFly, cron and I go way back. It is without doubt one of thing single most valuable tools you can use in linux system management. Though what I’ve learned over the years is that it can be hard to write jobs that reliably produce the results I want. I wish when I started writing these jobs executed by cron in the 1990s someone had told me a few of these things.
 

Don’t Allow Two Copies of Your Job to Run at Once

 
A common problem with cron jobs is that the cron daemon will launch new jobs while the old job is running. Sometimes this doesn’t cause a problem, but generally you expect only one job to launch at a time. If you’re using cron to control jobs that launch every 5 or 10 minutes, but only want one to run at a time its useful to implement some type of locking. A simple a method is to use something like this:

You can get more complicate using flock, or other atomic locking mechanisms. For most purposes this is good enough.
 

Sleep for a Bit

 
Ever have a cron job overload a whole server tier because logs rotate at 4am? Or, got a complaint from someone that you were overloading there application by having 200 server contact them at once? A quick fix is to have the job sleep for a random time after being launched. For example:

This does a good job of spreading the load out for expensive jobs, and avoid thundering herd problems. I generally pick an interval long enough so that my servers will be distributed throughout the period, but still meets my goal. For example, I might spread an expensive once a day job over an hour, but a job that runs ever 5 minutes may only be spread over 90 seconds. Lastly, this should only be used for things that you can except a loose time window around.
 

Log It


I’ll be the first to admit I do this all the time. I hate getting emails from cron, but in general you should avoid doing this. When everything is working this isn’t a big deal, but when something goes wrong, you’ve thrown away all the data that told you what happened. So, redirect to a log file, and overwrite or rotate that file.

Hopefully these tips help you out, and solve some of your cron pains.

DevOps Linux

SSH Do’s and Don’ts

Do Use SSH Keys

When ever you can use a key for SSH. Once you create it, you can distribute the public side widely to enable access where ever you need it. Generating one is easy:


ssh-keygen -t dsa

Don’t Use a Blank Passphrase on Your Key

This key is now your identity. Protect it. Select a sufficiently safe password, and enter it when prompted. This is basic security, plus allows you to “safely” move your keys between hosts without compromising the key security.

Do Use Multiple Keys

Its probably best to use a few keys when setting up access from different hosts. This makes it possible to shutdown a key without locking your self out.

Don’t Copy Your Private Key Around

Remember this is your identity, and authorization to access systems. Its never a good idea to copy it from system to system.

Do Use SSH Agents

Enabling the ssh agent on you laptop or desktop can save you from the tedium of password entry. Launching the agent is easy, then you just need to add key files to it.


# starts the agent, and sets up your environment variables
exec ssh-agent bash
# add your identities to the agent by using ssh-add
ssh-add

Don’t Leave You Agents Running After You Log Out

If you leave your agent running, this is like leaving your keys in a running car. Anyone can now assume your identity if they can gain access to your agent.

Do Make A Custom ~/.ssh/config

You’ll find from time to time that you’ll need special settings. You have a few options, like entering a very long command string, or creating a custom ~/.ssh/config file. I use this for short hostnames when I’m on a VPN, or when my username on my system doesn’t match my account on the remote system.


# A wild card quick example
Host *.production
User geoffp
IdentityFile ~/.ssh/prod_id_dsa
ForwardAgent yes

# Shortening a Host’s Name
# so ssh my-short-name will work
Host my-short-name
User gpapilion
ForwardAgent yes
Hostname my.fully.qualified.hostname.com

Do Use ForwardAgent

This approximates single sign-on using ssh keys. As long as you are forwarding agent requests back to your original host, you should never be prompted for a password. I set my ~/.ssh/config to do this, but I also will use ssh -a on remote systems to keep from reentering password information.

*** EDIT ***

I’ve received a lot of feed back about this point. Some people have pointed out that this should not be used on untrusted systems. Essentially your agent will always respond when prompted to a agent forward request with the response to a challenge. If an attacker has compromised the system or the file systems enforcement of permissions is poor, your credential can be used in a sophisticated man in the middle attack.

Basically, don’t ever SSH to non-trusted systems with this option enabled, and I’d extend this an say don’t ever login to non-trusted systems.

This article does a good job of explaining how agent forwarding works. This article on Wikipedia explains the security issue.

 

 

 

Don’t Only Keep Online Copies of Your Keys

Keep an offline backup. You may need to get access to a private key, and it always good to keep an offline copy for an emergency.