Category Archives: DevOps

DevOps

Operations On-Call Go-Bag

A go-bag (or bug-out bag) is commonly discussed in terms of emergency preparedness; it’s a bag containing things you would need to use to survive for 72 hours. For survival, you would be looking at things like food, water, a medical kit, and some basic tools. The idea being that you may have to leave quickly in an emergency, and may not have time to gather supplies. We can borrow ideas from this useful concept to help build out a kit that you should be carrying with you when you’re on-call.

The Basic On-Call Go Bag

Cell Phone

I know I shouldn’t have to say this, but if you’re on-call you’re going to need a cell phone. I would recommend setting it to an audible ring-tone, something obnoxious that you wouldn’t miss if it rings.

You should have any numbers you’ll need when on-call programmed into your phone. Obvious numbers you should have are things like your datacenter’s NOC, your data carrier’s NOC, and any vendor who is involved in the delivery of your service. This should probably include any developers, team leads, and managers you may need to call.

Mobile Host Spot or Tethering Plan

There is a chance you may be somewhere without WiFi, plan ahead and make sure you can keep in contact. It’s a terrible feeling to get an SMS and not be able to do anything for 20 -30 minutes.

Charger for Cellphone

If you need to use your phone to talk, you’re probably going to use more battery time. If you’re also planning to use your phone to tether or a mobile hotspot, you’ll be eating through your batter very quickly, so it best to carry one of these.

Headset for Cell Phone

If you end up on the phone, you’ll probably want your hands for other things. It helps you keep both hands on the keyboard, of allows you to easily take notes, without putting your smart phone at risk.

Computer

Well you are going to fix servers, right? Make sure you can connect to your production instances, your documentation servers, VPN, and your mobile internet. You should try all of these things first, so that you’re not caught trying to figure it out on the go. Make sure you have the following:

  • Production Access (VPN or bastion host access)
  • SSH keys, if needed
  • Offline copies of some documentation in case your documentation servers are inaccessible.
  • Links to bugtracking, and monitoring servers/services.

Laptop Power Adapter

You should be prepared to be on your computer for a while. Its can be pretty easy to run out of power when you’re on the go

Paper Notebook

Sometimes it helps to write things down. You may not be in a good place to file tickets, and communicate what is happening. It can also be a little faster to scribble a note on a piece of paper, and it may help you reconstruct things a little later. As a extra bonus you can take down a phone number, if you need to.

Pen

A notebook is pretty useless without it.

Data Center Badge (if needed)

If you have physical hardware, you may have to drive to the datacenter. Its better to have this with you in an emergency, then to have to stop at the office, or home first.

Conclusion

Its not a ton of equipment, but its enough to get the job done. What you’re looking to do is eliminate any reason why you’d to eliminate any thing that might stop you from resolving the issue.

 

DevOps Physical Hardware

Physical Infrastructure for the Win

Tired of cloud infrastructure performance? Wishing you could get a couple SSDs to solve your IOps issues in EC2? Trying to reduce your operating expenses, and are ok with the capital expense? There are plenty of reasons that you should consider physical, or managed hardware but managing it presents its own challenges. In order to do it right you’ll need to keep a few things in mind.

1. Start by turning out the lights.

I don’t know why but many people love trekking to the datacenter; I hate it. It is like working in a boiler room; there are thousands of fans, cold and hot moments, and a lot of physical security. When something goes wrong at two in the morning, and you’re at least an hour away, you’ll wish you didn’t have to call the smart hands service, or worse yet hop in the car to press a power button. So, making your physical hardware easy to deal with starts by building and then utilizing your lights out management resources.

You should only consider hardware that is meant for the datacenter. You’re looking for equipment with an IPMI card, and ideally with a virtual console exposed through a web ui, if you can’t get the virtual console, try to setup serial over lan access, since its better than nothing. When you install your equipment, your first step should be setting up your IPMI access, and connecting them to your management network(I’ll coverer setting up a management network in another blog post).

Often overlooked due to price, but totally worthwhile are managed PDUs. They often add to the underlying price, but will save you money in the long run, since a smart hands call will cost you about 200 dollars to pull a power plug.

2. Physical Tracking

Often overlooked in early small installations is the importance of tracking your hardware, wires, switches, DIMMs, and power. There will be times that you have to explain to someone else what to do in the data center, and the more you have documented the easier communicating with someone else is.

I would personally recommend installing something like Racktables (http://racktables.org/). Its fairly easy to setup, but makes life much simpler. If you think that’s too much, you can use a wiki, a spreadsheet, or Visio. You’re looking for something that you can send to someone in an emergency.

You should record what system interface is plugged into which port on your switch. You should give each Ethernet cable an id, and record the cable id associated with each connection. You should give each power cable an id, and you should record which socket you used on your PDU. Label your servers, and record the labels in your tracking solution. When you make changes update what you’ve changed.

3. Plan Your Trips

If you’re going to the datacenter, make a plan and try to stick to it. It’s easy to get distracted, and leave things in a partially configured state, or not update your documentation. I always spend an hour or so planning my day at the datacenter.

You should arrange your day starting with the most critical tasks, and finishing with the least critical tasks. If you get stuck on a task, you can skip things that aren’t as crucial, and also helps you make sure that if you’ve started a task you can finish it.

Think through what you’ll need on your trip, and think about what you’ll need. You may or may not be close to a place you can buy an Ethernet cable, or a power adapter for you laptop. If you need something, its best to have it with you in your bag, or worst case in you car. You can easily loose 2 or 3 hours going on a shopping trip. If you’re missing something, and its not a high priority, schedule the task for the next trip.

The most important thing to remember is that you need to have discipline. Keep track of things, plan ahead, and set your self up for success. Physical infrastructure has different challenges, but they can be solved easily.