Day 13 - Don't be a Human Keyboard

Written by Jordan Sissel (@jordansissel)

There are few things that quite irritate me more than when I accidentally take part in building a habit (or a culture) that treats people like human keyboards.

Humans are not meant to be buttons for pushing or functions for calling.

What does it mean to not be a human keyboard? Imagine for the moment that instead of typing on a keyboard or using a mouse, you had to speak at a person to perform any task?

"Go to my email."

"Show me that first message."

"Scroll down a bit."

"Scroll down a bit more."

"I want to reply to this email."

Imagine the terrible impact on productivity and agility by using such a computer interface! Further, your "check email" flow now requires two people (you + the human keyboard) - that means double the cost, double the typos/communication problems, etc! Yet, despite the silliness of this example, other, very similar situations occur quite often in business as interactions between employees making requests of each other.

In 2008, SysAdvent day 22 reported a subset of this problem - where people may tell you to implement a solution rather than telling you what problem they need solved. When Frank (fictional person) says "install Postgres on that server," he is treating you like a human keyboard. He isn't presenting you with a problem needing solved, he is telling you to implement a chosen solution (that may or may not work). You have a function being called and a result is expected. What result? You don't know the problem and lack context!

Let's ignore, for this situation, situations where it may be correct to accept input as "install this software." (or other examples)

The social problem with being treated like a keyboard is interesting. In the great acronym, PEBKAC, who is the user? I observe that you are the keyboard, here, and Frank (above) is the user. What happens if "install Postgres on that server" is obeyed correctly, but doesn't solve Frank's problem? Even though, in this instance, "Frank" is between the chair and keyboard (you), you may be blamed for his errors. That sucks, and you waste energy in a blame game rather than getting work done.

Another example of this is if a coworker asks you, rather than asking that well-publicized dashboard your team maintains, "is mysql down?" If you answer the question by checking the dashboard on their behalf and replying with the answer, you are only training the requester that you are the answer robot for mysql status. Instead, you should say "Check out the status here on the dashboard" and provide a URL. If you don't have a dashboard, maybe you have a FAQ page, or you dig and ask "What's the problem?"

Humans are creatures of habit. If you reinforce a behavior, it will persist, and even spread to other coworkers or users. You don't want to become the company-wide interface to answering the "is mysql down" question.

You don't want to become a human keyboard.

You have a options for avoiding turning yourself into a human keyboard, and solutions will vary by situation and audience. First, you can simply document common questions and answers. Second, you could (assuming skill/time/energy) automate answers that can't easily be documented - like creating a dashboard to answer service health status questions. Third, if you can't automate it, put requests like these through your ticketing system. Fourth, you can try to train the user to answer the question without invoking you.

Regarding the ticket system, if common requests require high effort, you can use features of your ticket system to track the number of these such requests and also time spent on them. This will help inform the business about the energy output of your group versus the energy requirements and can help steer hiring and other goals.

There are usually some red flags that tell me (even if only subconsciously) that a request is a "human keyboard" one: annoying requests, simple requests from technical coworkers, strange solution-based requests when you don't have context, etc. Personally, my signal is usually that a request is annoying.

Do you have users or coworkers who might treat you like a human keyboard? Go write that FAQ, make that dashboard, or add that ticket flow. You'll end up with less stress and more happy coworkers and users. The business will end up with higher quality support, faster turn-around times for simple requests, and better ideas about the kinds of things asked of you and your team.

Further reading:

• Stashboard - Open Source status/dashboard
• MediaWiki - a wiki for documenting things (like that FAQ you could write)

Day 4 - Communication and Organization of Changes

Change happens. Maintenance, (planned or unplanned) outages, upgrades, etc. A change can be something small, such as tweaking a configuration value, or they can be large, such as replacing power units on hundreds racks across multiple datacenters. Changes come in all shapes, sizes, scopes, and risks, and require varying degrees of planning, scheduling, and cooperation.

Communication is an important piece of the change process, and good communication allows you to inform users and groups about pending changes and change impact. Good communication requires using the right tools, so it's worth reviewing the tools that are available to you:

• Email. Email is easy to send, and maintaining mailing lists for users of any given component is pretty easy. Email seems cool by itself until you learn that nobody actually reads email (kind of like that intranet wiki...). I'll cover how to work around this shortly.
• Calendar. Everyone understands calendars. They grew up living with them, the presentation is familiar, and it is based on a concept everyone understands: time.
• Bug/Issue/Ticket systems. These systems are good for tracking work units, such as changes. They tend to have status-setting features including words like "need feedback," "in progress," and "resolved." Seems like a good fit for tracking the progress of a change.
• Phones. Phones are good when you need synchronous communication, such as when coordinating a change across geographics, or calling customers to get acknowledgement of a proposed change.
• Meetings. Meetings are good places to announce changes, scheduling, and impact. Attendees can nod quietly or object to the change or scheduling. This helps you review a change and fit its schedule to minimize risk and impact.

Every change will not need to involve every tool listed above. Further, the tools you use to communicate, plan, and log changes will depend greatly on the culture and size of your company and on the impact and risks of each change. Use the tools that fit best.

For email, have an 'it-changes' or 'ops-changes' or 'yourteamname-changes' mailing list that has your team (which can just be you) and anyone else who is interested in the changes you are making. Additionally, create a mailing list for any component that might need maintenance, such as a datacenter location, a service like Active Directory, or the network filer; document these and encourage people needing a particular component subscribe to those mailing lists. Use your judgement here. If your company is small, you can probably create fewer mailing lists.

Like engineers who need to consider unreliable networks in their design, you must consider unreliable readers in your change announcements. Folks don't read email; they skim and read things they think are important, which means they may skip your change announcement. You may have to resort to trickery in order to get people to read important change announcements; try prefixing your subject with "CAKE AND PIE" - it might work? Further, there is often no feedback from email - you won't get any acknowledgement of who has read an announcement, or more importantly, that they have understood it.

The best way to ensure your announcements are read are by targeting only the people who need the information and by repeating your message. Depending on the size and impact of the change, you may need to send your change announcement up to three times. First, to announce the scheduled change. Second, a day (or hour) before the change starts. Third, when the change starts. A final, "all clear," message should be sent when the maintenace is complete.

Email has some failings, like no acknowledgements. Calendars can do this, and more. Calendars are great visual tools for communicating schedules. Online calendars (in Exchange, Google Calendar, whatever) are great for several reasons. First, you can invite people to the event, which gives them a visual reminder in their calendar. Further, you get reminders for free: Just before the event starts, your invitees will likely get a popup reminding them of an change. Additionally, Calendars are shareable and publishable. Calendar data exchange is pretty standardized - iCalendar format sends well over email. Invitees can acknowledge receipt, helping you figure out who hasn't acknowledged and might need a phone call. Finally, calendars can often be downloaded to smartphones and other devices. All of these are excellent features of modern online calendars which will help you communicate changes more effectively.

You should schedule maintenance and outages in a calendar. Create a calendar (or multiple, using the same principles from the mailing list creation above) to track these events. Invite people and groups who need to know about the event. The data in each event should include two things: a short description (think email subject) and a link to wherever the detailed plan/discussion for the change lives, which is likely an issue/bug system. For an unplanned outage, create an event that represents the actual time and duration of the outage.

Ticket (aka bug or issue) systems should be used to track individual changes. As mentioned above, you get the state-tracking benefits (open, pending, in-progress, fixed, etc) and a reasonable place to record planning notes and actions taken during a change. Your emails and calendar entries should include links to the change ticket if it is relevant. The ticket should also, if possible, include a link to the calendar event for easy import into other calendars.

Phones and meetings are of similar use as they both grant you synchronous communication. Phones and in-person meetings are good for the planning stage. They are also both good for reviewing pending changes or for confirming that all involved acknowledge the change. Your use depends on your needs. For example, a previous job had weekly meetings to discuss impact and scheduling of planned changes. Phone is also a useful tool for calling in more experienced teammates when there's a problem with a change.

Lastly, you are a customer, too. Others will make changes that affect you. Datacenter facilities, ISPs, and other service providers make changes just like you do. I would love if my service providers sent me change notifications with calendar invites, but nobody does. Do your vendors send you change notifications? Do they include calendar data you can import into your own calendars? Do they follow the advice above? If you said no to any of these, it's worth having a chat with your vendors and providers to help work towards this. I'm trying to work with my providers to get them to do this, but it's a slow battle due to the current systems and practices, so be prepared and patient.

Remember, we are often guilty of undercommunicating and even communicating poorly. Focusing on effective communication of changes will help ensure your customers (coworkers, users, clients, etc) are well-informed of the changes that affect them. Informed customers are happy customers. As mentioned in the previous paragraph, you are a customer, too. Making you a happy may require you working with your service providers to sell them on the advice here.

Further reading:

• iCalendar and Applications that support iCalendar
• Sysadvent 2008's Visibility and Communication
• Sysadvent 2008's Change Management

Day 22 - What's the problem?

From Wikipedia's software development process article, "the most important task in creating a software product is extracting the requirements or requirements analysis." Requirements analysis is an early part in the engineering process. This means learning what problem needs to be solved and the parameters and constraints with which you must solve them.

The wikipedia article goes on, "customers typically have an abstract idea of what they want as an end result, but not what software should do." My experience in systems administration is that customers may have an idea of what they want as an end result, but they may not know what problem they are solving and often present their idea in the form of a solution.

To describe this with an example, let's say you have a small team of sysadmins who are familiar with mysql and your group supports a few mysql deployments in your company.

A customer says, "I need postgresql installed." This is a request for action, not a description of the problem. You are only given the solution that the customer believes will bring about their desired end result. Do you simply install postgresql for them, or do you ask why they need it? If you already have a well-supported mysql deployment, you should be asking why you need to support another database.

Ask about the problem they need solved. Get details. Why does he or she need postgres? Can the existing mysql deployment and knowledge be used instead? Most of the time customers who simply ask for actions, "please implement this solution," often are unaware of existing, similar options already available. It's also possible that this customer is trying to solve a problem that doesn't exist, doesn't affect your company, or isn't feasible to solve completely.

If you get requirements, you might find they are simply "I need a database that speaks SQL." Alternately, you might find that the requirements include "I need to run this 3rd party tool which requires postgres." Dig deeper. What does this tool do? Can it's features be provided by another tool that doesn't require burdening your team with additional products to support? Is the problem the customer wants to solve even in the scope of your team?

In addition to getting the necessary information about the problem, you should also make sure you are given other constraints and parameters. Is there a deadline? What is the scope of the problem, who is affected, etc? What is the priority?

Let's examine another common situation. Another customer says, "I need apache on serverfoo restarted." Again, you should ask for a description of the problem. What are the symptoms the customer is observing? Restarting apache is an action that could bring about a solution, but what are you solving? What is broken? What if a customer reports "mail is down?" What does "mail is down" mean? What are the symptoms being observed?

When digging for a description of the problem from your customer(s), be careful to not offend the customer. It's easy to dismiss the customer as an idiot if you the information you are given doesn't make sense or doesn't help you fix a problem. This issue can easily occur when a non-domain-expert interacts with a domain expert. Remember that perspective is reality, and that "mail is down" makes total sense to your customer but is confusing to you. Make sure your fellow sysadmins follow the advice in this paragraph, too.

Asking for requirements can be a tool to help push back on bad ideas. Sometimes a management hammer comes down from above and says you must implement something that you disagree with or don't understand. Being a domain expert, you might be disagreeing or lacking understanding because the request doesn't make sense. Ask for requirements! Sometimes ideas manifest themselves into requests (or mandates) without the idea being actually thought out.

Lastly, always remember you can say, "no." Not every idea is a good one. Bad ideas can come with urgency. Be understanding of any urgency from your customers, but remember that you have the most information about what makes a change bad. Otherwise, why would they be asking you or your team to do it? Be aware of things people will say to convince you to do something even though you can show it is incorrect, such as "the CEO said we have to do this." Facts are your ally, so use facts to show why a proposal is wrong, why a request doesn't make sense, or why a set of requirements are impossible to fulfill.

Day 19 - Visibility and Communication

A friend said to me tonight, "Once again, at a company function, the CEO has forgotten operations exists." This is a visibility problem that often hits operations and support teams.

Good systems administration is about hidden work and effort nobody ever sees. You are probably accustomed to only being highly visible when there is a problem - interrupts asking if something is broken, when it will be fixed, etc. If this is the only visibility you receive, how can you expect to be loved and adored by the world? You need to help yourself and your teammates with visibility. Your manager should help you, too.

Improving local visibility, that of your teammates and manager, is just as important as external visibility to your customers (employees or otherwise).

Keep track of your work: code commits, ticket resolutions, etc. A habit I developed while working at Google was to maintain a weekly report of things done. I've sinced modified this habit to include not only finished tasks, but things not done yet, progress blockers, and future todos. At the end of each week, send this data to your manager. Have the rest of your team do the same. For bonus points, send it to your team, so your coworkers will know what you did last week.

This weekly tracking will help you do two things: first, to maintain a high quality stream of communication to your manager and your coworkers, and second, to help you better track things that need to get done. If you're lucky (and you probably are), a fellow coworker will see that you have something on your todo list that he or she would like to do and offer to relieve you of this burden.

Tracking this data will also help you show how you can or can't take on that new project for time management reasons. Further, it's a huge help to have a document of accomplishments for career advancement.

To enhance visibility and communication with your manager, have periodic (weekly, etc) one-on-one meetings. Email is good for status reports, like above, but face to face contact is best for discussion. It's a two-way street, so use this time to make sure your visibility and perceived performance is what you expect it to be. Additionally, make requests of your manager if you have any. If you submit status reports that include things that are blocking you, your manager should ask how he or she can help remove these blocks.

Visibility to your customers and to your management should be handled differently. Your manager should be the funnel of information up (and down!) the management stack. Make sure he or she is performing this task. A good time to ask about this is in your one-on-one meetings.

Your customers are very important. Your work will directly, positively or negatively, affect their work. This is power and responsibility that can lead to resentment and anger if not handled properly. Creating interaction policies and informed expectations is critical to customer visibility and happiness.

Your interaction policy should explain how to contact your team, and be sure it's accessibly documented. I find bug systems to be great for tracking customer requests or problem reports, so require usage of this system for such things. Define escalation criteria, such as "if a critical problem is not responded to in X minutes, please email this pager address." You need to define "critical" in the previous statement, too. Don't use email alone for problem reporting, as it doesn't easily lend itself to historical tracking.

Set expectations! Planned changes that will cause outages should be announced ahead of time, at the start of the work, and at the end of the work. Any changes in planned change should be announced in a clear way. Announce known issues to anyone affected as soon as you are aware of the problem and include a contact (if not you), a time estimate on repair, and a description of the scope of the outage. Define an SLA for any service you support. An SLA is a common form of expectation declaration.

Additionally, don't waste someone's time. If you send an email about an upgrade to a specific component that only a subset of your customers use, then put a very clear header at the top, such as:

This is regarding an upgrade to the internal mysql servers. If you don't know what this is or don't use these systems, you can stop reading now.

Lastly, visibility and related communication does not have to be manually generated. Automation is sexy, and automatically informing customers about information important to them is a great way to avoid getting 15 tickets filed for the same problem. Have a web-based dashboard that includes a list of known problems and links to related trouble tickets, a list of upcoming planned changes, perhaps a "tip of the day," and any other useful information you see fit. There's plenty of content management systems available for free to help you get this dashboard site up and running in a very short time.

Healthy visibility is about good communication. Systems can go down and customers still be happy because you've involved them in the process by telling them and setting appropriate expectations. Your manager will be happy knowing your team is working effectively by knowing what everyone is working on without having to ask. Happy customers and happy managers means happy and appreciated sysadmins, even when things are on fire.