Day 8 - Breaking in a New Company as an SRE

By: Amy Tobey (@AlTobey)
Edited By: Tom Purl (@tompurl)

To put it mildly, 2017 has been an interesting year for everyone. My story has its own bumps and advances, including changing jobs twice recently. Changing jobs is a reality many of us face, hopefully voluntarily. It happens infrequently enough that it isn’t something we spend a lot of time talking about. Life’s trials tend to teach us quickly and I hope to share some of what I’ve learned with you.

Backstory

I started the year as an SRE on the CORE team at Netflix. For various reasons, mostly involving stress, I ended up being fired in late June. I only took a couple days to cool off before kicking off interviews. I knew my financial situation would be stable because of the severance package, but I still wanted to have something lined up as soon as possible. Interviewing can sometimes take many weeks or even months so I got started right away.

My first stop was Apple. I knew lots of engineers from my Cassandra days and a friend there had previously pinged me to see if I was interested. I let them know I was available and that process took off.

I also started talking to two other large tech companies, eventually withdrawing from both. One was too far away and had a lengthy study guide for interviews that I was not willing to commit to. The other was going to make an offer but I found their interview process too heavily tilted towards technical ability with little to no discussion of culture or the day-to-day work.

Finally, I interviewed with two smaller infosec companies. On one call we mutually recognized there was no fit while the other moved on to make me a compelling offer.

I eventually went with the Apple offer as it paid more and I knew more people. The difference I didn’t adequately account for was the commute; the security company is remote and Apple involved a long shuttle commute in bay area traffic.

I was only at Apple for 6 weeks and during that time I was trying to experience it with a beginner’s mind. I read docs, code, and listened a lot. In the end, it wasn’t a culture fit for me and the commute was unbearable at 1-2 hours each way, so I quit.

The day after quitting, I set up a call with Tenable and they graciously renewed their offer to me which I accepted immediately. This time I took a couple weeks off because I really wanted to give them my best start.

Day Zero

Since my job is remote, they mailed me my gear. It arrived the Friday before my first day and I took inventory and set it all aside. Some of my accounts had been activated early, but I resisted the urge to start nesting before I was official.

Week 1

I found some bugs with the Windows 10 installation on my laptop on the first day. I spent far too much time dinking with it and finally emailed my managers and IT and got it sorted out within a few hours. I still ended up installing Linux because I found some things about the standard image too constricting.

Besides messing with my laptop, I had a few meetings and spent a lot of time reading docs and code. I spent the remainder getting to know folks over chat and video calls.

Lessons:

• Start using collective speech immediately, i.e. “ours” instead of “yours”.
• Reach out for help with IT problems more quickly, even if the issue at hand appears user-serviceable.
• Reading docs and code in downtime was a great idea.
• Spending time keeping up with chat rooms is a good way to get to know people in a remote job.
• Keep notes about things that are troublesome like my laptop issues and bumps in setting up accounts (more on this later).
• Generate 4KB ssh keys with a good password to impress your colleagues when they copy/past your pubkey.
• Install Linux sooner :)

Week 2

With a working laptop and most of my accounts set up, it was time to start digging in. I got set up on the bastions and started looking at some tickets for our databases. Getting into live machines for analysis helped expose gaps in the new-hire onboarding workflow. Where possible I updated docs and other things were added to my notes.

Lessons:

• Find a ticket to work on right away and give yourself plenty of time. I hit a few snags with access and local conventions that slowed me down.
• Try to figure things out by searching docs and poking.
• Start committing small changes to the team’s primary repos. e.g. formatting, spelling, tests, and minor bug fixes.
• Set a time limit on self-discovery and ask for help.
• I don’t always take notes, but when I do, it’s usually during my first month on the job.

Week 3

For my third week, the company flew me to headquarters for the official orientation. The first day of company overview, culture introduction, and getting to know the other new hires was enjoyable and interesting. The next couple days were largely sales-focused. My manager and one of my peers flew in and sprung me from those days. We went over our ongoing projects and prioritized work while we got a feel for each other. On the last day of the week I started researching projects we prioritized and started coordinating with other SREs on how to approach things.

Lessons:

• Get to know one or two people in your hiring “class”. It’s fun and you’ll probably run into these folks again and again regardless of which department each of you are in.
• Ask about orientation and find out if you really need to attend all of it. At the SaaS companies I’ve experienced there is a lot of content for sales folks (often much higher turnover than other departments) that might not be very relevant to an SRE.
• Remember to install your password manager on your travel device (unless international).

Week 4

In my reviews of the Cassandra setup I found a few things that needed addressing. Rather than handing off the work that needed to be done, I decided to take it on a little before I felt ready. This gave me the opportunity to start committing to our Ansible repos. I wrote a bit of shell code and updated a bunch of playbooks that led me through learning the basics I needed to work with Ansible daily. I got one project done that was blocking progress for dev teams which led to making friends in those teams as I worked with them.

By this time I was starting to get questions from managers and peers about how I was liking the company and people. I was happy to reply in the positive and kept mental notes about specific things people asked.

Lessons:

• Find a project in your wheelhouse to start with. For example, I knew Cassandra or Linux tuning would be a good place for me. Find some familiar ground and use it to launch yourself forward.
• Clearing blocker tickets is an excellent way to make friends in dev teams.
• Remember to give yourself extra time to adapt to the new environment.

Beginner’s Mind

A few years ago when I was stressing out about a talk, Aaron Morton recommended that I read Zen Mind, Beginner’s Mind. It’s an excellent book if you’re interested in Zen of course, but one thing I took away from it is the value of seeing things with a beginner’s mind. Once we’ve acclaimated to a new environment - be it a company, home, city, or vehicle - we are less likely to notice things that are obvious to beginners. I took my notes and data spanning git logs, email, Jira, and even my shell history to create a document with all the things I noticed that were good and those that needed improvement. I chose to keep things as constructive and positive as I could.

Observations:

• Pointed out some technical debt, noting that it was not bad and totally fixable.
• Wrote a quick review of the state of our Ansible repos and procedures.
• Noted some good patterns I noticed in our AWS setup, particularly that we were leveraging IAM and VPC quite well for being relatively new to cloud ops.
• Included a detailed review of the Cassandra clusters since that’s an area of expertise for me.
• Pointed out gaps in operational tooling that we should focus on.
• Gushed about how I saw the company’s stated culture show up in my interactions with people as I onboarded.
• Wrote a bit about my experience with our product and threw out my BHAG for it.
• Recommended improvements for gear & provisioning specific to technical staff.
• Called out the need to simplify our edge architecture (under way!).
• Talked briefly about my orientation experience and recommended some changes for engineering hires.
• Linked some research about IT procedures that needed to updated.
• Celebrated that folks were using my name and pronouns consistently.
• Offered to advise IT folks on some improvements to laptop deploys.

I wrote the doc with my manager, their manager, and my peers in mind. The last I heard it was being passed around the C-levels with positive remarks. Even though I resolved some blockers and deployed a significant upgrade, this document was by far my largest contribution to date. The people I joined were used to their environment and by way of keeping some notes and adding some narrative, I was able to reach a large chunk of the management team and have an impact that benefitted the whole organization.

Wrapup

Starting a new job is stressful regardless of your role. For those of us working in operations and site reliability, the tech work is usually the easy part. Getting acclaimated into the social fabric and pacing yourself for the long haul is something I think we all struggle with. By being methodical and mindful about your approach to your first few weeks, it’s possible to make an impact without the usual stress of being new and vulnerable.

As with any tips of this sort, your experience may vary. Some things that worked for me may also work for you. Above all, find what works for you, make a plan, and communicate, communicate, communicate.