On May 17th, 2006, I responded somewhat off-the-cuff to an informal job posting for possible openings in Yahoo!'s System Administration team. One thing led to another and the next thing I know I was sitting in the plane flying back from my on-site interview in Sunnyvale with a job offer in my hands. (It was the Worldcup, and Germany played Poland, but only I and another guy a few rows in front of me seemed to be interested in watching the game. Germany won that game.) In August 2006, my wife and I moved to San Francisco and I started my new job on September 1st, 2006.
Five years and a little over three months later, I'm leaving Yahoo!; tomorrow is my last day. These five years have been quite a whirlwind; if you care, join me for a brief recap:
I started out at Yahoo! in the System Administration (SA) team, which at that time consisted of about 10 (or so) SAs in Sunnyvale plus a handful of SAs in Bangalore. We were in charge of all production systems; corporate systems were at that time supported by a separate team. In the first few months, I created a few tools that made day-to-day tasks easier, did some regular ticket work but ultimately focused on the development of one of our deployment systems. (Four weeks into the new job, Beck gave a concert at Yahoo!'s Open Hack Day, which felt very dot-commy-bubbly, in a good way.) I wrote the code to enable unattended cloning of our FreeBSD systems to go hand in hand with the existing RHEL (kickstart-based) deployment. This project was later picked up to develop completely unattended deployment and configuration within less than 30 minutes after hardware was approved in the then notorious Hardware Review Committee, in which David Filo was known to pose odd wagers such as the ``cupcake challenge'' to the requesters.
Early in 2007, soon after I gave a presentation entitled Useless Use of * at SCALE, my manager came to me asking ``How many systems do we have that are not prepared for the new Daylight Savings Time?'', which led to an interesting and rather intense couple of weeks, during which I developed scanmaster to allow us to rapidly ascertain certain pieces of information about all our hosts. After DST came and went without incident, I gave a presentation at one of the BayLISA meetings that were held at Yahoo! (there used to be a video of the meeting floating around Yahoo! Videos, but that does no longer seem to exist).
I learned a lot about our environment and about scalability in those couple of weeks, and the system has been extended, enhanced and been in continuous use ever since. In February of this year, at long last, scanmaster was open sourced, and I gave another presentation about it to friends at Twitter.
After my manager left the company in the Spring of 2007, I had the opportunity to join the "Network, Systems and Storage Architecture Team" to help fill in the "Systems" component therein. Myself and a colleague formed the "Systems Architecture" team, providing guidance to the rest of the company for anything that falls between Layer 2 and Layer 7. Being a new team, it wasn't easy to get buy-in and to set the direction for Operating System roadmaps, configuration management, infrastructure services etc., but with time our team grew. In this team, I helped develop the company's IPv6 strategy (and provided the internal IPv6 playground), review the OS strategy, continued ongoing discovery scans and designed a scalable logging architecture that had to handle syslog servers getting hammered by more than 55K messages per second.
I wrote a configuration management system for handling hosts in ``hostile'' environments (leased space in data centers on the edge of the network, ie hosts we don't have complete control over), advised on the development of a new host deployment system (to replace the one I helped write earlier), on the development of a new configuration management system (which now, at long last, has reached over 60% of our hosts) and contributed to a ground-breaking new load-balancing technology called ``L3DSR'', which I presented at NANOG 51 in Miami earlier this year and which has become the basis of Yahoo!'s load balancer strategy.
I was a member of the Open Source Working Group, overseeing incoming and outgoing contributions, where I tried to make it easier for Yahoos to contribute to Open Source projects and to make public their own work; I was a member of the Engineering Standards Group, which sets direction for the entire company on topics as wide-ranging as which programming languages to use to what timezones our servers should be in (I drove the motion to finally make UTC the standard, hooray!).
After a number of reorganizations, I ended up inheriting the ownership of our (then) largest deployed configuration management system, a rather awful piece of around 10K lines of perl-gunk deployed on nearly 100K machines that, despite its various awfulnesses continues to work and do what it was written to do. (The system is still in use on a number of hosts that's larger than what most company's ever will have, but a newer, better system is on its way to replace the old cruft.)
I wrote a vulnerability checker (conceptually based on NetBSD's audit-packages tool), an IRC bot currently sitting in ~120 channels on our corporate IRC server and which ultimately made the quantum leap to twitter, a signature verifying command interpreter (about which I wrote elsewhere) and a number of other miscellaneous tools.
I have reported 2070 bugs in our internal Bugzilla and closed 914 myself. I've received over 1822491 emails from 2313 addresses and sent over 22297 emails myself and am subscribed to 171 internal mailing lists. I have changed my UNIX, windows and intranet passwords at least twice a year and still find systems where I have to use my initial password from 2006. I've reported to a total of 9 managers -- 5 different managers in 2011 alone...
The last five years included two trans-continental moves (NY->SF in 2006, SF->NY in 2008), and both of my daughters were born while I worked at Yahoo!, one in 2007 and the other back in September. I have met a number of great people, learned a lot about System Administration and all sorts of things relating to computers, the internet, people, politics, programming, software engineering, Silicon Valley, San Francisco, and this and that.
It's been a very interesting five years, and I'm thankful for the experiences and opportunities given to me. To all the people I've worked with during this time: Thanks! See you... out there.
December 20, 2011