summer weddings

When I graduated college and started working at amazon, my very first office was a big room I shared with the rest of the production QA team. There were 5 of us there. Geoffrey, who was already married. And David, Jason, Russell, and me. 9 years later, and I've been to each of their weddings. (Well, except Geoffrey, who had beaten everyone to the punch!) It kind of made me pause and think about how blessed I've been to have such great friends.

David - Aix-en-Provence, June 2002



Jason - Seattle, August 2005



Russell - Orcas Island, July 2008

Amazon Web Services DCI - The Dog Computer Interface

This is Declan in early 2000 at Amazon.com:



It was an early version of the DCI - Dog Computer Interface. Here is a later version, from 2003 or so:


(notice integration with Official Amazon Door Desks in this one, enabling each web dev to have a Declan available under their work surface)

And today, the AWS group announces the official release of the DCI - The Dog Computer Interface!

"adserver on master"

I found this slip of paper in my perl cookbook a few mornings ago. It's probably from spring 2001 or so, and it was written by Brian & left on my desk.



What is it? It's a small piece of the deployment system for the first amazon ad server. Brian and I worked on it together, along with a couple other amazonians, and we all sat on one hallway on the first floor of pacmed. I was probably out walking Declan when he pushed the latest ad server software and needed a sign off before it could go out, so he wrote this note and left it on my desk. Amazon basically has three levels of deployment. First there's a test server, where all the data is from the test domain. Then there's a "master" environment that this note refers to- and finally production. Code has to be signed off on by QA (that would have been me) on the test server and then on the master server again before it can be pushed.

I'm trying to remember how I let him know that I'd given my signoff... it was probably over email. I'm not sure that the ad server got an official sign off tracking form until it was a year or two old. You'll notice that at some point over the last seven years I doodled in red in the eyes of the smiley face. I have no idea when that was- possibly while I was running the preflight, waiting for something to load. I also noticed the "please" and "thanks" in this note - despite all the protests he'll give you, Brian is really a southern gentleman at heart.

This has brought back a lot of great memories from our initial launch of the ad server and the first year that it was up. Making apricot scones the weekend before launch for our Sunday AM bug bash. Our friend Greg, who was a web dev at the time, naming the templating language that we wrote ads in. (It's called "schusteranto" after Brian. Pretty cool that he has a programming language named after him, even if it is just an tiny substitution language.) Sitting directly behind PacMed's espresso stand, which was sooo handy sometimes. Declan learning how to play frisbee, instead of running away terrified from one when you tossed it. Walking home from work at 8 PM that summer, as the dotcom crash kept going, watching the sun setting while I walked up 2nd Ave. Seems like yesterday in some ways.

AB Testing

37 signals just did a blog post on "Secrets to Amazon's Success. Here are my favorite three points:

Use measurement and objective debate to separate the good from the bad. I’ve been to several presentations by ex-Amazoners and this is the aspect of Amazon that strikes me as uniquely different and interesting from other companies. Their deep seated ethic is to expose real customers to a choice and see which one works best and to make decisions based on those tests.

Getting rid of the influence of the HiPPO’s, the highest paid people in the room. This is done with techniques like A/B testing and Web Analytics. If you have a question about what you should do code it up, let people use it, and see which alternative gives you the results you want.

Have a way to rollback if an update doesn’t work. Write the tools if necessary.

- http://www.37signals.com/svn/posts/600-secrets-to-amazons-success

That matches up to two of the biggest things that I learned at amazon- you don't know what your customers will do, so you shouldn't guess, you should A/B test. And you're going to screw up unbelievably, and constantly, so you need instant rollbacks. Here's the thing. Amazon was my first job out of college, so I really didn't know any other way to do things. One of my first testing jobs there was to test the very first automated A/B mechanism on the retail site (a version that had died long before I ended up leaving the company- like all features I worked on my first few years, it was re-done a few times to be better than the first version). Testing it wasn't all that fancy, we just checked that visitors were dropped into buckets, saw the treatments tied to those buckets, and reporting worked correctly. The bigger impact on me was the lesson that this was the most important way that new features were going to be launched. It's an incredible testament to the success of A/B testing in the company culture that when I left 7 years later, nearly every new update (and even little tweaky things) to the site was launched initially as an A/B test. Russell used to have all of his launch plaques up on the desk over his wall, with a couple flipped upside down. The upside down plaques were- yes- the projects that bombed in the launch A/B testing phase.

I couldn't even begin to describe the number of rollbacks we did over the years. If you can think of it, we screwed it up. The important part was that we could recover quickly by backing out the change, and that we had a culture of learning from the screwups. In the retail world, there was a tradition of writing "How I broke the website" emails. I will admit to writing a couple- they went out to all the engineers, and carefully analyzed how you had, well, broken the website. You discussed how the problem should have been caught, and what you were going to do differently going forward to prevent that from happening again. Now that I've been at some other companies, I can see how critical the emails were. Breaking things was bad, but more important was sharing how you messed up, and taking the time to do a post mortem of the mistake, and then freely circulating the information. It wasn't the mea culpa that mattered- although accepting responsibility was a big part of it. Larger, really, was the learning you shared with the teams, and the environment where you learned how to recover gracefully from large messes.

37 signals mentions the 'just do it' awards that Jeff handed out. Less known outside of amazon are the 'door desk awards'. I'm not sure that they're still given out, but they used to be handed out at every All Hands meeting. A door desk award was similar to a 'just do it' award, but it was given to an individual who tried something- without their manager's approval- and failed. Their feature bombed in an A/B test, their code blew up, the project never launched, whatever. It was another piece of making an engineering culture where you should just go for it, because failure wasn't the end of the world.

awww baby

The following items are being prepared for shipment by Amazon.com:
---------------------------------------------------------------------
Qty Item Price Shipping Subtotal
---------------------------------------------------------------------
Amazon.com items (Sold by Amazon.com, LLC) :
1 Harry Potter and the Death... $17.99 1 $17.99

when I learned what XSS is

Here's another old Amazon history post. February, 2000- I'd had Declan for barely over a month, and I remember being outside with him, walking around our parking lot, on a rather sunny day. (It's never sunny in Seattle in February, that's why I particularly remember this.) I was the oncall frontend QA- basically, if we had to do an emergency content push to the onlines, I would be the one checkpoint. Small responsibility for a 22 year old. My pager went off, I went in, and logged into my computer, and read the problem. It linked to CERT Advisory On Malicious HTML Tags on slashdot. That was the birth of XSS.

I spent the next several hours testing pushes for every single page on the site that echoed back user input. On a site like amazon, you can imagine what that was like- I seem to recall that a lot of my time was frantically deleting pages from my pager because it kept running out of space for stored messages. Thankfully I worked with some great people, I remember farming out a lot of the testing to Jason, who was still really a newbie at that time. This is one of my starkest memories: we had the slashdot article on this open, and would reload over and over again reading the comments as more vulnerable sites were found, more exploits related to this came to light. The comments are still an interesting read today.

It took a few days for me to wrap my head around what this bug was, at the time I was just trying to test with the sample input we had, without totally following the complete theory of what we were doing. To be fair, I doubt anyone that understood that. I can talk a lot now about filtering vs escaping, why I love <plaintext>, and so on, but that day was more about survival mode. Lots of fun, though, and another insane amazon experience that I wouldn't trade for anything.

Omakase Crime Stoppers!

Bandon's dad, one of the Omakase developers, sent me this link this morning. Who knew that an adserver could help uncover something like this!

Exposure or Exploitation? Mark Cuban's HDNet releases performance DVDs without telling local and national artists -- Is it the price of promotion or an abuse of creative rights?

From the article, here's how this all was uncovered:

In fact, we only found out about the existence of the DVD’s by sheer chance. Mike was testing a new contextual ad unit from Amazon on TexasGigs, and it pulled up ads for the video of the Cunniff benefit. And then, in the course of searching around for more information, we found episodes of True Music available as paid downloads on Google Video.

Based on the heavy use of local bands, we then sought to get some info on the series from HDNet and the bands involved for what we thought was a simple story about cool DVDs of local bands hitting the market.

The problem was that when we contacted bands for information, we got one of two answers:

“What?

or

“What the hell?”

Quotable Geeks

I collected a few goofy emails and quotes over my years at amazon, and here are some of my sharable favorites. There are a few which I love more, but I can't really post here. If you know me, you know which ones they are anyway. So...

"I don't think my ego has ever been more bruised. Imagine, being mocked by developers. Shouldn't I be taking your lunch money?" --A non-technical good friend, circa 2001ish

"Correct. The first set of numbers is an order id. Walk softly over the rice paper Grasshopper." --Alan

"The Fantastic Four is such a rip off of The Incredibles!" --Jonathan

Jon: "So this is SOL?" James: "Give or take."

John: "Let's reinvent the wheel." Jon: "Our wheels have to be upside down."

"Good morning Japan!" --David, 3:24 PM JST

"Let's replace the working software with Folgers Crystals!" --Scott

(after my friend's dog, Bandon, ran away...) "Thanks for all who helped look for Bandon. FWIW, he was where we suspected: smoking a cigarette, playing poker in the basement of the parking garage." --Greg

Pop Conference

emplive.org - Visit - Education - Pop Conference: "The 2006 Experience Music Project Pop Conference-Seattle - April 27 - 30, 2006" It's the annual Pop Music conference at EMP this weekend. I'm going to try to make it over on Saturday, as a bunch of the panels sound interesting.
I'm trying to find some new songs to download from the iTunes store tonight, but nothing really looks interesting. I wish they had better recommendations, although honestly, my amazon recs are never terribly great for discovering "just been released by a new artist" stuff either. I have lots of thoughts on music recommendations and the new (to me) artist problem that might best be summed up by "I wish there was a way to make a taste template and then match new songs against that." Lots of work has been done in lots of places to define what a "taste template" might be, but I've never seen one that really works. For instance, I love hip hop, but how do I describe the songs that I love and the songs that make me switch the radio station? What attributes do I rank Ride on against Slowdive against Curve, and what of those attributes make me not like a lot of newly released shoegazing?

On a less geeky note, remember the feeling you'd get in grade school the night before the last day of school? Ahhh bliss. Summer vacation never really quite lived to what you might have hoped it would be like, and you knew it, but right before it started was always the greatest time. I liked it even better than Christmas or my birthday. So, I haven't felt this way since I was about 10, and I'm loving it.

Drambuie Cocktails

Traditional Scottish Recipes - Drambuie Cocktails: Warm Woolly Sheep
1 measure Scotch whisky
1 1/2 measure Drambuie
Fill with warm Milk
Mix Scotch and Drambuie, top with warm milk.

I always drink my Drambuie either straight up, or by putting a few drops of it on top of a small glass of scotch. I'm half tempted to try this one, though; I can't even imagine what warm milk and scotch would taste like.

Today I decided that I'm moving back to the east coast. I'll be just shy of 7 years at amazon- 7 years which felt like 6 months. I feel like I just launched the Kitchen store last month, web services last week, and blinked and then here I am. So here is my favorite amazon memory... in summer 2002, Russell got me to sign up for his broomball team. It was Jeff Bezos's team, made up of a bunch of people from various parts of the company. I'd played on Jeff's team before (I got a compilment from him on my creative goal tending skills one year when I sat down on the ball to stop it from being pushed over the goal line), and always one of the big problems was coordinating everyone out on the field. So we decided to hold a practice game out on the south lawn after work. It was an amazing August evening- sunny, warm, just beautiful. I went out and helped tape up some broomball sticks. Then we assembled on the grass and split the team in two. Jeff was out there with us- I seem to recall he was playing barefoot in his work clothes. I was marginally more sensibly dressed in some running gear. We all faced off, and played an hour of great broomball. At the picnic, we made it to the semi-finals, but didn't win. It was still a ton of fun.

Geeking with Greg: Early Amazon: Door desks

This is great- Greg is doing a series of posts about old-skool Amazon. Here's his post on door desks:

Geeking with Greg: Early Amazon: Door desks: "In addition to being inexpensive, door desks offered a lot of surface area. Put your computer monitor on top and in barely makes a dent. If Amazon wasn't so bloody cheap, you could have put three more monitors up there. Plenty of space for all of your crap.Ergonomically, door desks leave a lot to be desired. Keyboards were usually too high. Typing for hours could be uncomfortable. And those angle brackets have sharp edges; accidentally scrapping exposed flesh against those was a mistake that wouldn't be repeated.But door desks came to symbolize the Amazon frugal culture. They took on a live of their own. Years later, in 2001, there was a 6.8 magnitude earthquake here in Seattle. We were in a different building by then -- the attractive PacMed building up on Beacon Hill above downtown Seattle -- but we still had our door desks. And I can't tell you how happy we were to have that door desk over our heads as the building shuttered and swayed around us."

Door desks are great. I love mine, it's more than 5 feet long, and insanely deep. Add a keyboard tray, and it's perfect. I have three computers on it, all my O'Reilly books, a coffeemaker, and there's still room to pile up papers and notebooks.

I really enjoyed reading his old amazon memories, so here's one of mine... I was hired in Dec 1998, but I still had a semester of college to finish up. So come May 1999, I began to think about my waiting job in Seattle. I sent my manager-to-be an email, telling him I was going to go out to Seattle for a week in May, to apartment hunt, and asking if I could stop by and say hello. He invited me to come by and meet everyone while I was there.

So I flew to Seattle, carrying a small thermos with water which held my fish, Puck. My friend- who worked at Microsoft- was going to watch Puck for me for my last few weeks before I moved out. I stayed with her up in Greenlake while I searched Belltown for a cheap apartment. (There were more of those then than there are now.) One day, I walked down to the Columbia Building and met my new teammates. I saw the Bat Cave, where most of the web devs sat. Everyone was using Emacs. I think Bosco showed me how to goof around with the desktops of people who had done "xhost +" by opening crazy windows on their screens. Then I went up and saw Pacmed, where we would soon be moving. And finally, I was invited out to a big lunch with lots of web dev folks, at what is now Dragonfish. That was a great lunch- I tried frantically to remember everyone's name and ingest all the moving-to-Seattle advice I got.

Two days later, I flew back to Boston and graduated. Then I flew back to Seattle, and settled into the apartment I'd found. My plan was to take 3 weeks to unpack and hang out before starting work. But quickly, I was bored doing nothing- I'd unpacked my boxes, I'd seen Star Wars Episode One at the Cinerama twice, and I didn't know what to do with myself. I asked my manager if I could start a little early. That Monday, I took a bus up to Pacmed to my first day as an official web geek. That was June 21, 1999.

We used to have paper badges, so my first day I got a little blue laminated piece of paper with my login on it. I set up my computer, and settled my fish onto my desk. The next day- my first real day of work- an email went out to all the developers. We were about to launch the toy store, and the photographers (who shot images of all the toys that would be on the site) had an emergency! Turns out, all the legos we had to photograph were in boxes. Photographs of boxes of legos are BORING, and so a plea was sent to the geeks of the company. Come assemble legos, and we'll feed you pizza! So my manager and I showed up, ate pizza, and assembled legos. We got our asses kicked by the 6 year old son of one of the PMs, who could assemble lego kits with lightening speed.

The part that killed me was that the toy store was a big secret, so I couldn't tell any of my friends that I had the greatest job ever, programming and putting together legos! After the store launched a few weeks later, I was able to pull up some toys detail pages & show off my creations.