The "Harry Potter" problem in recommendations

Greg Linden covered the Harry Potter problem in a blog post on recommendation technologies a few years ago:

A very sharp and experienced developer named Eric wrote the first version of similarities that made it out to the Amazon website. It was great working with Eric. I learned much from him over the years.

The first version of similarities was quite popular. But it had a problem, the Harry Potter problem.

Oh, yes, Harry Potter. Harry Potter is a runaway bestseller. Kids buy it. Adults buy it. Everyone buys it.

So, take a book, any book. If you look at all the customers who bought that book, then look at what other books they bought, rest assured, most of them have bought Harry Potter.

-http://glinden.blogspot.com/2006/03/early-amazon-similarities.html


When I worked on the personalization team we were still struggling with the problem- there are definite ways to identify a Harry Potter problem, but you have to remember to apply them. Adding to that, within certain genres there are Harry Potter books/music albums that are only runaway successes within those genres. If you compared those books to the general list of books that amazon sells, they wouldn't look like books that everyone has bought. Taking it a step further, if then if you narrow the scope to only related books you'll find that they are crazy popular.

The biggest side effect of the Harry Potter problem is that it weakens recommendations. For instance, I've bought the O'Reilly regex pocket book and the O'Reilly Python Cookbook and Ruby Cookbook. From those three books, you can pretty easily peg me as a web nerd and safely recommend a Steve Souder's website performance book. Those are very strongly correlated purchases in a narrow band of interest. However, because I'm a geek, I've also bought Neal Stephenson's latest book, Anathem. As have a few hundred thousand OTHER geeks. We could say that Anathem is a nerd's Harry Potter.

So I received an email today from amazon with a list of recommended books, most of which were based off Anathem and Daniel Silva's latest book, Moscow Rules (great book but also a bit of a Harry Potter widely-bought book). As you might guess, the recommendations were really bad. I wish that email had a link that I could click that would say "never recommend any of these books to me again please" -I could go to each detail page and mark that, but it would take a massive amount of time.

summer weddings

When I graduated college and started working at amazon, my very first office was a big room I shared with the rest of the production QA team. There were 5 of us there. Geoffrey, who was already married. And David, Jason, Russell, and me. 9 years later, and I've been to each of their weddings. (Well, except Geoffrey, who had beaten everyone to the punch!) It kind of made me pause and think about how blessed I've been to have such great friends.

David - Aix-en-Provence, June 2002



Jason - Seattle, August 2005



Russell - Orcas Island, July 2008

Amazon Web Services DCI - The Dog Computer Interface

This is Declan in early 2000 at Amazon.com:



It was an early version of the DCI - Dog Computer Interface. Here is a later version, from 2003 or so:


(notice integration with Official Amazon Door Desks in this one, enabling each web dev to have a Declan available under their work surface)

And today, the AWS group announces the official release of the DCI - The Dog Computer Interface!

"adserver on master"

I found this slip of paper in my perl cookbook a few mornings ago. It's probably from spring 2001 or so, and it was written by Brian & left on my desk.



What is it? It's a small piece of the deployment system for the first amazon ad server. Brian and I worked on it together, along with a couple other amazonians, and we all sat on one hallway on the first floor of pacmed. I was probably out walking Declan when he pushed the latest ad server software and needed a sign off before it could go out, so he wrote this note and left it on my desk. Amazon basically has three levels of deployment. First there's a test server, where all the data is from the test domain. Then there's a "master" environment that this note refers to- and finally production. Code has to be signed off on by QA (that would have been me) on the test server and then on the master server again before it can be pushed.

I'm trying to remember how I let him know that I'd given my signoff... it was probably over email. I'm not sure that the ad server got an official sign off tracking form until it was a year or two old. You'll notice that at some point over the last seven years I doodled in red in the eyes of the smiley face. I have no idea when that was- possibly while I was running the preflight, waiting for something to load. I also noticed the "please" and "thanks" in this note - despite all the protests he'll give you, Brian is really a southern gentleman at heart.

This has brought back a lot of great memories from our initial launch of the ad server and the first year that it was up. Making apricot scones the weekend before launch for our Sunday AM bug bash. Our friend Greg, who was a web dev at the time, naming the templating language that we wrote ads in. (It's called "schusteranto" after Brian. Pretty cool that he has a programming language named after him, even if it is just an tiny substitution language.) Sitting directly behind PacMed's espresso stand, which was sooo handy sometimes. Declan learning how to play frisbee, instead of running away terrified from one when you tossed it. Walking home from work at 8 PM that summer, as the dotcom crash kept going, watching the sun setting while I walked up 2nd Ave. Seems like yesterday in some ways.

cultures

I've been thinking for a long time that I should blog a couple of the more amusing culture clash exchanges that my sailor and I have shared. Let's be serious, the dotcom world and the military are... um... different. Vast chasms of different. We've gotten a lot of humor out of how each of us lives in opposite worlds over the last several years.

My favorite of all time was in August 2004, when I was sitting on the floor of the bridge of a boat that he was XO on, wearing jeans and flip flops, and working on my laptop. There were a couple sailors visiting from a nearby station and I chatted with them for a while. They were talking about how sad it was that I was usually stuck in an office on land during the day. I countered with 'yes, but I go to work in flip flops all summer.' Sadly that didn't sell them on the wonders of dotcom civilian life. I think they must have been imaging slipping on the deck wearing flip flops during a big storm and falling into the water- not something I worry so much about in my job.

This morning Brian and I were talking about annual reviews and how our career goals could kind of be summed up as 'write a lot of interesting code.' My sailor is also writing his version of annual reviews, for the LTJGs who work for him, and he didn't think 'write interesting code' was a valid career goal. I think his exact words were 'honestly, if you were in the CG, you would be kicked out for that. it doesn't show leadership.' Yes, because leadership is exactly what you need when you have a crashing webserver. Well, ok, you need some leadership, but you have a more pressing need for super sharp nerds.

AB Testing

37 signals just did a blog post on "Secrets to Amazon's Success. Here are my favorite three points:

Use measurement and objective debate to separate the good from the bad. I’ve been to several presentations by ex-Amazoners and this is the aspect of Amazon that strikes me as uniquely different and interesting from other companies. Their deep seated ethic is to expose real customers to a choice and see which one works best and to make decisions based on those tests.

Getting rid of the influence of the HiPPO’s, the highest paid people in the room. This is done with techniques like A/B testing and Web Analytics. If you have a question about what you should do code it up, let people use it, and see which alternative gives you the results you want.

Have a way to rollback if an update doesn’t work. Write the tools if necessary.

- http://www.37signals.com/svn/posts/600-secrets-to-amazons-success

That matches up to two of the biggest things that I learned at amazon- you don't know what your customers will do, so you shouldn't guess, you should A/B test. And you're going to screw up unbelievably, and constantly, so you need instant rollbacks. Here's the thing. Amazon was my first job out of college, so I really didn't know any other way to do things. One of my first testing jobs there was to test the very first automated A/B mechanism on the retail site (a version that had died long before I ended up leaving the company- like all features I worked on my first few years, it was re-done a few times to be better than the first version). Testing it wasn't all that fancy, we just checked that visitors were dropped into buckets, saw the treatments tied to those buckets, and reporting worked correctly. The bigger impact on me was the lesson that this was the most important way that new features were going to be launched. It's an incredible testament to the success of A/B testing in the company culture that when I left 7 years later, nearly every new update (and even little tweaky things) to the site was launched initially as an A/B test. Russell used to have all of his launch plaques up on the desk over his wall, with a couple flipped upside down. The upside down plaques were- yes- the projects that bombed in the launch A/B testing phase.

I couldn't even begin to describe the number of rollbacks we did over the years. If you can think of it, we screwed it up. The important part was that we could recover quickly by backing out the change, and that we had a culture of learning from the screwups. In the retail world, there was a tradition of writing "How I broke the website" emails. I will admit to writing a couple- they went out to all the engineers, and carefully analyzed how you had, well, broken the website. You discussed how the problem should have been caught, and what you were going to do differently going forward to prevent that from happening again. Now that I've been at some other companies, I can see how critical the emails were. Breaking things was bad, but more important was sharing how you messed up, and taking the time to do a post mortem of the mistake, and then freely circulating the information. It wasn't the mea culpa that mattered- although accepting responsibility was a big part of it. Larger, really, was the learning you shared with the teams, and the environment where you learned how to recover gracefully from large messes.

37 signals mentions the 'just do it' awards that Jeff handed out. Less known outside of amazon are the 'door desk awards'. I'm not sure that they're still given out, but they used to be handed out at every All Hands meeting. A door desk award was similar to a 'just do it' award, but it was given to an individual who tried something- without their manager's approval- and failed. Their feature bombed in an A/B test, their code blew up, the project never launched, whatever. It was another piece of making an engineering culture where you should just go for it, because failure wasn't the end of the world.

awww baby

The following items are being prepared for shipment by Amazon.com:
---------------------------------------------------------------------
Qty Item Price Shipping Subtotal
---------------------------------------------------------------------
Amazon.com items (Sold by Amazon.com, LLC) :
1 Harry Potter and the Death... $17.99 1 $17.99

when I learned what XSS is

Here's another old Amazon history post. February, 2000- I'd had Declan for barely over a month, and I remember being outside with him, walking around our parking lot, on a rather sunny day. (It's never sunny in Seattle in February, that's why I particularly remember this.) I was the oncall frontend QA- basically, if we had to do an emergency content push to the onlines, I would be the one checkpoint. Small responsibility for a 22 year old. My pager went off, I went in, and logged into my computer, and read the problem. It linked to CERT Advisory On Malicious HTML Tags on slashdot. That was the birth of XSS.

I spent the next several hours testing pushes for every single page on the site that echoed back user input. On a site like amazon, you can imagine what that was like- I seem to recall that a lot of my time was frantically deleting pages from my pager because it kept running out of space for stored messages. Thankfully I worked with some great people, I remember farming out a lot of the testing to Jason, who was still really a newbie at that time. This is one of my starkest memories: we had the slashdot article on this open, and would reload over and over again reading the comments as more vulnerable sites were found, more exploits related to this came to light. The comments are still an interesting read today.

It took a few days for me to wrap my head around what this bug was, at the time I was just trying to test with the sample input we had, without totally following the complete theory of what we were doing. To be fair, I doubt anyone that understood that. I can talk a lot now about filtering vs escaping, why I love <plaintext>, and so on, but that day was more about survival mode. Lots of fun, though, and another insane amazon experience that I wouldn't trade for anything.

Omakase Crime Stoppers!

Bandon's dad, one of the Omakase developers, sent me this link this morning. Who knew that an adserver could help uncover something like this!

Exposure or Exploitation? Mark Cuban's HDNet releases performance DVDs without telling local and national artists -- Is it the price of promotion or an abuse of creative rights?

From the article, here's how this all was uncovered:

In fact, we only found out about the existence of the DVD’s by sheer chance. Mike was testing a new contextual ad unit from Amazon on TexasGigs, and it pulled up ads for the video of the Cunniff benefit. And then, in the course of searching around for more information, we found episodes of True Music available as paid downloads on Google Video.

Based on the heavy use of local bands, we then sought to get some info on the series from HDNet and the bands involved for what we thought was a simple story about cool DVDs of local bands hitting the market.

The problem was that when we contacted bands for information, we got one of two answers:

“What?

or

“What the hell?”

Quotable Geeks

I collected a few goofy emails and quotes over my years at amazon, and here are some of my sharable favorites. There are a few which I love more, but I can't really post here. If you know me, you know which ones they are anyway. So...

"I don't think my ego has ever been more bruised. Imagine, being mocked by developers. Shouldn't I be taking your lunch money?" --A non-technical good friend, circa 2001ish

"Correct. The first set of numbers is an order id. Walk softly over the rice paper Grasshopper." --Alan

"The Fantastic Four is such a rip off of The Incredibles!" --Jonathan

Jon: "So this is SOL?" James: "Give or take."

John: "Let's reinvent the wheel." Jon: "Our wheels have to be upside down."

"Good morning Japan!" --David, 3:24 PM JST

"Let's replace the working software with Folgers Crystals!" --Scott

(after my friend's dog, Bandon, ran away...) "Thanks for all who helped look for Bandon. FWIW, he was where we suspected: smoking a cigarette, playing poker in the basement of the parking garage." --Greg

Pop Conference

emplive.org - Visit - Education - Pop Conference: "The 2006 Experience Music Project Pop Conference-Seattle - April 27 - 30, 2006" It's the annual Pop Music conference at EMP this weekend. I'm going to try to make it over on Saturday, as a bunch of the panels sound interesting.
I'm trying to find some new songs to download from the iTunes store tonight, but nothing really looks interesting. I wish they had better recommendations, although honestly, my amazon recs are never terribly great for discovering "just been released by a new artist" stuff either. I have lots of thoughts on music recommendations and the new (to me) artist problem that might best be summed up by "I wish there was a way to make a taste template and then match new songs against that." Lots of work has been done in lots of places to define what a "taste template" might be, but I've never seen one that really works. For instance, I love hip hop, but how do I describe the songs that I love and the songs that make me switch the radio station? What attributes do I rank Ride on against Slowdive against Curve, and what of those attributes make me not like a lot of newly released shoegazing?

On a less geeky note, remember the feeling you'd get in grade school the night before the last day of school? Ahhh bliss. Summer vacation never really quite lived to what you might have hoped it would be like, and you knew it, but right before it started was always the greatest time. I liked it even better than Christmas or my birthday. So, I haven't felt this way since I was about 10, and I'm loving it.

Drambuie Cocktails

Traditional Scottish Recipes - Drambuie Cocktails: Warm Woolly Sheep
1 measure Scotch whisky
1 1/2 measure Drambuie
Fill with warm Milk
Mix Scotch and Drambuie, top with warm milk.

I always drink my Drambuie either straight up, or by putting a few drops of it on top of a small glass of scotch. I'm half tempted to try this one, though; I can't even imagine what warm milk and scotch would taste like.

Today I decided that I'm moving back to the east coast. I'll be just shy of 7 years at amazon- 7 years which felt like 6 months. I feel like I just launched the Kitchen store last month, web services last week, and blinked and then here I am. So here is my favorite amazon memory... in summer 2002, Russell got me to sign up for his broomball team. It was Jeff Bezos's team, made up of a bunch of people from various parts of the company. I'd played on Jeff's team before (I got a compilment from him on my creative goal tending skills one year when I sat down on the ball to stop it from being pushed over the goal line), and always one of the big problems was coordinating everyone out on the field. So we decided to hold a practice game out on the south lawn after work. It was an amazing August evening- sunny, warm, just beautiful. I went out and helped tape up some broomball sticks. Then we assembled on the grass and split the team in two. Jeff was out there with us- I seem to recall he was playing barefoot in his work clothes. I was marginally more sensibly dressed in some running gear. We all faced off, and played an hour of great broomball. At the picnic, we made it to the semi-finals, but didn't win. It was still a ton of fun.

nerf chaos!

My office window this morning. (I didn't shoot all those darts at the window, but I did wield the digital camera.)

Geeking with Greg: Early Amazon: Door desks

This is great- Greg is doing a series of posts about old-skool Amazon. Here's his post on door desks:

Geeking with Greg: Early Amazon: Door desks: "In addition to being inexpensive, door desks offered a lot of surface area. Put your computer monitor on top and in barely makes a dent. If Amazon wasn't so bloody cheap, you could have put three more monitors up there. Plenty of space for all of your crap.Ergonomically, door desks leave a lot to be desired. Keyboards were usually too high. Typing for hours could be uncomfortable. And those angle brackets have sharp edges; accidentally scrapping exposed flesh against those was a mistake that wouldn't be repeated.But door desks came to symbolize the Amazon frugal culture. They took on a live of their own. Years later, in 2001, there was a 6.8 magnitude earthquake here in Seattle. We were in a different building by then -- the attractive PacMed building up on Beacon Hill above downtown Seattle -- but we still had our door desks. And I can't tell you how happy we were to have that door desk over our heads as the building shuttered and swayed around us."

Door desks are great. I love mine, it's more than 5 feet long, and insanely deep. Add a keyboard tray, and it's perfect. I have three computers on it, all my O'Reilly books, a coffeemaker, and there's still room to pile up papers and notebooks.

I really enjoyed reading his old amazon memories, so here's one of mine... I was hired in Dec 1998, but I still had a semester of college to finish up. So come May 1999, I began to think about my waiting job in Seattle. I sent my manager-to-be an email, telling him I was going to go out to Seattle for a week in May, to apartment hunt, and asking if I could stop by and say hello. He invited me to come by and meet everyone while I was there.

So I flew to Seattle, carrying a small thermos with water which held my fish, Puck. My friend- who worked at Microsoft- was going to watch Puck for me for my last few weeks before I moved out. I stayed with her up in Greenlake while I searched Belltown for a cheap apartment. (There were more of those then than there are now.) One day, I walked down to the Columbia Building and met my new teammates. I saw the Bat Cave, where most of the web devs sat. Everyone was using Emacs. I think Bosco showed me how to goof around with the desktops of people who had done "xhost +" by opening crazy windows on their screens. Then I went up and saw Pacmed, where we would soon be moving. And finally, I was invited out to a big lunch with lots of web dev folks, at what is now Dragonfish. That was a great lunch- I tried frantically to remember everyone's name and ingest all the moving-to-Seattle advice I got.

Two days later, I flew back to Boston and graduated. Then I flew back to Seattle, and settled into the apartment I'd found. My plan was to take 3 weeks to unpack and hang out before starting work. But quickly, I was bored doing nothing- I'd unpacked my boxes, I'd seen Star Wars Episode One at the Cinerama twice, and I didn't know what to do with myself. I asked my manager if I could start a little early. That Monday, I took a bus up to Pacmed to my first day as an official web geek. That was June 21, 1999.

We used to have paper badges, so my first day I got a little blue laminated piece of paper with my login on it. I set up my computer, and settled my fish onto my desk. The next day- my first real day of work- an email went out to all the developers. We were about to launch the toy store, and the photographers (who shot images of all the toys that would be on the site) had an emergency! Turns out, all the legos we had to photograph were in boxes. Photographs of boxes of legos are BORING, and so a plea was sent to the geeks of the company. Come assemble legos, and we'll feed you pizza! So my manager and I showed up, ate pizza, and assembled legos. We got our asses kicked by the 6 year old son of one of the PMs, who could assemble lego kits with lightening speed.

The part that killed me was that the toy store was a big secret, so I couldn't tell any of my friends that I had the greatest job ever, programming and putting together legos! After the store launched a few weeks later, I was able to pull up some toys detail pages & show off my creations.

France, natural selection, and sleeping dogs

This evening, I pulled out an old photo CD of my trip to a friend's wedding in Aix-en-Provence, way back in June 2002. I used to have them online at Artsy Fartsy but Alex took that website down a few months ago, so I'm uploading them to Flickr. While I was playing with the pictures and tagging them, Declan jumped up on the sofa & sat down right next to me. Then he started nodding his head... and leaning over to the side... slowly... until the side of his head bumped my ibook. He jumped up with a "how in the world am I supposed to nap with that thing on your lap?" look. I had to fold up a blanket and place next to me, so that he could nod his head slowly over onto it. He knows that I'm a sucker for how cute he is when he's sleeping, and he knows very well that I'm willing to contort myself into all sorts of crazy positions, let my arms fall asleep, put huge cricks into my neck, in order to let him fall asleep on my lap. I can't help it, he's so precious when he's fast asleep.

At work today, I got in a long discussion with two other P13Ners about how different dog breeds show how natural selection and evolution work. Of course, with dogs, it's not so much natural selection most of the time, but humans doing the trait selection. But it leads to analogous results. One of the other dogs on our floor is a little black daschund, Scooter, who's a cute little guy. The conversation started when Declan walked into an office and did a loop under the desk to look for crumbs. My friend commented that Declan didn't see - or didn't smell - a few crumbs that were down there, but Scooter would soon be by & would vacuum them up. Stephanie wanted to know why Scooter would be better at picking up crumbs than Declan, and I remarked "Oh, he's not a herding dog, so he's got way better eyesight and a better sense of smell." So we talked about how Declan is rather far-sighted but has super hearing, because those skills help you keep track of sheep, while Scooter, who was originally bred to hunt small animals, has a really good sense of smell. The we talked about Sammy's webbed retriever feet and fetch obsession. And Declan's duclaws and the rocky Shetland Islands, then how much humans knowingly selected different traits, while others just emerged. So yet another reason why it's fun having dogs at work, they lead to random discussions of Stephen Jay Gould.

More workers do business on a global schedule

In the Seattle PI yesterday More workers do business on a global schedule - This is actually a really good article about the odd hours that post-geographical (oooh buzzwords!) work entails. Despite years of video conferencing hype, it's really instant messaging, SMS, and email that make the 3 continent world I work in actually function, so the article is right on there. Maybe not surpisingly, the reporter missed the biggest pain of working like this. It's not being paged at 2 AM to help out Tokyo. It's lag- if I fire off an email in the AM to an office in Europe, I most likely won't see any answer for 24 hours. When you have 6 week product development cycles, 24 hours is a big bite of time. Really, I love my job, I love working crazy schedules, but even in college I did weird stuff like wake up at 6 AM one day and noon the next, so it works nicely for me. I wouldn't switch to a group that didn't work with people in our other offices, since it makes things so interesting on a lot of different levels. Not just technical ones, although parsing XML with kanji in it isn't exactly something I thought I'd be doing ever in my life. I got a huge grin on my face the first time one of our JP editors used "-san" in an email to me- it's really all the little things. I hate to think about doing it with kids, though.

Hymn of the Big Wheel

Today was programming crazy search interface day, and I spent a lot of it listening to Massive Attack's Hymn of the Big Wheel. It's a good song for concentrating and the zen state you kind of need to keep track of a hundred little API things in your head.

This morning also had a "you know you're a chick programmer when..." moment. My officemate was already working when I showed up, and noticed my red nail polish right away, so we started discussing favorite polish colors and whatever. Then she noticed a big group of guys with visitor tags walking past the open door, looking in at us, and said "Oh god, and we're standing here talking here talking about manicures." I mimed shouting "We're programmers!" into the hallway. Well, really, they could have heard a discussion of java or something had they walked by some other morning. (My nails, by the way, are a really nice sparkly red.)

I should never have typed this last night: "Tomorrow is even more broomball, so I hope that I don't break an arm playing." Well, it's not broken, but I sprained my thumb and wrist. What that means: I can't really type.
Music for icing your wrist: Loveless and Isn't Anything.

Broomball World - What the HELL is Broomball
I had no idea that there were broomball websites! I played it this evening (not on ice, but on a muddy field), then walked home exhausted, listening to Rjd2. Tomorrow is even more broomball, so I hope that I don't break an arm playing.
Exhausted-but-happy music mix for tonight:
Matthew Everybody Down - I caught a ride in to work this morning with a friend who was playing the Goo Goo Dolls; Matthew does their kind of music SO much better.
David Hudson, Walkabout
Mission of Burma, Progress and Peking Spring - from epitonic.com
Radio 4, Calling All Enthusiasts, How the Stars Got Crossed and Dance to the Underground - also from epitonic.com