Dogfooding: Find a way to be your own customer

This article was originally published as a guest post on Geekwire; it is republished here for the readers of this blog. 

In early ‘90s, while working on Windows NT, Microsoft popularized an idea to make everyone on the team use early builds of their own software.

Back then, it was quite a painful request — imagine developing an operating system on a box prone to crashes, where your basic tools don’t quite work right. This setup undoubtedly causes loss of productivity and frustration.

Are you eating your own dogfood?

Are you eating your own dogfood?

And yet, what it gains is something more valuable: an immediate feedback loop, where bugs are found quickly, where there is healthy peer pressure to urgently fix issues that are preventing your colleagues from doing their work. Since the cost of a bug goes up the longer it lives in the codebase, this feedback loop — dubbed “dogfooding” — is a significant net gain.

Today, it’s even easier for most technology companies to dogfood, because most of us aren’t developing operating systems. If your internal build doesn’t work quite right, you can still do basic things – so the downside of dogfooding is minimal. Let’s explore some simple, natural examples:

  • Facebook rolls out most features to employees first, and only then to a subset of external customers. Employees, of course, already use Facebook every day and can provide instant feedback.
  • Everyone loves LOLCats. Employees of the Cheezburger Network are natural customers, as they consume their own content every day.

dogfood1At my company, Wetpaint, we build tools for publishers to develop relationships with their readers through social media. We take dogfooding so far that we’ve built an entire media business — a very successful one — to be our own customers, and to test-drive our platform before our clients use it. This has allowed us to evolve our tools at record speeds.

But what if you’re working on a product that isn’t so easy to dogfood? You have to be creative and find a way to incentivize your employees to be users in order to get the information you need. One way to set this up is recurring competitions; here’s what I would do with these businesses, for example:

  • Redfin, Zillow, Trulia (real estate): Employees must find the best real estate deal in their area. Give them virtual currency. Have them use your products — and competing products — to make their virtual investment decisions.
  • SEOMoz (search engine optimization): Each team member sets up a blog and must use the company’s tools to make it rank the highest a couple weeks later.
  • Tableau (data visualization): A leader selects a dataset, and everyone is encouraged to find gems in that dataset; the fastest, most interesting insight wins.

Give real prizes to the winners. Set up a weekly beer gathering, get the winners to share their strategy. I bet some product ideas will come out of that. Make sure these events are regular, not one-off, to encourage employees to keep thinking competitively and creatively.

Dogfooding programs complement agile and lean development practices very well, because iterating with the rest of your team for a few days before releasing an experiment increases your chance of success. You’ll also get the obvious feedback out of the way early.

Think of it as a modern version of Joel Spolsky’s hallway usability tests: instead of having to interrupt a colleague to review your UI, you’ll overhear “Oooh” and “Ahhh” when your code goes up on dogfood. That’s the perfect time to ask – “Hey, what do you think about this new thing?”

Hackathons at Startups: Creative ‘Fresh Air’

This article was originally published as a guest post on Geekwire; it is republished here for the readers of this blog. 

Every now and again, we hear about hackathons: Startup Weekend, Facebook’s famed all-nightersHack Week at Dropbox.

However, in a startup, it’s so difficult to imagine how organizing a hackathon can be anything but harmful: “What do you mean, take a couple work days and drop what we’re doing? We’re in a race with competitors! There are holes in our product – and they’re blocking adoption! We can’t just waste a couple days!”

A scene from the recent Sports Hack Day

A scene from the recent Sports Hack Day

I know you’re under pressure. That’s the nature of startups.

And yet, allow me to ask you: how many times have you fought an issue for weeks, only to find an elegant solution that takes a day to implement? Have you ever built a feature that nobody ended up using? Have your employees begun talking about the “grind,” the soul-crushing 80-hour-week pace where bugs never end?

If so, hackathons can help with each of these. They let folks take a step back, concentrate on the big picture, and apply their passion – usually where it hurts the most.

Allow me to share a story from Wetpaint. In summer of 2011, we were struggling with our data warehouse system; we’ve had all the symptoms from the list above. This system was so unreliable that our internal customers didn’t trust the data. Engineers were burnt out from every-Saturday-is-a-workday routine. After a couple months of treading water, we realized that we needed to change something structural in our approach.

One key change we introduced was a framework for open-ended innovation. The intent was simple: all participants drop their day-to-day tasks for a couple days, and work on whatever is exciting for them, as long as it has something to do with our overall business. No top-down mandates. Work alone or with others. The only requirement is to show – demo, not PowerPoint! – your results to everyone else at the end.

We called this framework “hack days.”

I was amazed by the results. Initially planned as a morale-boosting exercise, there were somany great ideas that came out of it. One engineer’s 2-day project disposed of the majority of the issues we were having with the data warehouse. It took a completely different approach to the problem, challenging some of the foundational assumptions that no one ever doubted. Another engineer built a mind-blowing prototype of a tool that we ended up building over the next three months, and that tool had a significant impact on our bottom line.

Most importantly, this breath of creative, fresh air gave a sustained boost to everyone’s output, even as we re-entered regularly scheduled sprints. We made these hackathons a recurring activity – every quarter, coinciding with company-wide business reviews. Everyone in the company is invited to the debriefs now – and they walk out energized and motivated by the ingenuity of their peers.

Moreover, folks outside of engineering are adopting this framework, too. Our social marketing team, for example, drops their best-practices playbook for a couple days once a quarter, and encourages each team member to try their craziest, riskiest ideas. We’ve seen great results from it.

Give it a shot in your startup. Hackathons can become a “startup factory” in an established company, too. If you’d like help setting up a hackathon, send me a tweet, I’d be happy to help.

Measuring interruptions: How to keep your team in the ‘zone’

This article was originally posted as a guest post on Geekwire; it is republished here for the readers of this blog. 

When you look at productive output from a software development team, there’s one factor that almost always predicts problems. You can have top talent; an outstanding idea; great agile process. And yet, if you don’t treat interruptions as a significant source of danger, the progress will be slow and painful.

5368292088_37f5fce714_n[1]We all know and love the feeling of “flow:” You’re in the zone, coding away or deep in thought on a financial model. There’s plenty of research that suggests that we do our best work in this state. We are also happiest at work when we are in the zone often.

But far too often, this state is broken up by a tap on your shoulder or a phone call. There’s a small, innocent question – and it takes you five minutes to answer it. But when you come back to your original task, the inspiration is gone. The “zone” is gone. You need a half hour to just bring back all the variables back into your mind. Or worse, you may not catch the sense of “flow” again that day at all.

That’s the scary thing about interruptions – they typically only take a few minutes to handle, but then bring a trail of a scattered state of mind.  What’s worse, it’s easy to embrace this interruption as a good thing – hey, I just solved a problem, unblocked a customer, made someone’s day better. And yet, for an organization, more often than not, this interruption was a net negative.

Technology startups have a key property: if they stagnate – stay with the status quo, spend too much time on sustained engineering – they die. Time is of the essence; competition is fierce, someone else is going to win that customer, build a better product, hire stronger talent. So only the time that you invest into important AND non-urgent things is bringing you closer to your vision. Troubleshooting is a necessary evil. You must make your current customers happy. And yet, if you spend all of your time on it, you’ll never make it.

So I encourage you to take a systematic approach: actually measure interruptions and create goals to reduce them to a reasonable level.

Create a mailbox connected to your issue tracking system. Every time someone has a question or request outside of the current sprint’s priorities, have them send an email to that mailbox – or do it for them. There are, in fact, two mailboxes: one for emergencies (ex. site’s down) and one for regular issues (ex. data in the analytics DB doesn’t add up). At the end of each week, count the chickens.

Categorize each interruption by component. Draw a graph over time. Discuss it with your senior engineers.

You’ll be amazed. I sure was when we did this at Wetpaint. When we started this practice, we noticed that we had an average of 60 interruptions a week for a product development team of 15.

This, coincidentally, was a time when we were seriously struggling as an organization – we had a hard time moving the product forward. We dug into the root causes and noticed that most of these interruptions were triggered by two systems. We invested time in two sprints to systematically address these issues, and the interruption count dropped to 10-15 a week. Not surprisingly, our productive output shot up.

Morale also improved significantly. Folks felt like they are working on features that move the product forward, instead of constant firefighting.

An important factor in our setup: we established a rotation program to triage interruptions –and only placed managers on this pager duty rotation. Some interruptions are 3am emergencies that must be dealt with immediately; when managers have to be the first line of defense, root causes tend to get a systemic resolution surprisingly quickly.

Another positive side effect of being systematic with interruptions is that nothing falls through the cracks anymore. An internal customer would find an issue and send one of the devs an email, or just chat with them in the kitchen about it. Sometimes, requests like this would be lost – or delayed far enough to get the customer concerned. After we set up this rigorous interruption tracking, every client knew where to look up the status of their issue.

The movie Social Network has a magical moment. A loud house party is going on. And yet there’s a guy in the middle of the room with headphones on, coding away. People walk around him, careful not to disrupt – nobody dare interrupt his flow!!

Do you know how many times a week your product development organization is interrupted today? Can reducing that number take your crew to the next level of productivity?

Facebook: The personalization engine for all of the Web

This article was originally posted as a guest post on Geekwire; it is republished here for the readers of this blog. 

Facebook and its third-party applications today know a hell of a lot about each us: what content we read (Washington Post Social Reader); what music we listen to (Spotify); what movies we watch (Netflix).

Facebook opened up a green field for the game creators, too: games with friends are just so much more engaging. However, while Zynga and Electronic Arts fight for the attention of the social gamer, the only party that is guaranteed to win is Facebook – they gobble up data from ALL of the apps to compile a multi-dimensional data set that will ultimately allow them to build the best personalization and ad relevance engine on the web.

This strategy of Data Dominance – knowing more about each user than any competitor – is executed through a set of API’s that Facebook calls Custom Open Graph. Each micro-interaction in the vertical universe of a game, a social reader, or a music app is a way for the app to drive traffic; it is also a way for Facebook to learn more about each user.  It’s a powerful data mining operation that aligns the interests of all parties involved: the consumer, the app creator, and Facebook.

But how will Facebook use this data advantage? No matter how addictive Facebook is, consumers still spend six out of every seven minutes on other sites. To extend its data dominance to the rest of the web, Facebook should offer analyzed data up as a service to other sites – driving value for third parties and revenue for itself.

In fact, Facebook has already started down this path with its ad products. While Facebook started out by targeting ads on facebook.com using only their own internal data, they recently made the smart move to integrate insights from other publishers’ sites through Facebook Exchange. This was the first step toward an external ad network, and a direct challenge to the eternal enemy, Google, on their AdSense home ground.

With personalization, however, Facebook has been playing it close to the chest: the famous EdgeRank– the ranking algorithm behind the newsfeed that judges what content from your friends will be interesting to you – is so far available to Facebook only. No third party can leverage its great insights. Facebook made a weak attempt to unlock some of its power with the Recommendation Bar plugin; it was a move in the right direction, but the execution sunk it. Instead, Facebook should offer personalization as a cloud data service – and they should charge for it.

Imagine if The New York Times could tell if you’re going to like a particular story – and recommend you a different one if you wouldn’t. Imagine if an online store could know – based on your Facebook profile – which product you are more likely to buy. Both of these businesses would flourish. And Facebook could take a cut of the incremental revenue.

Let’s examine Outbrain, a premium syndication provider. Fundamentally, they are in the relevance business – given a piece of content, they suggest several other pieces of content a reader might like. Facebook could solve this exact problem a lot better – they have more data to base the recommendation on.

Facebook is in a position to unlock incredible new revenue for a whole slew of merchants, if only they allowed partners to tap into the personalization engine directly.

I work at Wetpaint, where we’re building a quantitative platform for audience development. Today, we use Facebook as an efficient content delivery channel to build loyal audiences. We’ve developed ways to run statistical experiments to learn about the audience’s interests, and this analytical approach has driven extraordinary results for us.

And yet we are only scratching the surface of the multiplier effect that is available through the world’s greatest optimization laboratory.  Facebook today is mostly a black box; but if Facebook were to open up its personalization engine, publishers and brands would be able to create far deeper and more engaging relationships with readers and customers.

We’re on the cusp of a personalization revolution in publishing.  Facebook, with its strong advantage in Data Dominance and its equally strong incentive to make the online experience more personalized (for both users’ and advertisers’ sakes), is uniquely positioned to take us there.

Offer a Conversion Path!

I’ve had a curious experience with Amazon EC2 recently, and it made me think of a the process of adoption of new systems and services. In short: it doesn’t matter how good your product is, if it’s too hard to switch from the old way of doing things to your new-and-revolutionary gizmo. Let me share two stories.

Microsoft Word vs WordPerfect, 1991. Out of the gate, WordPerfect is a giant with almost 50% market share. Microsoft just released a “better  mouse trap”; they also know that the competitor’s product has massive adoption, and Word will hit a big wall because of the incompatibility of the two document formats. If a customer buys Word, they can’t open their old WordPerfect documents:

“No matter how good Word is, I have to buy WordPerfect anyway to have access to my old stuff. Damn, I already spent money on one word processor.. Why do I need another?..”

Microsoft does the smart thing: they write adapters for WordPerfect import AND export. Now, if you are an early adopter of the new-and-amazing MS Word, you can still send documents to your dinosaur friends. The barrier has been lifted, the purchasing decision is now to be made only on merit; with this single move, they were able to wipe away most of the network effect advantage of their competitor.

Fast forward to 2012. VMWare and on-premise virtualization providers are under attack by platform-as-a-service vendors, first and foremost, Amazon EC2. Amazon has built a cost-effective, scalable, very advanced mouse trap. It’s not a perfect replacement – but it’s better in many ways. Lots of Amazon’s potential customers today are using various on-premise virtualization solutions. And yet, 6 years after the launch of EC2, there is still no way to take my VMWare Linux box and upload it – seamlessly – into EC2.

No wonder that only under 5% of top 5000 websites by traffic are using Amazon EC2 today.

Influence of Your Work on the World

As I was finishing school, I had a dream – I wanted my job to maximize my influence. I wanted the product of my craft to touch, in a meaningful way, as many people as possible, helping them in small and large ways. It’s mostly pride and desire to maximize the control over your environment: isn’t it awesome when everyone around you knows what you’ve been working on?

“You work on the search engine at Google? Wow, I use that every day!”

“You’re at Boeing developing the new 787? So many people are going to fly on that!”

A beautiful quote from a Microsoft engineer on this subject:

Very few projects at Microsoft have “small” impact. Everywhere you turn, the projects people are working on are likely to be used by thousands or millions of people. You have the opportunity to earn, save, or cost the company millions of dollars through your work. It’s an awesome responsibility, but an awesome chance to create widely influential software.

I’ve found a hole in this logic: it’s missing a key variable. It’s not just about the number of lives you touch. It’s also about the number of people that are working on this same product. The logic is simple: if there are thousands of people working side by side with you, your individual contribution will be lower. You will own and contribute to a smaller part of the puzzle.

Moreover, for technology projects, I’d argue the number of engineers working on the product has an even stronger, quadratic effect on each person’s influence. Ancient, yet so contemporary book Mythical Man Month makes this point well.

Here’s my attempt at quantitatively measuring your work influence number as an engineer in a high-tech company:

Let’s look at some examples:

  • Microsoft Office is one of the world’s most used products; yet there are quite a few engineers touching it. Spread-out, shared ecosystem of Office Shared services that own cross-product components and installation, as well as groups that own localization and documentation, makes the denominator in the equation above quite high.
  • Facebook is famous for having a restriction on the number of engineers that can work on a given product; I can’t find references to the exact number, but anemically, it’s under 10. So let’s say 10. Let’s look at the example of the Timeline: with Facebook’s 900M users, influence of an engineer working on that team is 900M/100 = 900K.
  • At Wetpaint, there are 3 engineers working on the wetpaint.com website. Last month, we had 12 million readers. Influence of each engineer = 12M / 9 = 1.3M.

If you’re looking for your job to have influence – right out of the gate – work in a small, isolated team that has full control over its destiny. Startups are by definition structured this way.

Bet on Yourself

How comfortable are you with the idea of relinquishing control over something important to you to someone who doesn’t really wish you well? How about relinquishing control over your career? Over your family’s livelihood?

Whenever alignment of interests isn’t present at the workplace, that’s exactly what employees are doing – they’re relinquishing control to the manager who has completely different incentives. If you’re working in a big company, your career success – your next years’ bonus, your promotion – are in the hands of someone who has no rational reason to wish you well.

Don’t give me the empathy argument. Don’t tell me that your manager wants to help you because it will make him/her feel good inside.

Incentives rule the world. People optimize for the way they are rewarded. The manager will optimize for what will make them get their next promotion. When it costs them nothing, they might even invest in you – if they’re what we consider a “good manager.” But do not kid yourself – there’s no incentive for them to systematically make you better off. You’re just a cog in the machine. A stepping stone.

… How comfortable are you with the idea of gambling? Of betting something valuable to you on a phenomenon with relatively unpredictable outcome? 

I’m not much of a gambler; the idea of having zero control over the spinning wheel of the roulette gives me a heartache. Someone else throwing the ball… Someone else programmed the roulette… Just standing there with my fingers crossed makes me feel powerless. A victim of the odds. A fool that disrespects the probability theory.

But what if you could change the game? What if you could be the one throwing the ball, programming the roulette, training for hours on how to beat the odds? What if after months of training, you were presented with a chance to make a bet – on yourself, on your own skill – and be the one playing your own game, with so much under your control?

That’s what startups are about.

You’re gambling – yes. You’re playing against the odds – most startups fail. But you’re betting on yourself – your own skill, your training, your intellectual horsepower. You’re creating a new world – every day. Your hunger inspires you to do more than humanly possible. You’re riding a roller-coaster – but it’s your own path.

Moreover, people around you – your manager, your peers – are all in exactly the same boat. If you don’t pull your weight, you will hear about it; others will jump in to make you better. If you’re doing well, your leaders have a purely logical incentive to get you the recognition you deserve – because it will make THEM succeed. Because it improves the odds of the startup succeeding. Because you personally are contributing a big, noticeable chunk to the bottom line of the value of the startup.

The most important factor here, of course, is the ability of every employee to make a meaningful difference on the company’s bottom line and having a share of the gains. This is what changes the incentives for everyone.

The wise Glenn Kelman says that “startups are the most intense way to live.” I couldn’t agree more. The thrill of a bet on myself – combined with the responsibility that it brings – is making me feel alive more than ever.