Your team just screwed up badly — here’s what you do next

disasterYou’ve been there. Someone on your team just screwed it up. Your production website went down in the middle of the night, it took hours to bring it back up. It’s 10am the next day, you’re at your daily standup, and the culprit is looking down, ashamed and quiet; the team is noticeably uncomfortable and is expecting you, their leader, to scream and shout about business impact and accountability and how bad this all is.

You’re upset. The outage already cost your group some reputation — you’re seeing tweets and a message from the investor, and you have no idea how something this dumb could have been overlooked.

You can allow your emotions to take over. You can do the screaming, you can shame the perpetrator, who will undoubtedly remember this occasion and probably won’t make a mistake of this kind again. You will scare others at the standup enough for them to be afraid of their own shadow for the next week.

Or you can take a breath.

And ask yourself.

What impact do I want to make on this group right now? How do I want this group to be different 15 minutes from now?

There’s only thing that matters when the sky has already fallen: how you prevent the issue from happening again. How can you make sure that your group, your system, your infrastructure is stronger as a result of this incident, and the same screwup cannot happen for *structural reasons*? That a repeat is almost physically impossible?

So utter the following words to the team: “How can we make sure an issue like this can never hit us in the future?” Here are some reasonable answers for typical situations:

  • Website went down and no one noticed? Add Pingdom monitoring and have the text messaging alerts go to the entire team.
  • Someone checked in a ridiculous bug that broke half the service? Add automation that tests a big chunk of your service end-to-end.
  • Your email newsletter was sent five times to each recipient? Monitor the number of emails sent to every recipient in the database. If anyone received more than one email in the last 24 hours, automatically turn off the master switch.

You’ll notice that each of these remedies is heavy-handed — a crude overkill — and intentionally so. This is to accomplish the second most important rule of “the day after sky has fallen”: Your structural, automated precaution from a repeat must be implemented the same day. The culprit drops what they are doing — heck, they were kicked out of their context by the outage already — and works tirelessly to implement it. Over time, the crude solution can be evolved to be a little bit more elegant.

Final rule for the day: No screaming, no shaming, no humiliating public speeches about accountability. Provided that you don’t have a talent issue, the culprit feels terrible already. By focusing on the solution — and making the team stronger — you not only inoculate the team; you motivate them by staying in the trenches, instead of speaking down. This cements the culture of continuous improvement — and collaboration, instead of fear.

(this article was originally published on VentureBeat)

Machine Learning and Digital Marketing: Melding Human and Machine

In digital, you can easily spot two opposing camps — the artists and the quants.

Artists are folks like the New York Times: Pulitzer prize-winning journalists use their intuition and skill — their unique talents — to create one-of-a-kind stories, and the judgment of the Chief Editor is pure gold. Artists create incredible brand value; true loyalty — lifelong fans.

human-man-and-machineshutterstock_250853188-620x615Quants are folks like They use Wall Street-style algorithms to identify long-tail Google queries that have weak competition, and pay amateur writers $5 to create short posts that address those queries. Queries like “how to remove gum from clothing.” Their quant models tell them that stories like this will make $7 on ads in the next year, so they pump out millions of such stories.

Both approaches have problems. If a New York Times writer gets hit by a bus, there’s no replacing them. Their talent dependency is not scalable. eHow stories — millions of them — inspire no loyalty, create no brand value. Let’s face it, it’s crappy content. No wonder Google did everything in its power to kill it.

You might might ask then, how do you get the best of both worlds? There’s clearly a spectrum here; is there such a thing as inspiration at scale, data-driven AND human powered content creation? This question is relevant for more than content publishing — the same applies to

Leaders in the field realize the dangers of dogma of both extremes, and try to find the middle ground. Let me share a few examples of folks that do this right.

  • Hubert Burda Media, a European media conglomerate with their flagship German news brand, curates stories and follows the news cycle just like New York Times. Unlike the New York Times, though, they are extremely quantitative about what they publish on their social channels and they use sophisticated machine learning tools to make decisions on timing, volume, and merchandising of that curated content.
  • Uber is a quant-first company: their complex mathematical models predict demand for cabs and direct their drivers to hot regions before demand occurs. And yet, their marketing has plenty of “soul” — just look at the Uber Kittens campaign and try telling me that this isn’t the most creative idea ever.
  • LinkedIn is a quant-first company: their relevance algorithms determine the best stories to show out of myriads of updates that your connections share. However, with the widely successful Influencer program, they are bringing top writers (“artists”) and exclusive content into the mix, thus allowing their machine learning algorithms to pick from the best raw materials.
  • eBay, with its strong merchant roots, curates deals and hand-selects the most appealing merchandising for each deal. But the composition of the deals that should be offered to each customer? That’s a job for a machine learning model.

These folks understand that winning is about art AND science. Human AND machine. Human does the creative work, machine takes care of personalization at scale.

P.S. This article was originally published on GeekWire; I also gave a talk at New Tech Seattle based on this article:

Paid Apps Model Revisited

don27tmakeyour0acustomers0ahesitate0a0a600a28fenshui29-defaultHow many times have you hesitated before buying a 99-cent app?

You’re staring at the reviews. Looking for a Lite version to try out first. Catching yourself at the thought: “I’ve spent more time thinking about buying this thing than the 99 cents that it’s worth.”

What if I don’t like that game?.. What if it doesn’t work on my phone? I sure don’t want to call customer service to ask for a one dollar refund, that would make me feel even more dumb..

There’s a grand canyon of friction between free and 99 cents. An ocean between free and a $4.99 game that everyone is talking about. All of these are symptoms of friction. Friction that App Store tzars have created and are oblivious to.

What if, instead of asking you to think, causing you the angst of making a decision, Apple and Google learned from Xerox?

In the early 1960s, Xerox was a pioneer in the copier space. They had a significant issue, however: a large, expensive machine wasn’t an easy sale, and decision-makers would hesitate before making a purchase. Do we really need one? Will the office workers use this? Returning it would be such a hassle…

So Xerox leadership invented a stratagem. Their salesperson would bring the fancy copier to the office for free and just leave it there. Play with it, they’d say. It’s free, no obligations whatsoever. I’ll pick it up in a month.

Guess what – nobody wanted to give it up in a month. It was a great product and sales went up tremendously as a result of this move. We now know this move as a “free trial” – lots of SaaS and shrink-wrap software is sold this way.

Why aren’t app stores facilitating this type of transaction, though? Apple, Google – how about giving each paid app out for free, and only charging the consumer if that app is still installed on their phone a day later?

Of course, there are one-and-done apps that offer a single-use value proposition: museum tours, cheat codes, contact sync apps. These and similar apps should be able to opt out of this scenario. But for the vast majority of apps – apps that aim to deliver value over multiple sessions – this would be a huge net gain.

Just imagine: no need to develop and maintain a separate Lite version. No buyers’ remorse. No fear or hesitation when buying an app.

Whether you’re an app developer or an avid app user, let me know what you think about this concept in the comments.

Are Your Company Values Just Empty Words?

values45463675I had a chance to interact with two companies recently: Homegrown and City of Bellevue Utilities. These two companies helped me crystallize the difference between “value statement on the wall” and “values that are coming through to customers.” Homegrown Logo

Homegrown’s tagline is “sustainable sandwich shop.” Their About Us page has the word “organic” mentioned 22 times. It says that “stores are designed to be as low-impact as possible… [using] reclaimed, recycled … building materials.” Their napkin dispenser asks you to think about the environment and only take as many napkins as you will use. They have metallic cutlery at every table. Clearly, owners at Homegrown are trying to project an identity that stands for sustainability and environmental awareness.

Yet, when you order a salad “for here,” you get a single-use plastic container with a salad inside.

I was stunned when I saw this – I even double-checked with an employee that they got my order right. What would cause such a mismatch of identity and execution?.. Getting this wrong is akin to Microsoft accounting using an abacus to do budgeting – contrary to the core reason why the company exists.

City of Bellevue incorporated in 1953. It has a legalized monopoly over certain utilities delivered to its residents – water, for example. Based on the lack of competition, one would expect their Utilities department to be worse than Comcast – long wait times, difficult-to-deal-with phone navigation trees, rude and incompetent customer service reps.

Au contraire, my friends, the exact opposite. One button press to talk to a human. Zero wait time. Extremely polite and knowledgeable account rep that not only answered my dumb questions but anticipated what my needs will be several months down the road and told me how to prepare.

How do you explain this? City of Bellevue staffers don’t get paid well. We typically think of government employees as the antithesis of entrepreneurial ambition: craving unionized benefits, job security, and guaranteed pensions. And yet… City of Bellevue folks knocked it out of the park.

How do you explain such a stark difference between the stereotypes and reality in these two cases?

Here is one possible explanation: actual buy-in into company values. At Homegrown, the person responsible for in-restaurant dining experience could not care less about company values – they were just getting paid. They weren’t doing their job out of the sense of connection to the company’s purpose.

Somehow – Bellevue Utilities must have a remarkable leader – their employees actually believe that their purpose is to

.. support public health and safety, quality neighborhoods and a healthy and sustainable environment and economy by effectively managing: drinking water, waste water, storm and surface water, and solid waste.

Heck, anyone that can galvanize employees to be excited about a mission statement that includes the words “solid waste” deserves a medal.

I believe this has to do with intrinsic motivation – the concepts of autonomy, mastery, and purpose that Dan Pink talks about in his remarkable TED talk (below) and book. The sense of purpose – a genuine conviction that what I’m working on is a part of a bigger mission that I believe in – is strong in the case of Bellevue Utilities and missing in the case of Homegrown.

Let me ask you: have you recently seen someone take actions at your company that are directly in line with your company’s values? Share your story in the comments below.

Full Price is for the Lazy, or Stop Financing Their Marketing

hard-work-pays-offIn life as in business, if you are willing to invest effort into something, you will do better – a lot better – than average. Today’s story is about a real-estate purchase – and how doing your homework makes a 10x price difference for services.

Did you notice that just paid your agent $1000/hour?

Basic dynamics of a real-estate transaction: when you buy a $500k home, 3% of the purchase price goes to the sellers’ agent; 3%, or $15k, goes to your (buyer’s) agent. What exactly are you paying for?

  • Touring homes
  • Negotiation advice
  • Access to boilerplate real estate transaction forms
  • Administrative services (shepherding the paperwork)

I was unable to find hard data about the average number of homes that buyers look at these days  and the average number of offers that people make before making a purchase, but anecdotal evidence (one, twothree) suggests that: consumers tour 5-10 homes before making a purchase and make 1-2 offers before reaching mutual agreement.

If you convert that to the number of hours that your real estate agent spends with you, you are looking at about 10-20 hours of work. Dividing their fee by the number of hours, you are paying your agent $750 to $1500 an hour. That’s more than what you’re paying your lawyer – heck, that’s more than what your brain surgeon charges.

There are better real-estate firms out there: Redfin, for example – I love Glenn Kelman and I’m thrilled that he’s working to disrupt this industry. For example, Redfin’s field agents that give you home tours have no incentives to sell you those homes. But what Redfin charges is still far too much: on a 500k home, they charge you (3% – $5000 rebate), which still turns out to be about $500/hour. You might argue that Redfin value prop is a lot higher: they enable the consumer to make better decisions through data on their very awesome site. Unfortunately, this is all but commoditized today (every real estate site does it) – and a savvy consumer has no incentive to subsidize software development efforts behind the self-service site.

Finally, if you look hard enough, you can find flat-fee firms: real estate firms that are specifically designed for savvy buyers – those that simply need help with the paperwork. I just worked with Shop Prop, a flat-fee agent in Seattle area. Shop Prop took several thousand dollars for the transaction and refunded the rest of the buyer’s commission back to me. I paid an order of magnitude less per hour than I would have paid an “old-school” agent. Why? Because I was willing to invest time into finding the right deal.

The craziest part: most consumers are irrational and the rebate is not the reason people go with Redfin. Glenn, the Redfin CEO, writes:

.. commission savings that Redfin offers home sellers and home buyers dramatically lowers our profits margins and there is no evidence that it drives more revenue… [We have] given away $100 million [in rebates]”

What’s behind this?

  • Lots of Emotions: the buyer is making a life-changing decision for his/her family. They want to know they’re getting a good deal, which is difficult to judge since they haven’t done many deals like this before. They want to go with a professional – social proof here (“my neighbor did business with this great agent”) is very important.
  • Apples and Oranges Marketplace: even though all houses are out there for consumers to review and evaluate, it’s hard to develop a quantitative sense for “this place is 200 sq ft less but has a remodeled bathroom… does it mean it should be similarly priced?”
  • Competition: real estate market is in the state of frenzy in Seattle area, with lots of international investors making cash-based offers; many homes get 5+ competing offers. Buyers think they need the help of a professional to win in such a market.

Problems with so-called “professionals” that are paid ~3% of the transaction, beyond their exorbitant fees:

  • Incentive Misalignment. There is no incentive for your agent to help you get a good deal on a house. He/she just needs you to buy any house as soon as possible, so they can get their commission and move on to the next client.
  • No Meaningful Expertise. Real estate agents require no meaningful training to practice their profession; the only actual requirement is a vast network to generate demand or supply. Buyers and sellers take 100% of the risk; moreover, as a buyer, if you end up closing on a place, you are so far in the weeds that your “commitment bias” (sunk cost) makes you so convinced that you got a good deal that you will biased to resist any evidence of the contrary – thus convincing you that you had a good real estate agent.

438110523b4cba59613c4dba6d34a734That is, most consumers would rather take $10-20k of their hard-earned money and gift it to a stranger for no meaningful reason – instead of spending it on vacation, their loved ones, or charity. A rational consumer – that’s not afraid to do research, that understands built-in irrational biases inside their mind – will fair an order of magnitude better.

Do you have an example from your life or work where careful research and effort have paid off handsomely? 

Why Are We Working on This?

why-shutterstock_126628475-600x709You’re a recent grad from a top engineering school. You come to a hot startup, and in your second week, you volunteer to implement an ambitious new feature. You slave away at it for a week, burning the midnight oil, trying to impress your new colleagues. You’re brilliant: you find an ingenious algo that solves the problem elegantly and with a lot less code than anyone thought was possible. You proudly check the code in, it ships to the site.

You’re proud of your accomplishment. You move on to the next big thing.

Spoiler alert: you screwed up.

You thought that you were done when the code was checked in and shipped.

Not even close… It turned out that the feature you so proudly shipped actually didn’t make a difference for the business; it didn’t improve customer experience in a meaningful way, didn’t move the bottom line for the business at all.. Your hard work didn’t bring value to your company. All of that effort to make a beautiful algo was a waste. That’s because you cared about the code, the “how to solve the problem,” not the “why” behind it.

Peter Drucker famously said that “There is nothing quite so useless, as doing with great efficiency, something that should not be done at all.”

Next time a feature is assigned, consider asking: “Why are we doing this? What value do we expect this to bring to our business?”

It might sound like a challenge to your boss or to your product manager — it actually is not; their primary job is to make sure that every engineer is working on something that moves the business forward. They’ll happily explain the intent.

When you find out the “why” behind a task, it is much easier to make sure the implementation gets to the heart of it — without any extraneous effort. Maybe the task you’re assigned is just an experiment — with a super high business risk, and it doesn’t matter whether it’s scalable at this phase. Or maybe — just maybe — you will have a better idea on how to solve this specific “why” than what’s in the work item.

If you want your work to have an impact on the company you work for, you have to be an entrepreneur, not just a coder. Challenge the assumptions when you’re given a task; find out what metrics are expected to move as a result of this task being done; instrument your code so that you can actually know.

Don’t start the work until you have an objective way to judge whether it’ll have a positive impact on the business. Don’t stop until you see that your work made the business forward. Look at the data after your feature ships, share your findings with your team, suggest iterations.

Let’s explore some examples.

 —A task is to change the layout of the signup page.
Why? To improve our conversion rate and make more money.
What’s the actual objective measure? Conversion rate over time.
How do we know whether it’s successful? We need to have an objective measure of conversion rate before and after the layout change.

—A task is to fix a bug in the retry logic for Facebook Insights data collection.
Why? So our customers should always have stats about their Facebook activity.
And why does that matter? So that our customers could judge the effectiveness of their campaigns and use that knowledge to make more successful campaigns.
How do we know whether this work made our business better? We’ll see improved stats of Facebook campaigns that our customers initiate.

—A task is to fix a rare crash of our iPhone app on old versions of iOS.
Why? Crashes create really unhappy customers.
No really, why? Because a handful of folks that use an ancient iPhone get really angry and leave 1-star Apple Store reviews.
What’s the measure of success for the business? Number of 1-star reviews of our app that mention crashes and old iOS is down.
What’s a good proxy of this metric? Actual number of crashes — you need to be capturing crash stats for your app and monitoring it over time.

Eric Ries created a fun term: “achieving failure” – perfectly executing a flawed plan. He talks about this phenomenon happening on a large scale (entire startup), but it happens all the time on a small scale, too (individual work items).

By asking “why are we doing this,” you help insulate your group against it, and also remind the group of its shared purpose, bringing the team together.

This article was originally published as a guest post on Geekwire; it is republished here for the readers of this blog.