The Great Grocery Game

It was about 9 o'clock in the morning, early Saturday, with the sun shining and my refrigerator totally empty.

I had to buy groceries. Luckily I live in San Francisco, a city filled with startups who can't wait to outsource the minutiae of everyday life. At the tap of a button a complete stranger will come over to my house and live life on my behalf. It's both disturbing and beautiful.

With so many choices it can be hard to know which one to pick. So instead of take a gamble, I decided to put them to the test. Three delivery services, head-to-head, in a race to purchase and deliver an identical order of groceries on a Saturday afternoon. Game on.

The Experiment

I started by compiling my grocery list. I took my last 12 trips to Safeway and built a comprehensive database of every item I had purchased. From that I crafted a standard basket - the order that I felt was representative of a normal trip. After a few adjustments to make sure I didn't get too much food, I ended up with this.

• Breakfast: Milk, Cereal, Yogurt, Donuts, Bacon
• Produce: Strawberries, Avocado
• Snacks: Crackers, Soda, Guacamole Mix
• Dinner: Chicken Breast, Frozen Dinner, Wine

While this list may seem simple, it includes some hidden landmines:

• Milk - I left a note to check expiration dates and pick up the freshest carton. I always pull my milk from the back to ensure the longest shelf life in my fridge
• Guacamole mix - I asked for McCormicks, a specialty brand that's out of the way and hard to find.
• Avocado - I left a note to purchase a ripe avocado. You'd be surprised how many people don't know how to tell if an avocado is ripe.
• Strawberries - These can be very hit or miss. If you pick the package on top you could end up with a bunch of nasty fruit. A rational shopper will check for the best.

With my cart assembled it was time to order.

Ordering Process

When ordering on Postmates or Instacart, you start by choosing the store. You then go through a loop of picking a category, scrolling through the list to find your item, and then adding it to your cart.

The merchandising in these apps is terrible. Sometimes the item you want is right at the top of the list. Other times you have to scroll through 15 screens of junk to find what you want.

Search in these apps can be painful. Typing in a brand will bring up every single item under that brand name across the store. When you get in to store brands there's no way you'll find what you need.

The biggest value to Instacart over Postmates is the ability to schedule ahead and plan out substitutes. Instacart let's you take any order and pick the day it will show up where Postmates restricts you to just “Get It Now”. You can also pick and choose what alternative items you want on Instacart in the event your favorites are out of stock.

Ordering on Exec was much simpler in comparison. Getting an Exec Errand is just a simple text box to fill out. My typical grocery store process involves my wife texting me a list and me going to the store so it was easy to just copy & paste the list in. Having simple text is great for fuzzy search. For example, I asked Exec for the cheapest bacon on sale and I ended up with a much higher quality product at the same price as the others. The big issue is specificity. For certain items we struggled with how to word our request such that it was perfectly clear what we wanted.

I took my standard cart and entered it into each app without triggering delivery, double checking each to make sure they were at parity.

Cost

Each service has a very different cost structure. No one service is the best at any pricepoint.

Postmates has a dynamic pricing model based on time, demand, distance, and order size. I paid $12 for delivery on this order, though I've seen the price be as low as $7.50. They also charge a flat 5% fee on top of the purchase price.

Exec has a model that's purely labor based. The more time your Exec spends working, the more you pay. Their rate is $25/hour with a 45 minute minimum. On top of that, you pay 50¢/mile for travel and a 3% fee on any purchases.

Instacart's pricing is the most ambiguous. They charge a flat delivery fee based on order size: $7.99 for under $35 and $3.99 otherwise. Often on the weekends they will offer free delivery for orders over $60. Where it gets tricky is that you pay the retail price for grocery items plus an unspecified fee on top. So if you're shopping at Safeway where discounting is rampant, it's not clear just how much more you're paying. For my order, the Instacart cost for any food item was ~20% over what I would pay at the checkout.

My Costs

Cost By Order Size

This graph helps show how the costs change with order size.

Timeline

When noon on Saturday rolled around, I started the race. I triggered each app simultaneously to delivery my groceries as soon as possible. I kept a record of every call, text, and push notification I received and compiled them together. This timeline tracks all communication I received along the way. Each event is an SMS or Push Notification unless otherwise specified

Results

As the orders came in I went through each item and cataloged them along the following criteria

• Did I get the item I wanted?
• When items were unavailable, did I get an acceptable replacement?
• Did they follow my special instructions?
• Is the item of a quality I would I have purchased myself?

So how did they do? As expected, each service had no problem fulfilling the easy items on my list. Though, for about half of my grocery list, there were problems.

Exec

Exec was the first order to arrive with a total delivery time of 65 minutes. Exec makes no promises on delivery time and if they run over, it costs you directly. Of all three services, they were the only ones able to fill my 14 item order. Overall the order quality was top notch.

• ✔ The strawberries, milk, and guacamole mix came in perfect
• ✔ The donuts I wanted were out. He messaged me through the app and I told him a suitable replacement
• ~ The exec thought the frozen dinner I wanted was out and asked about a replacement. I realized my description of the item was not specific enough, as he bought the completely wrong thing.
• ✘ The avocado was not ripe. I was never asked about a replacement.

Postmates

Postmates was the second order to arrive with a total delivery time of 75 minutes. Postmates promises a 60 minute delivery, making them 15 minutes late. My postmate was the only competitor who was a woman, a detail that weighed heavily in their favor. She did a fantastic job on the produce, blowing away everyone on her handling of my avocado request.

• ✔ The strawberries and milk were great
• ✔ Text messaged me that there were no ripe avocados available in singles, but there were in a 3 pack. I told them to pick any suitable single avocado.
• ~ Text messaged me that the frozen chicken was unavailable, but substituted it with an unnecessarily expensive alternative.
• ✘ The guacamole mix was substituted with a cheaper variety without consulting me. This is the variety right next to the avocados in the store.
• ✘ Postmates did not have bakery items listed and did not have functionality for custom requests. Therefore I did not get any donuts.

Instacart

Instacart was the third order to arrive with a total delivery time of 111 minutes. They promise delivery within a 3 hour window, so this delivery was well within the normal range.

• ✘ The milk was not the freshest available, expiring 4 days earlier than the others
• ✘ The strawberries were of very poor quality as judged by my wife, the Chief Strawberry Eater
• ✘ Instacart reported that the Guacamole mix was sold out and did not supply a substitute
• ✘ The avocado was not ripe
• ~ I got a text message that some items had been substituted. The chicken had been swapped out for a larger size (1.5lb) and higher coset.
• ✘ Instacart does not deliver alcohol so I did not receive wine.
• — Instacart handles custom requests but I failed to place one for the donuts.

Summary

Postmates was hands down the best value per dollar. With their inexpensive delivery fees and great accuracy, I feel like I had a great experience relative to the amount I paid. It's hard to say whether this was due to the startup or because the competitor was a female and thus had more attention to detail.

Exec excelled at speed and accuracy. I was stunned by how fast they put my order together and how spot on the items were. My Exec communicated promptly; I never felt like I was out of the loop. However, the one thing they messed up on they got majorly wrong, partially due to me not being specific enough.

Instacart wins at convenience and price. My wife liked the fact that Instacart “got the job done without getting in your way”. They try to frontload all of the decision making so you won't be bothered when the job is happening. And despite their cloudy pricing model they came out cheapest for small-to-medium shopping trips.

Going forward I'll use Exec for all my same-day trips and in the event I need complicated shopping orders. I'll continue to use Instacart for small, simple orders that I want to plan in advance.

If you'd like to use Postmates, click here to sign up

If you'd like to use Exec, click here to sign up

If you'd like to use Instacart, click here to sign up

Special thanks to my wife for helping me organize and coordinate this experiment and to Kim MacVaugh for playing along.

Focus on The How

When building a company it's important to reflect on the way we use our time. Seth says the best teams do this by focusing on how they operate.

Great teams are set apart by The How, they focus on making every piece of code better, they push to build art faster, reduce iteration time on design and squeeze more out of a single sprint. These people understand that time is the only thing you cannot buy and that growing the team or raising more money is just a way to push back an inevitable deadline

Seth Sivak's The How

The Power of Aspiration

As product creators, we have a lot of tools at our disposal. Tapping into raw human emotion is one of the most powerful of all. One of the emotions best used by games is aspiration.

Aspiration is an emotion that can produce incredible results. James Bond is a character that's many people aspire to be. In the recent 007 flick Skyfall, there's a scene that features our british masculine ideal getting a close shave by a beautiful woman. This one scene has increased sales of straight razors by 4x.

Nick Gibbens, The Shaving Shack’s Brand Director, believes Bond’s legendary style and iconic status have inspired men to try out a new method of wet shaving. “Bond fans have an emotional attachment to 007, they love his Aston Martin cars, Omega watches and dinner suits. So it comes as no surprise that they have jumped on his love for cut throat razors.

http://www.shaving-shack.com/blog/james-bond-skyfall-boosts-cut-throat-razor-sales-by-405/

There's a huge opportunity in games to drive aspiration. You're creating a fictional world and allowing other people to play inside it. It's your job as a product creator to shape their feelings as they traverse the game experience. Products that drive aspiration have higher retention, better growth, and a stronger brand.

Creating aspiration is actually a brilliant psychological hack. It gets a person to set a lofty, innate goal for themselves. In the same way that entrepreneurs itch for world domination or artists yearn to change the way people view the world. People will then act irrationally and go to any length to meet that goal. Dan Pink talks about this intrinsic motivation in his book Drive, as not only a powerful motivator, but the best possible way for someone to stay excited and productive.

Products create aspiration in a few ways. It starts with communicating the ideal. Take a look at Pokémon:

Nintendo has a single priority in their marketing of Pokémon. They don't just put any monsters on the front of their games. They put the most bad-ass, epic monsters the game has to offer. It's building an aspiration towards the Pokémon ideal, assembling the coolest monster team possible.

Instagram does this too, albeit in more subtle ways. Their app screenshots are dedicated to showing you the mega-popular photographer you could be, with thousands of ♥s on your photos and just as many followers. They start with the ideal up front.

The next piece is dangling the ideal. The game World of Warcraft executes this brilliantly. Part of the core game loop requires visiting major hubs core to your faction. When you arrive, you're instantly surrounded by people with better gear, cooler weapons, and exclusive mounts.

The game has strong signals of success and attaches them like headlights to your shoulders. When you're walking through Orgrimmar, you're literally bathed in the glow of other people's victories, a strong reminder that you're not the best yet.

The final piece is letting players live the ideal. The road to aspiration in any game has to end in the player hitting their goal. In Skyrim, the player starts by being told they are dragonborn and have a destiny to slay dragons. By the end of the game, they're cutting through dragons like a hot knife through butter. The player was set up to fear and respect dragons and follows the path to become the ultimate dragon killing machine.

Consumer apps do this too. Letting people live the ideal is exactly what Uber does. When they were first written about, the service was described as an experience that makes you “feel like a rockstar”. For $30, you too can live the dream.

Aspiration is one of the strongest human emotions. It inspires us to greatness, puts men on the moon, and pushes humanity forward. With the right design, games can make people aspire too.

Building a Data-Informed Startup

At MinoMonsters, we use data every single day. We spend a lot of time looking at it and we believe it's important. Today we will monitor and record over 10M points of data. But what are we doing with all that information?

We're building a mobile game, but we use data just like any mobile startup would. We use data to track the health of our business and we use it to inform certain decisions. Data helps us understand if we're growing, it helps us debug product flows, and it helps us discover new opportunities for engagement and revenue.

Just as important as the data itself, the cadence we use to look at the data is critical. We can't look at everything, we simply don't have the time or attention. We must be disciplined in how we look at data and ensure we're using high-yield visualizations of our data.

One way we look at data is daily health check-ups where we look at DAU, Installs, and Revenue. For each of these we look at today's numbers and how that compares to yesterday and last week. Fluctuations are to be expected, but drops are not. This helps us see our growth and alerts us if anything is going wrong.

We put these numbers up on a TV in the middle of the office. This keeps us accountable and helps quantify the impact each person is having on the world.

(*The numbers aren't censored on the real thing*)

We also use data to help us understand how players use our products. This in turn helps us optimize the products. We use this data in ad-hoc product analysis to help us better understand how our features perform.

One way we optimize is with a funnel analysis. A funnel analysis can quickly identify the parts of a feature that confuse, bore, or crash on our players. We use this often to look at our first time user experience (FTUE). Before we ship, we identify each step in the FTUE. After we ship, we examine each step and determine where users are lost.

These are two of the tools we use to improve our product. What's important is that we display the data efficiently to quickly gain insights and that we only use the most important metrics to keep us focused. Our jobs are as game designers, first and foremost. Our goal is to ensure that players are having fun. Data helps us do just that.

Owning your destiny

Patrick Wyatt, creator of the Guild Wars series and one of the original developers at Blizzard Entertainment, talks about the birth of the Warcraft franchise. Originally, they sought to license the Warhammer brand for use in the game. They decided against it.

It’s surprising now to think what might have happened had Blizzard not controlled the intellectual property rights for the Warcraft universe — it’s highly unlikely Blizzard would be such a dominant player in the game industry today.

Blizzard has made over 2 billion dollars on the Warcraft franchise from the games alone. That's not including the card games, conferences, swag, virtual pets, or their upcoming feature film.

Activision Blizzard is now worth over $13,000,000,000.

Increasing retention for social games

Retention is a tough topic. Every consumer app startup is concerned with it but no one really knows how to move it. Executing on retention can be brutal. You can try hundreds of different tricks, changes, tweaks, and pivots and still not see the needle move. It's also a really important piece of data. For free consumer apps, your retention is a key factor in your customer lifetime value. Retention is one of the hardest and important metrics to move in a startup.

As Nabeel Hyatt points out, most people cannot move the needle on retention unless something is fundamentally wrong.

… in general you will not make retention go up materially post launch. I have seen virality increase materially, engagement go up, monetization improve, but rarely do you see retention really pop after launch.

Retention is a really hard problem.

At MinoMonsters, we've also struggled with retention. Our initial alpha product was a leaky bucket, shedding over 95% of new users within the first few minutes of play. Since then, we been ruthless about improving retention. Through strategic product changes, we've been able to raise our Day 1 retention to over 50%, where we sit today. The process has taken over a year and we've learned a lot along the way. Here are some of the things we did that helped.

• Have great aspiration. You want players to feel like they're working towards something grandiose. You want them setting their own goals.
• Craft a clear, fun, first time user experience (FTUE). You need to teach players the game and how you teach it makes all the difference.
• Don't make me read. Communicate visually in your game as much as possible. Show, don't tell your players how to play the game. If players have to read just to understand what's going on, you'll lose them.
• Play with friends. Connecting and interacting with friends allows players to bring each other back into the game. Make this core.
• Build a fun game. There's not much to say here. Fun games help retain players.

There are also some tactics you can use to increase retention. These aren't silver bullets, but they apply to a broader set of consumer apps.

• Email lifecycle marketing. Email is an engagement channel that has stood the test of time.
• Constant content updates. If players know that you are constantly curating the game, they'll come back to see what's new.
• Timers. Give the players game content where coming back is part of the core game loop.
• Notifications. Use the notification channel of your platform to send relevant messages to your players.

And lastly, there are some things that are guaranteed to hurt retention.

• Slow load times
• Crashes
• Being confusing
• Being boring

We've used these strategies and tactics over the last year to 10x our Day 1 and Day 7 Retention. If you'd like to start moving the needle on retention today, here's the best place to start.

• Do a funnel dropoff analysis of your FTUE. Fix the steps that are broken. If you don't have a FTUE, build one.
• Start tracking Day 1 Retention. Make tweaks to your core game loop until it goes up. Do not focus on anything else until you reach 30%.
• Put the game in the hands of random people from Craigslist. Just watch how they interact with the game. Give them as little instruction as possible. Fix the user experience issues you observe.

Retention is a complex topic and there's no one-size-fits-all solution. We increased retention through focus, hard work, and bold changes to our product.

This post was originally published on Quora.

Thanks to Nabeel Hyatt for reading a draft of this.

Sharecropping on Facebook

It wasn't that long ago that Viddy was raising $30M for it's “Instagram for Video” app that had stormed Facebook. It was being heralded as the return of virality for its spammy Open Graph integration. I wonder what happened to that?

Facebook pushes big updates on Tuesdays. They have taken these Tuesdays as an opportunity to systematically crush Viddy's DAU to under a million. Why would they do that? Why give free distribution away when they can charge for it with ads.

When building for Facebook you can abuse viral opportunities all day long, but when it comes down to it, you're playing on Facebook's turf. And Facebook doesn't fight fair.

The motives behind Facebook Camera

Facebook has just announced Camera, a standalone iOS app that's been over a year in the making.

This app execution is fantastic. Compared to the classic Facebook app, Camera runs smoothly, loads fast, has intuitive use, and feels native. But why would Facebook release a separate camera app? Don't they already own Instagram?

The answer lies in Facebook's secret sauce to engagement - photos. The entire engine of Facebook runs on users taking, tagging, liking, and commenting on photos. Without it, Facebook is nothing. And if you take a close look, you'll see that the entire Camera app is designed to get you to post as many photos as possible.

They have camera roll integration, allowing me to quickly pick a photo to upload.

Their story post form includes an extra call to action to add more photos.

They used a batch-style picker that lets me upload lots of photos to Facebook at once.

Also, using some nefarious device-based tracking, Camera knows exactly who I am, as soon as I've installed it.

And if it wasn't already obvious, Facebook clearly wants to be a replacement for the default Camera app.

This all makes complete sense for Facebook. They know that active user numbers for the desktop website are drying up. They also know that those people are moving to mobile. If Facebook plans to survive, they need to make sure their primary viral channel makes it on to mobile as well. They need to do whatever they can to get more users taking more photos.

If you can't find the app on the store, you can download it here.

The OMGPOP acquisition: 6 weeks later

Now that the dust has settled on Zynga's purchase of mobile game developer OMGPOP, the outlook isn't quite as rosy as it was just a month ago. Most Zynga titles take months before hitting their peak and then slowly decline. Draw Something has lost about five million users in a single month just weeks after the acquisition. That's bad news for Zynga.

Zynga doesn't simply dish out $180M for a team of 40 in New York. When they bought OMGPOP they expected their games to perform. Newtoy, the Texas-based mobile games developer, was acquired by Zynga for $50m. Their hit title Words With Friends has held in Top Grossing for over two years. Draw Something could be out in the next two months.

Zynga beat earnings estimate for Q1 this year, largely due to the OMGPOP acquisition. If the Draw Something studio can't hold on to it's DAU, Zynga could be hurting hard in Q2.

How Hacker News Saved My Startup

I'm co-founder of a startup called MinoMonsters, we make an iOS game by the same name. We went through YCombinator in the Winter 2011 batch. On December 18th, the eve of our big launch, I nearly fucked everything up. This is the story of what happened.

MinoMonsters is a monster battling game. Think of it as Pokémon for the iPhone. As is the case with many apps, our game is internet connected. During the course of play, the app talks to a backed server. The server diligently performs tasks like finding your friends, letting you purchase in-game items, and letting you play with others. When planning our server, we needed a solution that was quick to build, had fast performance, and was light over the wire. We chose to build the server in Ruby with EventMachine and Synchrony speaking in protocol buffers over TCP.

Since our soft launch on December 6th, the server has been chugging away without fault. But then the problems began to emerge. It all started with intermittent crashes. I would wake up to find that the running EventMachine process had simply stop running. The approach to understanding this problem is two-fold: recreate the crash to understand the “how” and capture a log to understand the “why”. Since the crashes were intermittent, I had no way of predictably crashing the server. So instead I started with logging.

First, by adding logging to my init script:

#!/usr/bin/env /home/ubuntu/.rvm/bin/ruby-1.9.3-p0
require 'daemons'
Daemons.run_proc(..., :backtrace => true) do
  ...
end

Next, by logging errors in EventMachine's error_handler:

EM.synchrony do
  ...
  EM.error_handler do |e|
    MinoServer.log_error "Error raised during event loop: #{e.message}"
    MinoServer.log_error(e.backtrace.join "\n")
  end
  ...
end

Between these two, I figured I was covered. The next step was to wait and watch it crash again. As I monitored the server, I started to see a variety of errors across my application:

#Errno::EMFILE: Too many open files - content/627/inventory.plist
...
Error in .../new_relic/data_serialization.rb:27:in `read_and_write_to_file'
...
#LoadError: no such file to load -- strscan
...
HTTP request to https://graph.facebook.com/me returned HTTP 0, ""

I hadn't found the source of the crashes, but I had found some issues. Upon seeing these issues, my first gut reaction was that I must be running out of file descriptors.

$ ulimit -n
1024

Bingo. Let's fix that before doing anything else.

$ sudo sh -c '(echo "ubuntu soft nofile 40000" && echo "ubuntu hard nofile 40000") > /etc/security/limits.conf'
$ ulimit -n
40000

Then I methodically went through each error and devised a solution. It wouldn't solve my problem, but it would narrow down the options. Over two weeks, I casually “fixed” more crashes as they cropped up and eventually they stopped.

Then the day came. We had planned to set the price of our game to free. As soon as we did, users started flooding in. And then the problems started anew.

As before, the server was crashing. I couldn't recreate the crash, but I didn't need to. At first, it was crashing every hour. Then it was every 30 minutes. Soon enough, the server crashed within seconds of starting up. Logs became increasingly useless, as the places in code that threw exceptions were random at best. Something was falling apart and I couldn't figure out what it was. Without a backend, we couldn't verify virtual goods purchases. And without virtual good purchases, we would make no money off the crazy initial traction we received. This was bad news.

I spent four hours trying everything I knew to solve the problem. I scoured the web for obscure bugs. I completely ripped out Synchrony. I triple checked OS resources like file descriptors, disk space, and others. I was out of ideas. That's when I sent out a distress beacon on Hacker News.

Within minutes I had dozens of people helping me get connected with the right Rubyists. In the end, two people helped me out: Ryan Stout (@ryanstout) and Aman Gupta (@tmm1). With Ryan, we immediately started with generic diagnosis steps. What does the server do? What systems does it touch? We systematically went through each touchpoint and observed it's potential contribution to the problem. This helped us eliminate any external factors (OS, Ruby, Library) as the culprit. Then we got Aman involved. As one of the maintainers of EventMachine, he was able to quickly start looking at my configuration. Within minutes, he noticed something odd in my setup code:

EM.synchrony do
  EM.epoll
  EM.start_server "0.0.0.0", 8090, MinoServer
  EM.error_handler do |e|
    MinoServer.log_error "Error raised during event loop: #{e.message}"
    MinoServer.log_error(e.backtrace.join "\n")
  end

  MinoServer.initialize_server
end

This code was taken near-verbatim from an Ilya Grigorik post on EventMachine from 2008. The code from that post originally looked like this:

EventMachine::run {
  EventMachine.epoll
  EventMachine::start_server("0.0.0.0", 8080, Handler)
  puts "Listening..."
}

Aman saw the issue right away. EventMachine.epoll is a configuration method that must be called before the run loop. From the documentation:

Call the method EventMachine#epoll anytime before you call EventMachine#run, and your program will automatically use epoll, if available

It turns out that this basic setup code that I copied from a 3-year-old blog post was incorrectly implemented. If I had been more careful about reading the documentation, it's possible I might've caught this issue myself. The fix is easy:

EM.epoll
EM.synchrony do
  EM.start_server "0.0.0.0", 8090, MinoServer
  EM.error_handler do |e|
    MinoServer.log_error "Error raised during event loop: #{e.message}"
    MinoServer.log_error(e.backtrace.join "\n")
  end

  MinoServer.initialize_server
end

I made that change, deployed, and crossed my fingers. Within minutes we knew that it worked. Users were connecting en masse. Thousands of monster battles started to ensue. Virtual goods transactions began flowing. All was right with the universe once more.

Let's look at what happened:

• EventMachine uses Ruby's select by default for network I/O. This is a sane default
• Ruby's select performance degrades as the number of persistant connections goes up. It has an upper limit of 1024 connections.
• I was aware of both EventMachine's default behavior and the limitations of select.
  
• Before the game launched I looked at the code to make sure I was making the configuration call to switch to epoll. It was there.
• I did not confirm that the configuration call was working. I never inspected the running state of the app and double-checked that it was using epoll instead of select.
• Ultimately, I failed to ensure that a critical scaling component was activated before going live. Since being fixed, the rest of the system has run smoothly, resulting in thousands of concurrents and millions of monster battles.

Huge thanks to Aman Gupta, Ryan Stout, and the supportive Hacker News community for helping me come to a solution quickly. With you all, I was able to take a multi-day problem and get it fixed in a matter of hours. Thanks.