Maintaining Quality 🏅 Phase 2 of Kickstarting and Scaling a Marketplace Business

Insights from today's biggest marketplaces, including Airbnb, DoorDash, Thumbtack, Etsy, Uber and many more

“French fries and buffalo wings are the most difficult problem in the restaurant delivery industry today. We are obsessed with challenging ourselves to consistently deliver them hot and crispy.”

— Micah Moreau (VP Growth, DoorDash)

Welcome to the second-to-last 😳 post of my eight-part limited series on kickstarting and scaling a marketplace business. In this post, I’ll share the top ten strategies that today’s biggest marketplace companies use to maintain the quality of their marketplaces. And later this week, in our final post, I’ll share what the leaders I spoke to would do differently if they could do it all over again. 🔮

This post contains some of my most favorite stories from the entire series — funny, surprising, and super insightful. Read on!


Step 3: Maintain quality 🏅

The topic of maintaining quality is one of the top five most common questions I get from leaders at marketplace companies. As you scale and attract a more mainstream audience, preserving the level of service that keeps customers coming back becomes increasingly difficult, but also increasingly critical. You’re forced to move beyond your eager early adopters, deal with issues you never anticipated, and develop rules and processes on the fly. I’ve personally spent a lot of time working on marketplace quality at Airbnb (e.g. the Superhost program, reviews, standards enforcement), so I was excited to learn from leaders across seventeen different marketplaces. I’m even more excited to share it all with you. Let’s dive in.

The top ten strategies today’s biggest marketplaces employ to maintain quality:

1. Standards + penalties

The single most common strategy to maintain quality in their marketplace, utilized by about two-thirds of the companies I spoke with, was to define (and share) clear standards for the minimum level of service. They then incentivize good behavior, and penalize bad behavior. At most companies, including Airbnb, this was an evolutionary process, starting with the simplest possible version, and iterating from there.

Eventbrite:

We published community guidelines that outlined what kind of events were permissible on the platform and invested in a fraud/spam team to build technology to catch bad actors. We saw ourselves as the fair arbiter between guests and sellers.”

— Tamara Mendelsohn

Uber:

Since early-on, drivers got a warning when they were below a 4.4 rating. In some markets there was remedial school, like if you had a 4.0.”

— Andrew Chen

Lyft:

We created a set of levers to control the quality of the experience. As a passenger, you live in Oakland, driver comes and finds out you want to go to SF — cancels. That’s a terrible experience. We had a few levers to avoid that, mostly through education and penalties on the driver side. We also understand you have your life as a driver, we understand life gets in the way. For us it was about maintaining that balance, while having cancellation policies that give you a bit of freedom.”

— Benjamin Lauzier

DoorDash:

To manage quality, we take a carrot vs. stick mentality. Sticks would be charging more if they don't hit SLAs for example. What we think is more valuable and better for restaurants long-term is to give them financial incentives to have great operations. For example, we'll lower your commission if you can reduce dasher wait time at the restaurants.”

— Micah Moreau

Caviar:

Our quality standards were  based on the local team's sense of quality. We felt that each region (eg: Dallas vs NYC) had a different sense of quality, so it didn't make sense to standardize nationally.”

— Gokul Rajaram

Etsy:

We cared a lot about making sure items conformed to the terms of service, but this was never actually very clearly defined. Any subjective notion of item ‘quality’ was emphatically not a thing we were policing. I did once have to write a heuristic to remove felt penises from a specific feature.”

— Dan McKinley

2. Manually onboard early supply

The second most common approach, utilized by about half of the marketplaces, was manually onboarding new supply through 1:1 instruction, classes, and hands-on support.

Lyft:

An early part of our foundation was onboarding classes for drivers. We would bring bagels, and have dozens of driver applicants come at once to go through a series of educational videos & chats.”

— Benjamin Lauzier

Airbnb:

For every new listing we brought onboard, we went through a 12-point checklist (e.g. number of pictures, length of title, etc), so that every listing that went live had a minimum standard and was immediately bookable for guests. After listing the spaces, local teams helped to educate the hosts at every single touchpoint. For example, when there was a booking request, the Sales and Account Manager called or emailed the host to explain what is happening and how to convert the request into an actual booking. This global operation was supported by a central data team that allowed us to compare progress across countries and share learnings. Being in touch with nearly every host worldwide helped to get direct market feedback and adapt processes and tools quickly.”

— Georg Bauser

DoorDash:

“First and foremost, the delivery experience reflects on the restaurant’s brand. We are taking ownership of their brand for as short period of time, which terrifies restaurants (understandably so!), and we take that responsibility very, very seriously. We do a lot of unsexy operational work with restaurants in order to help them feel comfortable with this, and to help them succeed. We have a whole team dedicated to merchant operations that helps restaurants understand what this experience (enabling delivery) will do to their kitchen, how do you handle the logistics, how they fill orders, hour by hour. We walk them through the setup, how Dashers will do pickups, how to create a flow.

— Micah Moreau

Instacart:

“One element of quality in our marketplace is our product listing data. For the most part, our customers don’t buy items if they don’t have a good picture. So we went to incredible lengths to get image data for our catalog, even to the point where we would go to a store and buy one of every item just so that we could create the catalog for that store. With a high quality catalog we were able to offer customers a good experience, since what they ordered would be what showed up on their doorstep.”

— Max Mullen

Caviar:

“We found that food standards don't go down — what goes up and down is operations / reliability. In fact, ironically, some of the best food restaurants might not have the best operations. So we had to really help work with the restaurant owners to focus on operations. Because the diner lumps both things together. Quality = great food + great reliability.”

— Gokul Rajaram

Uber:

Funny enough, in the early days, every driver was interviewed by an employee at Uber. The first 20 were interviewed by Travis, trying to get to a certain personality. Also, from a regulatory standpoint, you needed a vehicle inspection, so there was always an offline component.“

— Andrew Chen

3. Reviews

Having a review system was fundamental for maintaining a level of quality (and trust) for about a third of the marketplaces I spoke with.

GrubHub:

On the supply side, we looked at ratings, reviews, and customer service contacts. If you got poor-enough reviews or too many complaints, we had someone talk to you and potentially remove you. This was done qualitatively by the CX team. An issue we ran into and had to work around is if you were a new restaurant and your first few orders go badly and you get screwed long-term.”

— Casey Winters

Airbnb:

“At Airbnb we take reviews and ratings very seriously. In the early days, reviews were a great tool to vet supply (hosts and their listings) and demand (guests) on the growing marketplace. We called hosts and guests to provide feedback about their stay with Airbnb. We connected with everyone, the ones who received good and bad reviews, and the ones that didn't provide reviews. By encouraging people to give reviews we fostered the Airbnb ecosystem and accelerated the growth. A host with many reviews has a higher chance to get booked compared to hosts without reviews. By focussing on reviews we continuously gathered valuable customer insights in different markets and also from various cultures. This helped to improve product, localization and translation as well as on-boarding activities for guests and hosts.”

— Georg Bauser

Zillow:

“We had different tactics in different marketplaces, but one of the biggest ones was customer reviews. This was extremely controversial – they had never been done in the real estate space before).”

— Nate Moch

4. Subsidize it

A similar number of marketplaces, about a third, found ways to subsidize the customer experience in various forms in order to make sure that the customer had a good experience.

Breather:

“With low size transactions (e.g. 2hrs), you can’t walk in and have a place be schmucky. Lower size transactions means they have to be higher quality experience by definition, so you have to subsidize it. We heavily subsidize quality of the room, with great tables, great sound proofing. Meeting rooms are garbage normally. We went for quality through design. And if something went wrong, we solved this through exceptional customer service, and exceptional cleaning.”

— Julien Smith

Instacart:

We would absolutely go out of our way to make sure customers are happy. For instance, we would often redeliver a whole order at our cost in order to save dinner for a customer who received an item they couldn't use or didn't receive a key item in a recipe.”

— Max Mullen

Lyft:

“We always tried to align incentives as much as possible. When giving refunds for things like cancellations for example, it was always as safety net / guarantee for some form of behavior that created a higher quality experience. In other words: if a user does all the things expected out of them to provide a high quality transaction for all sides, and something happened in the middle, we would then rely on some subsidies.”

— Benjamin Lauzier

5. Use search ranking to promote the best supply

One of the most interesting, and also most effective, levers for maintaining marketplace quality turns out to be the search ranking algorithm. When utilized well it’s one of the cheapest yet most powerful tactics you’ll find.

Airbnb:

At Airbnb we thought about quality from both side of the marketplace: how could guests find listings most likely to give them an amazing stay, and, how could hosts ensure they got guests who would love what they had to offer? One of the most hotly discussed features was 'sort by price'. For a long time we didn't offer the ability to sort by price. Why? Because in a naive version, it lowered quality. Price was not always a good indication of quality. The listings at the ends (the most expensive and the cheapest) were more often than not just mis-priced, leading to mis-matched expectations. So we worked on many iterations until we figured out how to both identify mis-priced listings, and let guests express price preferences.”

— Dan Hill

Rover:

Search results were the lever that we used to control everything in the business. That was the product. The reason Rover pulled ahead of competition is that we had a much better understanding of marketplace dynamics. We knew where to seed supply in the search ranking, based on the investment we made in data science. We found that high quality supply drove repeat usage. For example, we prioritized sitters who have done multiple transactions with the same owner. We were constantly updating your quality score, seeding each new sitter appropriately in the search results. That allowed us to spend more to acquire supply, and get more LTV from each customer. We were confident they would be loyal to the platform. My advice is to figure out what the germane signals are for your transaction, and do everything you can to get more of that.”

— David Rosenthal 

Eventbrite:

To keep supply quality high, we initially used a basic algorithm that prioritized events in search results that were selling the most tickets.”

— Tamara Mendelsohn 

6. Quality as differentiator

A broad strategy that emerged from this research, that AngelList, Breather, Caviar, and Lyft utilized, is making quality a differentiator for your business (vs. the competition). It then becomes less about a few tactics to improve quality and more about how to infuse quality into every part of your offering.

Lyft:

Quality was always seen as a differentiator for us. We knew we had a higher bar than our competitors. Wanted to be the company with the most stringent background checks, highest bar for quality in the marketplace. Part of it was very intentional from the beginning.”

— Benjamin Lauzier

Caviar:

The Caviar brand is all about quality — so we focused on restaurant supply above all else.”

— Gokul Rajaram

AngelList:

We only included high-quality investors and only sent them high-quality startups that we personally vetted.”

— Babak Nivi

7. Customer service

Another effective tactic for maintaining quality is having a great customer service experience when things go wrong — something Breather, GrubHub, and OpenTable heavily invested in.

OpenTable:

We invested heavily in customer support. On the restaurant side, it wasn’t just a direct sales force. We built a customer service org from the very beginning. We posted a 1-800-opentable help line on front of our hardware at the host stand. We likely knew more about the restaurant layout then any other supplier. We diagramed every outlet, power cord, phone line and electrical closet.  If a restaurant called we told them to walk 10 feet and check this cord and plug it in beyond panel. We knew if they were going from pen and paper to software, it had to be up available on a Friday night. If it wasn't, we were done as a business. We needed to be their IT staff, so we invested in that. We also created a client management organization that made sure they were getting the most out of the software.”

— Mike Xenakis

GrubHub:

“On the demand side, we defined personas for eaters, a 2x2 of Order with Other vs. Order by Yourself, and Planned vs. Spontaneous. We found the toughest group was Order by Yourself + Planned — they complained the most, found coupon codes, etc. We trained CX to keep the time they give these folks limited. The only eaters we actually kicked off were fraudsters.”

— Casey Winters

8. Find early signals of quality

Possibly the most interesting approach that I’ve come across is companies figuring out a very early signal from new supply that is predictive of long-term quality, and then helping that supply thrive.

TaskRabbit:

“We looked at the most successful TaskRabbits -- asked them to take a mini-Myers Briggs survey, which ended up showing two personality types were going to be good supply: Very service-oriented people, and people who took pride in making people happy. That became part of the onboarding questionnaire.” 

— Brian Rothenberg

Rover:

We started to figure out what were the predictive signals of what would make good supply. When we would onboard supply, we asked them specific questions, and the question that was the most predictive of being a good sitter was ‘Where the dog slept at night?’ If they answered ‘In my bed’, we’d know they’d be a great sitter. So we boosted them in the search rankings. We figured out that people who have experience working with Vets were going to be amazing supply as well.”

— David Rosenthal

9. Add friction

If you find that you have more supply than you know what to do with, like TaskRabbit did, adding friction to the onboarding process may be your best bet.

“Eventually, we had new supply pay an application fee.”

— Brian Rothenberg

10. Set an example

And finally, Etsy discovered a free and underutilized tactic for creating the type of behavior you seek.

Etsy:

“One fun accident was an early investment in merchandising. Etsy built a tool called Treasury that allowed sellers to curate collections around a theme that was then featured on their homepage. The incentive for sellers was that, if their Treasury was picked for the homepage, they would see a bump in views/sales due to the increase in traffic. The merchandising team only picked treasuries where all the items were beautifully photographed. Sellers saw the style Etsy’s merch team liked and mimiced it, creating a high quality bar with a little education/merchandising.

— Nickey Skarstad

As a closing note, two meta-learning that emerged from my interviews:

  1. Almost all of the quality issues are supply-side (i.e. the customer is always right), so most of the focus went there.

  2. You never solve quality. It’s an ongoing battle, with unforeseen bumps and shifting expectations. And it only gets harder as you scale. Nearly every company ended up having full-time teams focused on maintaining (and improving) quality, with ever-increasing levels of sophistication, and no one was immune from quality faux pas.

Which is an excellent segue to our next, and FINAL, post — what leaders at these marketplace companies would have done differently if they were to do it all over again. Don’t miss it. And if you’re finding this series valuable, share it with your friends and click the little gray heart at the bottom (and top) of this email. <3

Share


Overview

Phase 1: Cracking the chicken-and-egg problem:

Phase 2: Scale your marketplace 📈