Airbnb Superhost Badge: Causal Impact Study (Regression Discontinuity)
A regression discontinuity re-test of whether Airbnb's Superhost badge still moves host outcomes now that Guest Favorite has taken over discovery. Five of six outcomes show no jump at the 4.8 cutoff; only review volume does.
Problem Statement
Airbnb awards the Superhost badge quarterly to hosts who clear four bars over a trailing year: a rating of at least 4.8, a response rate of at least 90%, a cancellation rate under 1%, and enough completed stays. Historically the badge did two jobs: it pushed hosts to maintain quality, and it signalled trust to guests, who could filter search results by it. A 2023 study used that 4.8 cutoff as a regression discontinuity and found a significant positive effect on bookings and revenue. The platform has since changed: Guest Favorite launched in November 2023 as a listing-level badge covering roughly two million homes, and around January 2024 the Superhost filter was removed from on-site search and replaced by a Guest Favorite one. The badge survives; the discovery mechanism behind half of its value does not. The question is whether the causal effect survived with it. It cannot be answered by comparing Superhosts to everyone else, because the hosts who clear the bar were already better hosts; any gap would be selection, not the badge. What makes the cutoff useful is that it is arbitrary at the margin: a host at 4.79 and a host at 4.81 are, on average, indistinguishable in underlying quality, and the badge is essentially the only systematic difference between them. Comparing those two narrow groups is the whole design.
Results & Impact
1 of 6
Significant Effects
Only review volume jumps at the 4.8 cutoff
0.225 – 0.974
p-value Range
Across the five null outcomes, HC1 robust
6,311 hosts
Analysis Sample
4,308 treated, 2,003 control
125,685 hosts
Panel Built
From 34 U.S. metros, ~3 GB raw
± 0.05
Bandwidth
Ratings 4.75 – 4.85 around the cutoff
+79.07
Review Volume
Reviews per host at the cutoff, p = 0.02
Crossing the 4.8 Superhost cutoff moved five of the six outcomes not at all: no significant jump in guest sentiment, and none in how guests rate value for money. The one outcome that does jump is review volume, and it is the one the design is least able to vouch for.
The six were positive review sentiment, negative review sentiment, neutral review sentiment, overall (compound) sentiment, review volume per host, and the value-for-money sub-rating. Every one was estimated the same way, an OLS jump term at the cutoff with robust standard errors. Five came back statistically indistinguishable from zero, with p-values from 0.2253 to 0.9744. The sixth, review volume, cleared the 5% line: hosts just above the cutoff carry about 79 more total reviews than hosts just below, p = 0.0201. Total reviews is a stock accumulated over a host's entire history rather than a flow generated after the badge was awarded, so that step reads more like the treated side being the more established side than like the badge doing the work.
Before Guest Favorite
Mishra, Huang & Kalwani (2023) ran a regression discontinuity on this same 4.8 cutoff and found a significant positive effect of the badge on bookings and revenue. Guests could filter search by Superhost, so the badge did real discovery work.
After Guest Favorite
Airbnb launched Guest Favorite in November 2023 and dropped the Superhost filter from search around January 2024. Re-run on post-change data, the same design finds no detectable effect on guest sentiment or perceived value.
The reading that fits both results is that Guest Favorite has taken over the job the Superhost badge used to do. The badge itself never went away; it is still awarded quarterly and still shown on listing pages, but the mechanism that made it move guest behaviour, being a filter guests searched by, was handed to a different badge. What is left at the cutoff is a label that no longer visibly changes how guests respond.
Worth stating plainly: this is a before-and-after across two separate studies on different samples, not one design that observes the switch happening. It shows the earlier effect is absent now on five of six outcomes; it does not by itself isolate Guest Favorite as the cause, and Guest Favorite status is not observable in this dataset. The review-volume result would need a manipulation test and a covariate-balance check, neither of which this notebook runs, before it could be read as a badge effect.
The Discontinuity, Plotted
Each panel plots one outcome against host_rating inside the ±0.05 bandwidth, with the dashed line marking the 4.80 Superhost cutoff. The red fit is the control side, the green fit the treated side. What a badge effect would look like is a visible step between the two lines where they meet the dashed line. In all six panels they meet it at effectively the same height. That includes the review volume panel: its jump is real in the regression, but a handful of hosts with over twenty thousand reviews stretch that y-axis far enough that a step of seventy nine reviews is invisible by eye.
Click the figure to open it full size.
| Outcome | Superhost effectjump at the cutoff | p-valuerobust | Sig. at 5% |
|---|---|---|---|
| avg_pos_sentimentPositive review sentiment | −0.0020 | 0.3469 | No |
| avg_neg_sentimentNegative review sentiment | −0.0003 | 0.4157 | No |
| avg_neu_sentimentNeutral review sentiment | +0.0026 | 0.2253 | No |
| avg_compound_sentimentOverall sentiment score | −0.0001 | 0.9744 | No |
| total_reviews_across_listingsReview volume per host | +79.0745 | 0.0201 | Yes |
| review_scores_valueValue-for-money sub-rating | −0.0012 | 0.8436 | No |
Five of the six p-values fall between 0.2253 and 0.9744, nowhere near the 5% line, and those point estimates are tiny in absolute terms as well: a few ten-thousandths of a sentiment scale that runs from 0 to 1. For those five the result is not “a small effect we lacked the power to detect” so much as no detectable step at all. Review volume is the exception, and it is worth reading slowly rather than celebrating: total_reviews_across_listings is a stock a host builds up over their whole history, most of it earned long before the current badge was awarded, so a step of ~79 reviews at the cutoff is about as consistent with the treated side simply being the more established side as it is with the badge generating anything.
Read narrowly, this is the badge's demand-side effect at the margin, in a single post-Guest-Favorite cross-section. The absence of any sentiment or perceived value response is consistent with the removal of the Superhost search filter having drained the discovery channel that the 2023 finding partly rested on. It is not evidence that the badge does nothing at all: the supply-side incentive (hosts working to earn and keep it) is a different question that this design cannot answer, and no manipulation test, covariate-balance check or bandwidth-sensitivity check was run here. Those missing checks are exactly what the review-volume jump would need before anyone could call it an effect rather than an imbalance.
Approach & Methodology
Why a Discontinuity, and Not a Comparison
The naive estimate (average outcomes for Superhosts minus average outcomes for everyone else) measures host quality far more than it measures the badge, because the badge is awarded for quality in the first place. A regression discontinuity sidesteps that by shrinking the comparison to a narrow window around a threshold nobody can precisely control: hosts land at 4.79 or 4.81 for reasons that have nothing to do with which side they would have chosen. Inside that window the badge is as good as randomly assigned, so a jump in outcomes exactly at 4.80 is attributable to the badge rather than to the hosts. That local identification is the trade the design makes: a credible estimate of the effect at the cutoff, and no claim at all about hosts at 4.2 or 5.0.
Building a Host-Level Panel From 34 City Extracts
Inside Airbnb publishes per-city files rather than one dataset, so the pipeline unions listings across 34 U.S. metro areas: New York, Los Angeles, Chicago, Boston, Seattle, Broward County and 28 others, roughly 3 GB raw. Each listing's review text is scored with VADER into positive, negative, neutral and compound sentiment; the calendar extract is collapsed into an average nightly price and an availability ratio over the scraped window. Only the final analysis notebook and its saved outputs live in the repository, since regenerating the intermediate table means re-running the full multi-gigabyte pipeline.
Aggregating to the Host, and the Weighting That Matters
Superhost is a host-level designation while the data is listing-level, so every listing is rolled up into one row per host_id, producing 125,685 U.S. hosts. The load-bearing detail is how the running variable is built: host_rating is the host's listings' review_scores_rating averaged weighted by each listing's number of reviews, not a plain mean. Weighted, a host with one heavily-reviewed 4.85 property and one new 4.5 listing stays where their guests actually put them; unweighted, the new listing swings them across the cutoff and into the wrong group. Sentiment scores are aggregated the same way, and review counts are summed.
Defining Treatment and Control Around 4.80
Treatment is hosts with a rating in [4.80, 4.85] who actually hold the badge: 4,308 hosts; control is hosts in [4.75, 4.80) who do not: 2,003. Two inclusion filters apply to both sides: an average response rate of at least 90%, which is Airbnb's own bar, and at least three reviews as an activity floor. Both exist so that Airbnb's other Superhost criteria are held roughly constant and crossing 4.80 is close to the only thing that differs. Conditioning on the badge agreeing with the rating side, rather than on the rating alone, matters because Airbnb re-evaluates Superhost quarterly while the scraped rating is closer to live: hosts whose badge and rating disagree are dropped rather than mis-assigned. The final sample is 6,311 hosts out of 125,685.
The Specification, Stated With Its Assumptions
For each outcome the model is an OLS regression of that outcome on the rating centered at 4.8 plus a treatment dummy, fit with HC1 heteroskedasticity-robust standard errors. Centering makes the treatment coefficient read as the jump exactly at the cutoff. The specification runs one common slope across the bandwidth rather than separate slopes on each side, which is a simplification of the fuller sharp-RDD form; hosts with a fully blocked calendar have their average calendar price imputed from the static listing price rather than being dropped; and the response-rate and review-count filters are proxies for Airbnb's real criteria, not reproductions of them. Each of those is a stated design choice rather than an unexamined default.
Six Outcomes, Because One Would Be a Choice
Running a single dependent variable invites picking the one that cooperates, so the same specification is estimated on six: positive, negative, neutral and compound review sentiment, total reviews across a host's listings, and the review-scores value sub-rating. Together they cover guest experience, engagement volume and perceived value. Each model is plotted as well as tabulated (scatter plus separate fits on either side of the dashed 4.8 line) because a discontinuity is something you should be able to see, and its absence is something a reader should be able to check rather than take on trust.
Reading the Result Honestly
Five of the six do not move. All four sentiment measures and the value-for-money sub-rating come back indistinguishable from zero, with p-values from 0.2253 to 0.9744 and point estimates in the third decimal place. The sixth does move: hosts just above the cutoff carry about 79 more total reviews than hosts just below, p = 0.0201. The honest reading of that one is cautious rather than triumphant, because total reviews is a stock accumulated over a host's entire history and most of it predates the current badge, so a step in it at the cutoff looks more like the treated side simply being the more established side than like the badge generating new reviews. The pattern that survives is the one the sentiment and value results describe: no detectable demand-side response at the margin, consistent with the search-filter removal having drained the discovery channel the earlier finding partly rested on. It says nothing about the supply-side incentive (hosts may still work to keep the badge whether or not guests respond to it) and the design carries real caveats: no manipulation or covariate-balance test, no bandwidth-sensitivity check, one snapshot rather than hosts followed across the threshold over time, and Guest Favorite status not observable in the data to control for directly.
Architecture
── DATA ────────────────────────────────────────────────────────────
┌──────────────────────────────────────────────────────────────────┐
│ Inside Airbnb 34 U.S. metro areas, ~3 GB raw │
│ listings . reviews . calendar, one set of files per city │
│ NYC, LA, Chicago, Boston, Seattle, Broward, 28 more │
└────────────────────────────────┬─────────────────────────────────┘
│ union all 34 cities
↓
┌──────────────────────────────────────────────────────────────────┐
│ Listing-level enrichment │
│ VADER sentiment over each listing's review text │
│ pos / neg / neu / compound, averaged per listing │
│ calendar rollup: avg nightly price, availability ratio │
└────────────────────────────────┬─────────────────────────────────┘
│ roll listings up to their host
↓
┌──────────────────────────────────────────────────────────────────┐
│ Host-level table 125,685 U.S. hosts │
│ host_rating = mean review_scores_rating across a host's │
│ listings, WEIGHTED by each listing's │
│ review count <- the running variable │
└──────────────────────────────────────────────────────────────────┘
── DESIGN - THE 4.8 CUTOFF ───────────────────────────────────────
┌──────────────────────────────────────────────────────────────────┐
│ Running variable host_rating, cutoff c = 4.8 │
│ bandwidth +/- 0.05 treatment D = 1 if rating >= c │
│ a host is kept only if the badge AGREES with their side, │
│ so a stale quarterly badge cannot mis-label a row │
└────────────────────────────────┬─────────────────────────────────┘
│ hold the other criteria fixed
↓
┌──────────────────────────────────────────────────────────────────┐
│ Inclusion filters │
│ host_response_rate >= 90% (Airbnb's own bar) │
│ total_reviews >= 3 (activity floor) │
└────────────────────────────────┬─────────────────────────────────┘
┌──────────────────┴──────────────────┐
↓ ↓
┌───────────────────────────┐ ┌───────────────────────────┐
│ CONTROL │ │ TREATMENT │
│ rating 4.75 - 4.80 │ │ rating 4.80 - 4.85 │
│ host_is_superhost = f │ │ host_is_superhost = t │
│ 2,003 hosts │ │ 4,308 hosts │
└───────────────────────────┘ └───────────────────────────┘
└──────────────────┬──────────────────┘
│ 6,311 hosts in the sample
↓
── ESTIMATION ──────────────────────────────────────────────────────
┌──────────────────────────────────────────────────────────────────┐
│ statsmodels OLS, HC1 heteroskedasticity-robust errors │
│ │
│ y = b0 + b1 (rating - 4.8) + b2 D + e │
│ ^ │
│ b2 is the jump at the cutoff, i.e. │
│ the causal effect of the badge │
└────────────────────────────────┬─────────────────────────────────┘
│ one model per outcome
↓
┌──────────────────────────────────────────────────────────────────┐
│ Outcomes pos / neg / neu / compound sentiment, │
│ total reviews, review_scores_value │
│ │
│ Verdict 5 of 6 show no jump at the 5% level │
│ p from 0.2253 to 0.9744 on those five │
│ total reviews is the exception: +79.07, p = 0.02 │
└──────────────────────────────────────────────────────────────────┘