December 2022: Authored by the Trustworthy Autonomous Recommender Systems on Music Streaming Platforms project team
Online platforms are the virtual marketplace that bring together two sides of online markets, the customers and the suppliers. Customers on these platforms often face a plethora of choices from the suppliers that sell their products on the platform, whether these be goods and services on marketplaces, posts on social media sites, jobs, or web links on search sites. Amazon has an inventory of over 12 million products (over 350 million if Marketplace is included), music streaming platforms offer access to 100 million songs and podcasts, 
YouTube hosts over 800 million videos, Booking.com has over 28 million accommodation listings. In practice, end-users can struggle with such a large choice set. Recommender systems (RS) are designed to mitigate this problem by suggesting relevant products based on an end-user’s preferences, which could help lower/eliminate customer search costs by recommending items that match the consumer’s preferences. This can be good both for end-users and for effective competition between suppliers.
However, recommender systems may not have the ability or incentive to carry out this role perfectly. They function by predicting end-users’ preferences, but they necessarily do this based on incomplete information. Even where RS are intended to be customer-centric, this process of prediction can generate systemic biases in the recommendations made. The nature of these biases depends on the choice of RS model design, the data that feed into the RS model, and feedback loops between these two elements.
There is a large body of computer science literature analysing the impact of these biases on customers (for example from irrelevant or sub-optimal recommendations), but much less has been said about the impact on the other side of the market, suppliers. In our project, we study how these systemic biases risk not only worsening end-user choices but also harming competition between suppliers. They can potentially increase concentration, raise barriers to entry and expansion, create market segmentation and reduce variety. This supplier-side impact is important not just from the perspective of suppliers, but also of the customers, as limited competition between suppliers on a platform can lead to higher prices and reduced innovation. This can happen even when a platform’s interests are broadly aligned with those of its end-users, and the situation may be worsened where these incentives diverge (for example a self-preferencing platform). Our work focuses on these important effects.
How can we measure the impact of RS on supplier competition?
We look at three metrics for supplier competition:
- Market concentration (what share of the market the largest suppliers hold): If biases in RS mean that some items are disproportionately over-represented in recommendations, and some are under-represented, this could make markets more concentrated, especially if these effects are further amplified by feedback loops.
- Market entry (how quickly can new suppliers gain market shares): If RS biases create barriers to entry and expansion, this could make/keep markets more concentrated. If popular or well-established items feature disproportionately in recommendations, it will be difficult for smaller or new entrants to be recommended. This is amplified by feedback loops.
- Product homogeneity (how homogeneous are the recommendations across the population of users). Previous works have found that recommender systems lead to more homogeneity at both the individual but also the population level. We consider homogeneity across the population of users. High product homogeneity means a higher disutility for each user, as very different users are recommended similar items. In turn, this will then reduce variety and ultimately reduce sales of items in the long tail.
Using a number of different RS designs our simulations suggest that RS increases all three of the above metrics, although some RS designs are more likely to do this than others. For example, popularity RS leads to the highest increase in these metrics, and content-based RS are much less harmful on competition.
There are well-tried ways to mitigate the problem but there’s a quality-competition trade-off
If the adverse market impact is due to biases in the outcome generated by the RS, then mitigating or eliminating these biases could help reduce the undesired market impact. We use three intuitively simple approaches to reduce these biases: various hybrid models, add additional objectives (e.g. to force the RS to recommend new items), and implement probabilistic recommendations. All of these enhance exploration to help reduce the tendency of feedback loops to amplify individual and population level homogeneity. These do indeed help reduce the negative impact of RS on supplier competition.
However, this finding itself is not sufficient to fully inform policy. Policymakers need to be aware of the consequences of reducing these biases. We show that the improvement we have in our three metrics of supplier competition come at a cost of reduced recommendation quality. This trade-off should not be ignored when designing an optimal policy response. The decision on which side of the trade-off to prefer depends also on the policymaker’s attitude to short-term and long-term customer harm. Reduced recommendation quality is more likely to be a short-term harm to the customer, whereas reduced competition harms customers in the longer run (through higher prices, lesser choice, and less product variety).