Evaluating Music Recommendation Systems: Binary Feedback & Stakeholders

In the age of streaming, music recommendation systems are vital, shaping listener experiences and artist visibility. Evaluating these systems requires navigating complex feedback and diverse stakeholder needs. This article delves into evaluating music recommendations using binary feedback, emphasizing multiple stakeholder considerations to better understand the complexities of algorithmic evaluation. This includes identifying implicit and explicit biases in data.

The Challenge of Explicit Feedback Datasets: Self-Selection Bias

Explicit feedback datasets, commonly found on platforms like YouTube

, Amazon, and Uber, present a significant challenge due to self-selection bias. This bias arises because users are consciously providing ratings or reviews, and their decision to do so is often influenced by their strong opinions—either very positive or very negative. This means the available data may not accurately represent the preferences of the general user base.

Consider YouTube, where a Music video might have millions of views , but only a fraction of viewers actively engage by liking or disliking the video. Similarly, on Amazon, customers are more likely to leave a review if they are exceptionally pleased or disappointed with a product. These ratings are often biased, skewing the overall perception of the quality or relevance of items being recommended.

This bias has implications for the training and evaluation of music recommendation systems. If a system is trained primarily on explicit feedback, it may disproportionately cater to the vocal minority, potentially overlooking the preferences of the less engaged majority.

Key Takeaways:

• Explicit feedback data is readily available but often skewed.
• Self-selection bias can lead to inaccurate representations of user preferences.
• Relying solely on explicit feedback may limit the diversity and relevance of recommendations.

The Noisy Reality of Implicit User-Item Datasets

Implicit user-item datasets, which track user behavior without explicit feedback (e.g., plays, skips, shares), offer an alternative but come with their challenges

. A core issue is the presence of noise in the data. For instance, in a music streaming service, a track might be automatically added to a user's playlist or played passively without genuine endorsement. A song may have just been passively played in a playlist and may not be an endorsement of the song.

In an implicit dataset, a '1' often indicates that a user has played an item, while a '0' suggests it was not played. The interpretation of '0' is complex because it may mean the user dislikes the item or that they simply have not encountered it.

The result is a dataset skewed towards '0s', creating an imbalance that poses difficulties for recommendation algorithms. Such algorithms are trained on this type of data set. This dataset leads to the assumption that a song that has not been listened to will not be liked.

Moreover, relying solely on play counts or similar metrics risks reinforcing existing popularity biases, as the algorithm might favor items already trending.

Key Takeaways:

• Implicit datasets, while abundant, can be noisy and unreliable indicators of user preference.
• The interpretation of 'not played' data is ambiguous, leading to potential misrepresentations.
• Imbalanced datasets require careful handling to avoid reinforcing existing biases and filtering bubble risks.

Understanding the Piki User Interface

To address the shortcomings of both explicit and implicit feedback, innovative data collection methods have emerged

. The Piki user interface (UI) offers one such approach.

The Piki interface is designed to balance the need for explicit feedback with the desire to minimize self-selection bias. Users are presented with snippets of music videos, and their engagement is tracked over time. Crucially, users are not immediately prompted to rate or review the content. This forces the user to evaluate their interest in the music video in a very short time frame.

The Piki interface provides a series of engagement options over time:

• Initial Exposure: Immediately upon seeing the video.
• Skip/Dislike (After 3 seconds): The option to skip if uninterested is available quickly, reducing the risk of passive engagement.
• Like (After 6 seconds): A 'like' indicates moderate interest, demonstrating engagement beyond a fleeting impression.
• Super Like (After 12 seconds): A 'super like' signals strong positive sentiment, differentiating highly favored content.
• Replay (After 30 seconds): Provides an implicit binary feedback dataset, where it is considered if an item is played or not played.

Key Takeaways:

• The Piki interface seeks to bridge the gap between explicit and implicit feedback mechanisms.
• Graduated engagement options provide nuanced insights into user preferences.
• The time-sensitive nature of the interface encourages more deliberate and representative feedback.

Analyzing the Piki Music Dataset: Data Structure and Insights

The data collected through the Piki interface forms the Piki Music Dataset

, which offers a rich source of information for evaluating music recommendation algorithms. It’s made of 500,000 rows, and it can be found on the presenter's Github. The Dataset is available to developers.

The dataset comprises the following key elements:

1. Timestamp: Records the time of user interaction.
2. User ID: Identifies the individual user.
3. Song ID: Identifies the specific musical piece.
4. Liked: Captures user sentiment as:
   • 0 = Dislike (skip).
   • 1 = Like.
   • 2 = Super Like.
5. Personalized: Indicates whether the recommendation was from a personalized or random algorithm (0 for random, 1 for personalized).
6. Spotify Popularity: Reflects the artist’s Spotify popularity (0-100, from lesser-known to well-known).

The Piki-ness of a track is correlated with Spotify popularity.

Key Takeaways:

• The Piki Music Dataset provides granular information about user engagement with music.
• It incorporates both explicit (likes, dislikes) and implicit (time spent) feedback signals.
• The inclusion of Spotify popularity allows for analysis of recommendation biases and fairness.

The weighted regularization matrix factorizarion model is able to train with likes and dislikes

Weighted Regularized Matrix Factorization (WRMF) is a useful algorithm that can be used to make a recommendation and be more accurate in their suggestions. When compared to other benchmarks like Popularity (recommend well-known) and Anti-popularity (recommend less known), WMRF is able to increase the accuracy of recommendations by almost 20%. The most benefit can be seen with lesser-known songs that would often be ignored by a typical algorithm. The formula used to train the algorithm is here:

$$min \sum\limits_{(u,i) \in TP} \alpha (1-\hat{p}{ui})^2 + \beta \sum\limits_{(u,i) \in TN} \hat{p}{ui}^2 + \gamma \sum\limits_{(u,i) \in TM} \hat{p}{ui}^2 + \lambda R(x,y)$$

Where:

• $T_P$, $T_N$, and $T_M$ refer to positive, negative, and missing user feedback, respectively.
• $\alpha$, $\beta$, and $\gamma$ are the weights given to those sets of user feedback.
• $R(x,y)$ Regularization terms.

The most important of these terms are \$\alpha\$ and \$\beta\$, as they can make a difference on what data sets the algorithm is trained with. By default the algorithm is often used for positive results. Here are the equations for WRMF with Likes and WRMF with Likes and Dislikes:

• WRMF with Likes: $\alpha = \gamma = 0.5$, $\beta = 0$. This is meant for algorithms that only sample positive user input. The equation works by ignoring the negative feedback, and then sampling the positive and missing data.

• WRMF with Likes and Dislikes: $\alpha = \beta = 0.5$, $\gamma = 0$. This is meant for algorithms that sample positive and negative user input, without having to sample from missing data.

Metrics Used

Evaluating a recommendation system for music can be very tricky and challenging. To fully measure different algorithms there needs to be multiple different metrics, and each stakeholder values a different metric that they should focus on. By testing in different areas, the quality of music data sets are improved.

There were three types of stakeholders that metrics were created around:

• Well-Known Artists What % of songs from well-known artists in the evaluation set are highly liked.
• Lesser-Known Artists What % of songs from lesser-known artists in the evaluation set are highly liked.
• Consumers What % of recommended songs in the evaluation set are actually liked.

A Stakeholder Approach

Traditional evaluation metrics for recommendation systems often prioritize overall accuracy. But they can overlook the diverse and sometimes conflicting interests of different stakeholders, such as artists (well known and lesser known), consumers, and music labels.

Recognizing this limitation is critical, and the need to take a more stakeholder centered evaluation approach needs to take place. These metrics are:

• To measure consumer satisfaction the evaluation system needs to measure how accurately it predicts the needs and desires of users. If it suggests songs that users enjoy, then this benefits consumer satisfaction. An algorithm does not make this a priority, that causes customer fatigue. If a specific customer does not like a recommendation it causes customer fatigue.

• To measure artist satisfaction the system needs to also recommend a large number of known/unknown artists to the audience. If an artist is highly suggested, their sales, streams, and overall visibility would be positively affected. A common issue is for recommendation algorithms to only show the most popular artists. To measure this accurately, it is vital to balance out metrics that account for a good mix of all artist types to avoid creating algorithms that only push the most popular artists.

Improving Music Recommendation Algorithms

The dataset provides a benchmark to measure the effectiveness of musical recommendation algorithms. Some areas that can be measured is the degree to which they help or are useless to different areas of stakeholder needs. This ensures algorithms are optimized to meet different objectives and avoid negative effects.