2025 · October

An Introduction to Matrix factorization and Factorization Machines in Recommendation System, and Beyond

a rating matrix R

collaborative filtering algorithms: (statistics method)

• User-based: items which similar users have liked

• Item-based: similar items according to users’ part behaviour

Matrix factorization

• recommendation process is time-consuming

• You first do SVD

• dot product between two vectors to express similarity

• mask matrix to mask out empty ratings

PeaPOD: Personalized Prompt Distillation for Generative Recommendation

• integrates a user’s individual preferences with collective preferences from similar users to build personalized soft prompts

discrete prompt templates:

• perform additional fine-tuning to allow the LLM to effectively understand these IDs

collaborative user prompt:

• distill user preferences into a soft prompt

• leveraging collaborative knowledge from users with similar preferences

• multi-head attention to create a contextualized user representation that combines individual and group-level information

• select an effective input to the multi-head-attention module

  • utilize a user embedding generated from probabilistic matrix factorization

• extended from a single-head implementation to a multi-head attention

• We first initialize a set of user embeddings using probabilistic matrix factorization. We then apply multi-head attention, where the user embedding of target user 𝑢 is passed to the query vector and the embeddings of the top-n most similar users 𝑆𝑢 are passed to the key and value vectors. The final output is the collaborative user prompt 𝒑𝑚ℎ𝑎, which dynamically captures collaborative knowledge from 𝑆𝑢 based on 𝑢’s personalized preferences.

I disagree with the paper, after probabilistic matrix factorization top-n most similar users are already skewed for the multi head attention

Use Sparse Autoencoders to Discover Unknown Concepts, Not to Act on Known Concepts