Tech

Should we use ML?

• Well defined problem

• Lack of easy solution, can provide examples but can't code

• High quality data

• Clear and meaningful Evaluation

• Loss Function

• Solution is justified

Decision Trees

• Bagging

• Boosting (Adaptive boosting): combining small decision tress

• Gradient Boosing: fits Trees on error residuals (like gradient descent, figure out error and mimimize it)

Continued taking Large Scale AI lectures.

Lecture 3

Supercomputer lecture by CSCS

Lecture 4

Communication overhead

Lecture 5

Model & Data Parallelism

Lecture 6

Mixture of Expert Models

An Introduction to Matrix factorization and Factorization Machines in Recommendation System, and Beyond

a rating matrix R

collaborative filtering algorithms: (statistics method)

• User-based: items which similar users have liked

• Item-based: similar items according to users’ part behaviour

Matrix factorization

• recommendation process is time-consuming

• You first do SVD

• dot product between two vectors to express similarity

• mask matrix to mask out empty ratings

PeaPOD: Personalized Prompt Distillation for Generative Recommendation

• integrates a user’s individual preferences with collective preferences from similar users to build personalized soft prompts

discrete prompt templates:

• perform additional fine-tuning to allow the LLM to effectively understand these IDs

collaborative user prompt:

• distill user preferences into a soft prompt

• leveraging collaborative knowledge from users with similar preferences

• multi-head attention to create a contextualized user representation that combines individual and group-level information

• select an effective input to the multi-head-attention module

  • utilize a user embedding generated from probabilistic matrix factorization

• extended from a single-head implementation to a multi-head attention

• We first initialize a set of user embeddings using probabilistic matrix factorization. We then apply multi-head attention, where the user embedding of target user 𝑢 is passed to the query vector and the embeddings of the top-n most similar users 𝑆𝑢 are passed to the key and value vectors. The final output is the collaborative user prompt 𝒑𝑚ℎ𝑎, which dynamically captures collaborative knowledge from 𝑆𝑢 based on 𝑢’s personalized preferences.

I disagree with the paper, after probabilistic matrix factorization top-n most similar users are already skewed for the multi head attention

Use Sparse Autoencoders to Discover Unknown Concepts, Not to Act on Known Concepts