Ranking: What 'The Score' Actually Is

What does a score of 0.83 mean? It's not relevance—it's a predicted probability of an action you might take. In episode 381 of the recommender systems series, Joey explores ranking: how systems order hundreds of candidates to show you the top few dozen. He works through published papers from Google and YouTube, learning how the third layer of recommendations actually works. Key moments: 00:00:00 - What is a score? 00:01:48 - From millions to hundreds: retrieval vs. ranking 00:05:19 - Hand-built cross features 00:06:57 - Features matter more than tuning 00:10:06 - YouTube's pointwise ranking approach 00:11:43 - Measuring a better thing 00:13:21 - Judging execution: the diving analogy 00:16:38 - The plumbing problem: architecture first 00:18:29 - Offline and online disagreement 00:20:22 - Models predict; people decide --- Sources & further reading: • Cheng, Koc, Harmsen et al. (Google), Wide & Deep Learning for Recommender Systems, arXiv:1606.07792, 2016-06-24 — — O(10) ms / O(100)-item funnel, feature families, memorization vs generalization, crosses, Tables 1 and 2 (read from the arXiv PDF).: https://arxiv.org/abs/1606.07792 • Zhao, Hong, Wei et al. (Google), Recommending What Video to Watch Next: A Multitask Ranking System, RecSys '19, September 2019 — — engagement vs satisfaction, MMoE, pointwise for serving, "manually tuned" combination, shallow tower, Tables 1 and 2. ACM PDF returned 403; text read from a mirror of the published PDF (https://daiwk.github.io/assets/youtube-multitask.pdf, DOI and front matter match); authors cross-checked at https://research.google/pubs/recommending-what-video-to-watch-next-a-multitask-ranking-system/.: https://doi.org/10.1145/3298689.3346997 • Covington, Adams, Sargin (Google), Deep Neural Networks for YouTube Recommendations, RecSys '16, September 2016 — — user-item history features, retrieval source as a feature, churn, clickbait, weighted logistic regression for watch time.: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45530.pdf • Burges (Microsoft Research), From RankNet to LambdaRank to LambdaMART: An Overview, MSR-TR-2010-82, June 2010 — — RankNet, flat/discontinuous metrics, Figure 1, lambdas as arrows, Yahoo! LTR Challenge.: https://www.microsoft.com/en-us/research/publication/from-ranknet-to-lambdarank-to-lambdamart-an-overview/ • He, Pan, Jin et al. (Facebook), Practical Lessons from Predicting Clicks on Ads at Facebook, ADKDD 2014 — — trees + LR, historical features dominate, Normalized Entropy, calibration, downsampling re-calibration.: https://quinonero.net/Publications/predicting-clicks-facebook.pdf • Wang, Shivanna, Cheng et al. (Google), DCN V2: Improved Deep & Cross Network..., WWW '21 — — feature crosses as "the key" (abstract only).: https://arxiv.org/abs/2008.13535 • All external URLs fetched and quoted from on 2026-10-15. This podcast episode was fully generated by AI — research, script, voices, and production. Built with Claude, Piper TTS, and automated pipeline tooling.