TY - GEN
T1 - On sampling collaborative filtering datasets
AU - Sachdeva, Noveen
AU - Wu, Carole Jean
AU - Mcauley, Julian
N1 - Publisher Copyright:
© 2022 Owner/Author.
PY - 2022/2/11
Y1 - 2022/2/11
N2 - We study the practical consequences of dataset sampling strategies on the ranking performance of recommendation algorithms. Recommender systems are generally trained and evaluated on samples of larger datasets. Samples are often taken in a naive or ad-hoc fashion: e.g. by sampling a dataset randomly or by selecting users or items with many interactions. As we demonstrate, commonly-used data sampling schemes can have significant consequences on algorithm performance. Following this observation, this paper makes three main contributions: (1) characterizing the effect of sampling on algorithm performance, in terms of algorithm and dataset characteristics (e.g. sparsity characteristics, sequential dynamics, etc.); (2) designing SVP-CF, which is a data-specific sampling strategy, that aims to preserve the relative performance of models after sampling, and is especially suited to long-tailed interaction data; and (3) developing an oracle, DATA-GENIE, which can suggest the sampling scheme that is most likely to preserve model performance for a given dataset. The main benefit of DATA-GENIE is that it will allow recommender system practitioners to quickly prototype and compare various approaches, while remaining confident that algorithm performance will be preserved, once the algorithm is retrained and deployed on the complete data. Detailed experiments show that using DATA-GENIE, we can discard upto 5x more data than any sampling strategy with the same level of performance.
AB - We study the practical consequences of dataset sampling strategies on the ranking performance of recommendation algorithms. Recommender systems are generally trained and evaluated on samples of larger datasets. Samples are often taken in a naive or ad-hoc fashion: e.g. by sampling a dataset randomly or by selecting users or items with many interactions. As we demonstrate, commonly-used data sampling schemes can have significant consequences on algorithm performance. Following this observation, this paper makes three main contributions: (1) characterizing the effect of sampling on algorithm performance, in terms of algorithm and dataset characteristics (e.g. sparsity characteristics, sequential dynamics, etc.); (2) designing SVP-CF, which is a data-specific sampling strategy, that aims to preserve the relative performance of models after sampling, and is especially suited to long-tailed interaction data; and (3) developing an oracle, DATA-GENIE, which can suggest the sampling scheme that is most likely to preserve model performance for a given dataset. The main benefit of DATA-GENIE is that it will allow recommender system practitioners to quickly prototype and compare various approaches, while remaining confident that algorithm performance will be preserved, once the algorithm is retrained and deployed on the complete data. Detailed experiments show that using DATA-GENIE, we can discard upto 5x more data than any sampling strategy with the same level of performance.
KW - Benchmarking
KW - Coreset mining
KW - Large-scale learning
KW - Sampling
UR - http://www.scopus.com/inward/record.url?scp=85125798224&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85125798224&partnerID=8YFLogxK
U2 - 10.1145/3488560.3498439
DO - 10.1145/3488560.3498439
M3 - Conference contribution
AN - SCOPUS:85125798224
T3 - WSDM 2022 - Proceedings of the 15th ACM International Conference on Web Search and Data Mining
SP - 842
EP - 850
BT - WSDM 2022 - Proceedings of the 15th ACM International Conference on Web Search and Data Mining
PB - Association for Computing Machinery, Inc
T2 - 15th ACM International Conference on Web Search and Data Mining, WSDM 2022
Y2 - 21 February 2022 through 25 February 2022
ER -