Evaluating the Positive Unlabeled Learning Problem

Evaluating PU learning models poses challenges that are not present when evaluating standard supervised classification models. Because all negative data labels are missing in PU datasets, standard evaluation techniques that rely on calculating truth tables cannot be used as neither true negative samples nor false negative samples can be calculated. Because of this, neither a model’s predicted precision nor accuracy can be calculated. Even the methods used to train PU models are different, as the standard supervised train—validate—test modeling technique is not possible when a substantial portion of the training dataset is unlabeled. Supervised classification uses Inductive learning to train a model that can be used on new, unlabeled data as shown in Figure 3.1a. In PU learning, as with many semi-supervised learning methods, either Inductive or Transductive learning is possible. The differences between these are summarized in Figures 3.1b and 3.1c.