Discriminative dictionary learning has been widely used in many applications such as face retrieval / recognition and image classification, where the labels of the training data are utilized to improve the discriminative power of the learned dictionary. This paper deals with a new problem of learning a dictionary for associating pairs of images in applications such as face image retrieval. Compared with a typical supervised learning task, in this case the labeling information is very limited (e.g. only some training pairs are known to be associated). Further, associated pairs may be considered similar only after excluding certain regions (e.g. sunglasses in a face image). We formulate a dictionary learning problem under these considerations and design an algorithm to solve the problem. We also provide a proof for the convergence of the algorithm. Experiments and results suggest that the proposed method is advantageous over common baselines.