TY - JOUR

T1 - Direct estimation of density functionals using a polynomial basis

AU - Wisler, Alan

AU - Berisha, Visar

AU - Spanias, Andreas

AU - Hero, Alfred O.

N1 - Funding Information:
Manuscript received February 20, 2017; revised July 28, 2017 and October 16, 2017; accepted November 6, 2017. Date of publication November 27, 2017; date of current version December 22, 2017. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Chandra Ramabhadra Murthy. The work of V. Berisha was supported in part by the Office of Naval Research under Grants N000141410722 and N000141712826. The work of A. O. Hero was supported by the Army Research Office under Grant W911NF-15-1-0479. (Corresponding author: Alan Wisler.) A. Wisler is with Arizona State University, Tempe, AZ 85281 USA (e-mail: alanwisler@gmail.com).
Publisher Copyright:
© 2017 IEEE.

PY - 2018/2/1

Y1 - 2018/2/1

N2 - Anumber of fundamental quantities in statistical signal processing and information theory can be expressed as integral functions of two probability density functions. Such quantities are called density functionals as they map density functions onto the real line. For example, information divergence functions measure the dissimilarity between two probability density functions and are useful in a number of applications. Typically, estimating these quantities requires complete knowledge of the underlying distribution followed by multidimensional integration. Existing methods make parametric assumptions about the data distribution or use nonparametric density estimation followed by high-dimensional integration. In this paper, we propose a new alternative. We introduce the concept of "data-driven basis functions"-functions of distributions whose value we can estimate given only samples from the underlying distributions without requiring distribution fitting or direct integration. We derive a new data-driven complete basis that is similar to the deterministic Bernstein polynomial basis and develop two methods for performing basis expansions of functionals of two distributions.We also show that the new basis set allows us to approximate functions of distributions as closely as desired. Finally, we evaluate the methodology by developing data-driven estimators for the Kullback-Leibler divergences and the Hellinger distance and by constructing empirical estimates of tight bounds on the Bayes error rate.

AB - Anumber of fundamental quantities in statistical signal processing and information theory can be expressed as integral functions of two probability density functions. Such quantities are called density functionals as they map density functions onto the real line. For example, information divergence functions measure the dissimilarity between two probability density functions and are useful in a number of applications. Typically, estimating these quantities requires complete knowledge of the underlying distribution followed by multidimensional integration. Existing methods make parametric assumptions about the data distribution or use nonparametric density estimation followed by high-dimensional integration. In this paper, we propose a new alternative. We introduce the concept of "data-driven basis functions"-functions of distributions whose value we can estimate given only samples from the underlying distributions without requiring distribution fitting or direct integration. We derive a new data-driven complete basis that is similar to the deterministic Bernstein polynomial basis and develop two methods for performing basis expansions of functionals of two distributions.We also show that the new basis set allows us to approximate functions of distributions as closely as desired. Finally, we evaluate the methodology by developing data-driven estimators for the Kullback-Leibler divergences and the Hellinger distance and by constructing empirical estimates of tight bounds on the Bayes error rate.

KW - Bernstein polynomial

KW - Direct estimation

KW - Divergence estimation

KW - Nearest neighbor graphs

UR - http://www.scopus.com/inward/record.url?scp=85035817032&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85035817032&partnerID=8YFLogxK

U2 - 10.1109/TSP.2017.2775587

DO - 10.1109/TSP.2017.2775587

M3 - Article

AN - SCOPUS:85035817032

VL - 66

SP - 558

EP - 572

JO - IEEE Transactions on Acoustics, Speech, and Signal Processing

JF - IEEE Transactions on Acoustics, Speech, and Signal Processing

SN - 1053-587X

IS - 3

ER -