TY - JOUR
T1 - Effectively mining and using coverage and overlap statistics for data integration
AU - Nie, Zaiqing
AU - Kambhampati, Subbarao
AU - Nambiar, Ullas
N1 - Funding Information:
This research was supported in part by US National Science Foundation grant IRI-9801676 and Arizona State University Prop. 301 grant ECR A601 (to ET-I3). Preliminary versions of this work have been presented at Proc. Third Int’l Workshop Web Information and Data Management (WIDM) 2001 [22] and Proc. ACM Conf. Information and Knowledge Management (CIKM) 2002 [23].
PY - 2005/5
Y1 - 2005/5
N2 - Recent work in data integration has shown the importance of statistical information about the coverage and overlap of sources for efficient query processing. Despite this recognition, there are no effective approaches for learning the needed statistics. The key challenge in learning such statistics is keeping the number of needed statistics low enough to have the storage and learning costs manageable. In this paper, we present a set of connected techniques that estimate the coverage and overlap statistics, while keeping the needed statistics tightly under control. Our approach uses a hierarchical classification of the queries and threshold-based variants of familiar data mining techniques to dynamically decide the level of resolution at which to learn the statistics. We describe the details of our method, and present experimental results demonstrating the efficiency of the learning algorithms and the effectiveness of the learned statistics over both controlled data sources and in the context of BibFinder with autonomous online sources.
AB - Recent work in data integration has shown the importance of statistical information about the coverage and overlap of sources for efficient query processing. Despite this recognition, there are no effective approaches for learning the needed statistics. The key challenge in learning such statistics is keeping the number of needed statistics low enough to have the storage and learning costs manageable. In this paper, we present a set of connected techniques that estimate the coverage and overlap statistics, while keeping the needed statistics tightly under control. Our approach uses a hierarchical classification of the queries and threshold-based variants of familiar data mining techniques to dynamically decide the level of resolution at which to learn the statistics. We describe the details of our method, and present experimental results demonstrating the efficiency of the learning algorithms and the effectiveness of the learned statistics over both controlled data sources and in the context of BibFinder with autonomous online sources.
KW - Association rule mining
KW - Coverage and overlap statistics
KW - Query optimization for data integration
UR - http://www.scopus.com/inward/record.url?scp=19944371158&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=19944371158&partnerID=8YFLogxK
U2 - 10.1109/TKDE.2005.76
DO - 10.1109/TKDE.2005.76
M3 - Article
AN - SCOPUS:19944371158
SN - 1041-4347
VL - 17
SP - 638
EP - 651
JO - IEEE Transactions on Knowledge and Data Engineering
JF - IEEE Transactions on Knowledge and Data Engineering
IS - 5
ER -