DataXFormer: A robust transformation discovery system

Ziawasch Abedjan; John Morcos; Ihab F. Ilyas; Mourad Ouzzani; Paolo Papotti; Michael Stonebraker

doi:10.1109/ICDE.2016.7498319

DataXFormer: A robust transformation discovery system

Ziawasch Abedjan, John Morcos, Ihab F. Ilyas, Mourad Ouzzani, Paolo Papotti, Michael Stonebraker

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

53 Scopus citations

Abstract

In data integration, data curation, and other data analysis tasks, users spend a considerable amount of time converting data from one representation to another. For example US dates to European dates or airport codes to city names. In a previous vision paper, we presented the initial design of DataXFormer, a system that uses web resources to assist in transformation discovery. Specifically, DataXFormer discovers possible transformations from web tables and web forms and involves human feedback where appropriate. In this paper, we present the full fledged system along with several extensions. In particular, we present algorithms to find (i) transformations that entail multiple columns of input data, (ii) indirect transformations that are compositions of other transformations, (iii) transformations that are not functions but rather relationships, and (iv) transformations from a knowledge base of public data. We report on experiments with a collection of 120 transformation tasks, and show our enhanced system automatically covers 101 of them by using openly available resources.

Original language	English (US)
Title of host publication	2016 IEEE 32nd International Conference on Data Engineering, ICDE 2016
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1134-1145
Number of pages	12
ISBN (Electronic)	9781509020195
DOIs	https://doi.org/10.1109/ICDE.2016.7498319
State	Published - Jun 22 2016
Externally published	Yes
Event	32nd IEEE International Conference on Data Engineering, ICDE 2016 - Helsinki, Finland Duration: May 16 2016 → May 20 2016

Other

Other	32nd IEEE International Conference on Data Engineering, ICDE 2016
Country/Territory	Finland
City	Helsinki
Period	5/16/16 → 5/20/16

ASJC Scopus subject areas

Artificial Intelligence
Computational Theory and Mathematics
Computer Graphics and Computer-Aided Design
Computer Networks and Communications
Information Systems
Information Systems and Management

Access to Document

10.1109/ICDE.2016.7498319

Cite this

Abedjan, Z, Morcos, J, Ilyas, IF, Ouzzani, M, Papotti, P & Stonebraker, M 2016, DataXFormer: A robust transformation discovery system. in 2016 IEEE 32nd International Conference on Data Engineering, ICDE 2016., 7498319, Institute of Electrical and Electronics Engineers Inc., pp. 1134-1145, 32nd IEEE International Conference on Data Engineering, ICDE 2016, Helsinki, Finland, 5/16/16. https://doi.org/10.1109/ICDE.2016.7498319

@inproceedings{e676820b725442ed93114adc124c73b9,

title = "DataXFormer: A robust transformation discovery system",

abstract = "In data integration, data curation, and other data analysis tasks, users spend a considerable amount of time converting data from one representation to another. For example US dates to European dates or airport codes to city names. In a previous vision paper, we presented the initial design of DataXFormer, a system that uses web resources to assist in transformation discovery. Specifically, DataXFormer discovers possible transformations from web tables and web forms and involves human feedback where appropriate. In this paper, we present the full fledged system along with several extensions. In particular, we present algorithms to find (i) transformations that entail multiple columns of input data, (ii) indirect transformations that are compositions of other transformations, (iii) transformations that are not functions but rather relationships, and (iv) transformations from a knowledge base of public data. We report on experiments with a collection of 120 transformation tasks, and show our enhanced system automatically covers 101 of them by using openly available resources.",

author = "Ziawasch Abedjan and John Morcos and Ilyas, {Ihab F.} and Mourad Ouzzani and Paolo Papotti and Michael Stonebraker",

year = "2016",

month = jun,

day = "22",

doi = "10.1109/ICDE.2016.7498319",

language = "English (US)",

pages = "1134--1145",

booktitle = "2016 IEEE 32nd International Conference on Data Engineering, ICDE 2016",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

address = "United States",

note = "32nd IEEE International Conference on Data Engineering, ICDE 2016 ; Conference date: 16-05-2016 Through 20-05-2016",

}

TY - GEN

T1 - DataXFormer

T2 - 32nd IEEE International Conference on Data Engineering, ICDE 2016

AU - Abedjan, Ziawasch

AU - Morcos, John

AU - Ilyas, Ihab F.

AU - Ouzzani, Mourad

AU - Papotti, Paolo

AU - Stonebraker, Michael

PY - 2016/6/22

Y1 - 2016/6/22

N2 - In data integration, data curation, and other data analysis tasks, users spend a considerable amount of time converting data from one representation to another. For example US dates to European dates or airport codes to city names. In a previous vision paper, we presented the initial design of DataXFormer, a system that uses web resources to assist in transformation discovery. Specifically, DataXFormer discovers possible transformations from web tables and web forms and involves human feedback where appropriate. In this paper, we present the full fledged system along with several extensions. In particular, we present algorithms to find (i) transformations that entail multiple columns of input data, (ii) indirect transformations that are compositions of other transformations, (iii) transformations that are not functions but rather relationships, and (iv) transformations from a knowledge base of public data. We report on experiments with a collection of 120 transformation tasks, and show our enhanced system automatically covers 101 of them by using openly available resources.

AB - In data integration, data curation, and other data analysis tasks, users spend a considerable amount of time converting data from one representation to another. For example US dates to European dates or airport codes to city names. In a previous vision paper, we presented the initial design of DataXFormer, a system that uses web resources to assist in transformation discovery. Specifically, DataXFormer discovers possible transformations from web tables and web forms and involves human feedback where appropriate. In this paper, we present the full fledged system along with several extensions. In particular, we present algorithms to find (i) transformations that entail multiple columns of input data, (ii) indirect transformations that are compositions of other transformations, (iii) transformations that are not functions but rather relationships, and (iv) transformations from a knowledge base of public data. We report on experiments with a collection of 120 transformation tasks, and show our enhanced system automatically covers 101 of them by using openly available resources.

UR - http://www.scopus.com/inward/record.url?scp=84980367317&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84980367317&partnerID=8YFLogxK

U2 - 10.1109/ICDE.2016.7498319

DO - 10.1109/ICDE.2016.7498319

M3 - Conference contribution

AN - SCOPUS:84980367317

SP - 1134

EP - 1145

BT - 2016 IEEE 32nd International Conference on Data Engineering, ICDE 2016

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 16 May 2016 through 20 May 2016

ER -

DataXFormer: A robust transformation discovery system

Abstract

Other

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this