Performance comparison of machine learning platforms

Asim Roy, Shiban Qureshi, Kartikeya Pande, Divitha Nair, Kartik Gairola, Pooja Jain, Suraj Singh, Kirti Sharma, Akshay Jagadale, Yi Yang Lin, Shashank Sharma, Ramya Gotety, Yuexin Zhang, Ji Tang, Tejas Mehta, Hemanth Sindhanuru, Nonso Okafor, Santak Das, Chidambara N. Gopal, Srinivasa B. RudrarajuAvinash V. Kakarlapudi

Research output: Contribution to journalArticle

1 Scopus citations

Abstract

In this paper, we present a method for comparing and evaluating different collections of machine learning algorithms on the basis of a given performance measure (e.g., accuracy, area under the curve (AUC), F-score). Such a method can be used to compare standard machine learning platforms such as SAS, IBM SPSS, and Microsoft Azure ML. A recent trend in automation of machine learning is to exercise a collection of machine learning algorithms on a particular problem and then use the best performing algorithm. Thus, the proposed method can also be used to compare and evaluate different collections of algorithms for automation on a certain problem type and find the best collection. In the study reported here, we applied the method to compare six machine learning platforms – R, Python, SAS, IBM SPSS Modeler, Microsoft Azure ML, and Apache Spark ML. We compared the platforms on the basis of predictive performance on classification problems because a significant majority of the problems in machine learning are of that type. The general question that we addressed is the following: Are there platforms that are superior to others on some particular performance measure? For each platform, we used a collection of six classification algorithms from the following six families of algorithms – support vector machines, multilayer perceptrons, random forest (or variant), decision trees/gradient boosted trees, Naive Bayes/Bayesian networks, and logistic regression. We compared their performance on the basis of classification accuracy, F-score, and AUC. We used F-score and AUC measures to compare platforms on two-class problems only. For testing the platforms, we used a mix of data sets from (1) the University of California, Irvine (UCI) library, (2) the Kaggle competition library, and (3) high-dimensional gene expression problems. We performed some hyperparameter tuning on algorithms wherever possible.

Original languageEnglish (US)
Pages (from-to)207-225
Number of pages19
JournalINFORMS Journal on Computing
Volume31
Issue number2
DOIs
StatePublished - Jan 1 2019

Keywords

  • Classification algorithms
  • Comparison of algorithms
  • Comparison of platforms
  • Machine learning platforms

ASJC Scopus subject areas

  • Software
  • Information Systems
  • Computer Science Applications
  • Management Science and Operations Research

Fingerprint Dive into the research topics of 'Performance comparison of machine learning platforms'. Together they form a unique fingerprint.

  • Cite this

    Roy, A., Qureshi, S., Pande, K., Nair, D., Gairola, K., Jain, P., Singh, S., Sharma, K., Jagadale, A., Lin, Y. Y., Sharma, S., Gotety, R., Zhang, Y., Tang, J., Mehta, T., Sindhanuru, H., Okafor, N., Das, S., Gopal, C. N., ... Kakarlapudi, A. V. (2019). Performance comparison of machine learning platforms. INFORMS Journal on Computing, 31(2), 207-225. https://doi.org/10.1287/ijoc.2018.0825