Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge

Catherine L. Lawson; Andriy Kryshtafovych; Paul D. Adams; Pavel V. Afonine; Matthew L. Baker; Benjamin A. Barad; Paul Bond; Tom Burnley; Renzhi Cao; Jianlin Cheng; Grzegorz Chojnowski; Kevin Cowtan; Ken A. Dill; Frank DiMaio; Daniel P. Farrell; James S. Fraser; Mark A. Herzik; Soon Wen Hoh; Jie Hou; Li Wei Hung; Maxim Igaev; Agnel P. Joseph; Daisuke Kihara; Dilip Kumar; Sumit Mittal; Bohdan Monastyrskyy; Mateusz Olek; Colin M. Palmer; Ardan Patwardhan; Alberto Perez; Jonas Pfab; Grigore D. Pintilie; Jane S. Richardson; Peter B. Rosenthal; Daipayan Sarkar; Luisa U. Schäfer; Michael F. Schmid; Gunnar F. Schröder; Mrinal Shekhar; Dong Si; Abishek Singharoy; Genki Terashi; Thomas C. Terwilliger; Andrea Vaiana; Liguo Wang; Zhe Wang; Stephanie A. Wankowicz; Christopher J. Williams; Martyn Winn; Tianqi Wu; Xiaodi Yu; Kaiming Zhang; Helen M. Berman; Wah Chiu

doi:10.1038/s41592-020-01051-w

Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge

Catherine L. Lawson, Andriy Kryshtafovych, Paul D. Adams, Pavel V. Afonine, Matthew L. Baker, Benjamin A. Barad, Paul Bond, Tom Burnley, Renzhi Cao, Jianlin Cheng, Grzegorz Chojnowski, Kevin Cowtan, Ken A. Dill, Frank DiMaio, Daniel P. Farrell, James S. Fraser, Mark A. Herzik, Soon Wen Hoh, Jie Hou, Li Wei HungMaxim Igaev, Agnel P. Joseph, Daisuke Kihara, Dilip Kumar, Sumit Mittal, Bohdan Monastyrskyy, Mateusz Olek, Colin M. Palmer, Ardan Patwardhan, Alberto Perez, Jonas Pfab, Grigore D. Pintilie, Jane S. Richardson, Peter B. Rosenthal, Daipayan Sarkar, Luisa U. Schäfer, Michael F. Schmid, Gunnar F. Schröder, Mrinal Shekhar, Dong Si, Abishek Singharoy, Genki Terashi, Thomas C. Terwilliger, Andrea Vaiana, Liguo Wang, Zhe Wang, Stephanie A. Wankowicz, Christopher J. Williams, Martyn Winn, Tianqi Wu, Xiaodi Yu, Kaiming Zhang, Helen M. Berman, Wah Chiu

Molecular Sciences, School of (SMS)

Research output: Contribution to journal › Article › peer-review

56 Scopus citations

Abstract

This paper describes outcomes of the 2019 Cryo-EM Model Challenge. The goals were to (1) assess the quality of models that can be produced from cryogenic electron microscopy (cryo-EM) maps using current modeling software, (2) evaluate reproducibility of modeling results from different software developers and users and (3) compare performance of current metrics used for model evaluation, particularly Fit-to-Map metrics, with focus on near-atomic resolution. Our findings demonstrate the relatively high accuracy and reproducibility of cryo-EM models derived by 13 participating teams from four benchmark maps, including three forming a resolution series (1.8 to 3.1 Å). The results permit specific recommendations to be made about validating near-atomic cryo-EM structures both in the context of individual experiments and structure data archives such as the Protein Data Bank. We recommend the adoption of multiple scoring parameters to provide full and objective annotation and assessment of the model, reflective of the observed cryo-EM map density.

Original language	English (US)
Pages (from-to)	156-164
Number of pages	9
Journal	Nature Methods
Volume	18
Issue number	2
DOIs	https://doi.org/10.1038/s41592-020-01051-w
State	Published - Feb 2021

ASJC Scopus subject areas

Biotechnology
Biochemistry
Molecular Biology
Cell Biology

Access to Document

10.1038/s41592-020-01051-w

Cite this

Lawson, C. L., Kryshtafovych, A., Adams, P. D., Afonine, P. V., Baker, M. L., Barad, B. A., Bond, P., Burnley, T., Cao, R., Cheng, J., Chojnowski, G., Cowtan, K., Dill, K. A., DiMaio, F., Farrell, D. P., Fraser, J. S., Herzik, M. A., Hoh, S. W., Hou, J., ... Chiu, W. (2021). Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge. Nature Methods, 18(2), 156-164. https://doi.org/10.1038/s41592-020-01051-w

Lawson, CL, Kryshtafovych, A, Adams, PD, Afonine, PV, Baker, ML, Barad, BA, Bond, P, Burnley, T, Cao, R, Cheng, J, Chojnowski, G, Cowtan, K, Dill, KA, DiMaio, F, Farrell, DP, Fraser, JS, Herzik, MA, Hoh, SW, Hou, J, Hung, LW, Igaev, M, Joseph, AP, Kihara, D, Kumar, D, Mittal, S, Monastyrskyy, B, Olek, M, Palmer, CM, Patwardhan, A, Perez, A, Pfab, J, Pintilie, GD, Richardson, JS, Rosenthal, PB, Sarkar, D, Schäfer, LU, Schmid, MF, Schröder, GF, Shekhar, M, Si, D, Singharoy, A, Terashi, G, Terwilliger, TC, Vaiana, A, Wang, L, Wang, Z, Wankowicz, SA, Williams, CJ, Winn, M, Wu, T, Yu, X, Zhang, K, Berman, HM & Chiu, W 2021, 'Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge', Nature Methods, vol. 18, no. 2, pp. 156-164. https://doi.org/10.1038/s41592-020-01051-w

@article{4942b7526767407b9edead389dc1cbfa,

title = "Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge",

abstract = "This paper describes outcomes of the 2019 Cryo-EM Model Challenge. The goals were to (1) assess the quality of models that can be produced from cryogenic electron microscopy (cryo-EM) maps using current modeling software, (2) evaluate reproducibility of modeling results from different software developers and users and (3) compare performance of current metrics used for model evaluation, particularly Fit-to-Map metrics, with focus on near-atomic resolution. Our findings demonstrate the relatively high accuracy and reproducibility of cryo-EM models derived by 13 participating teams from four benchmark maps, including three forming a resolution series (1.8 to 3.1 {\AA}). The results permit specific recommendations to be made about validating near-atomic cryo-EM structures both in the context of individual experiments and structure data archives such as the Protein Data Bank. We recommend the adoption of multiple scoring parameters to provide full and objective annotation and assessment of the model, reflective of the observed cryo-EM map density.",

author = "Lawson, {Catherine L.} and Andriy Kryshtafovych and Adams, {Paul D.} and Afonine, {Pavel V.} and Baker, {Matthew L.} and Barad, {Benjamin A.} and Paul Bond and Tom Burnley and Renzhi Cao and Jianlin Cheng and Grzegorz Chojnowski and Kevin Cowtan and Dill, {Ken A.} and Frank DiMaio and Farrell, {Daniel P.} and Fraser, {James S.} and Herzik, {Mark A.} and Hoh, {Soon Wen} and Jie Hou and Hung, {Li Wei} and Maxim Igaev and Joseph, {Agnel P.} and Daisuke Kihara and Dilip Kumar and Sumit Mittal and Bohdan Monastyrskyy and Mateusz Olek and Palmer, {Colin M.} and Ardan Patwardhan and Alberto Perez and Jonas Pfab and Pintilie, {Grigore D.} and Richardson, {Jane S.} and Rosenthal, {Peter B.} and Daipayan Sarkar and Sch{\"a}fer, {Luisa U.} and Schmid, {Michael F.} and Schr{\"o}der, {Gunnar F.} and Mrinal Shekhar and Dong Si and Abishek Singharoy and Genki Terashi and Terwilliger, {Thomas C.} and Andrea Vaiana and Liguo Wang and Zhe Wang and Wankowicz, {Stephanie A.} and Williams, {Christopher J.} and Martyn Winn and Tianqi Wu and Xiaodi Yu and Kaiming Zhang and Berman, {Helen M.} and Wah Chiu",

note = "Publisher Copyright: {\textcopyright} 2021, The Author(s).",

year = "2021",

month = feb,

doi = "10.1038/s41592-020-01051-w",

language = "English (US)",

volume = "18",

pages = "156--164",

journal = "Nature Methods",

issn = "1548-7091",

publisher = "Nature Publishing Group",

number = "2",

}

TY - JOUR

T1 - Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge

AU - Lawson, Catherine L.

AU - Kryshtafovych, Andriy

AU - Adams, Paul D.

AU - Afonine, Pavel V.

AU - Baker, Matthew L.

AU - Barad, Benjamin A.

AU - Bond, Paul

AU - Burnley, Tom

AU - Cao, Renzhi

AU - Cheng, Jianlin

AU - Chojnowski, Grzegorz

AU - Cowtan, Kevin

AU - Dill, Ken A.

AU - DiMaio, Frank

AU - Farrell, Daniel P.

AU - Fraser, James S.

AU - Herzik, Mark A.

AU - Hoh, Soon Wen

AU - Hou, Jie

AU - Hung, Li Wei

AU - Igaev, Maxim

AU - Joseph, Agnel P.

AU - Kihara, Daisuke

AU - Kumar, Dilip

AU - Mittal, Sumit

AU - Monastyrskyy, Bohdan

AU - Olek, Mateusz

AU - Palmer, Colin M.

AU - Patwardhan, Ardan

AU - Perez, Alberto

AU - Pfab, Jonas

AU - Pintilie, Grigore D.

AU - Richardson, Jane S.

AU - Rosenthal, Peter B.

AU - Sarkar, Daipayan

AU - Schäfer, Luisa U.

AU - Schmid, Michael F.

AU - Schröder, Gunnar F.

AU - Shekhar, Mrinal

AU - Si, Dong

AU - Singharoy, Abishek

AU - Terashi, Genki

AU - Terwilliger, Thomas C.

AU - Vaiana, Andrea

AU - Wang, Liguo

AU - Wang, Zhe

AU - Wankowicz, Stephanie A.

AU - Williams, Christopher J.

AU - Winn, Martyn

AU - Wu, Tianqi

AU - Yu, Xiaodi

AU - Zhang, Kaiming

AU - Berman, Helen M.

AU - Chiu, Wah

PY - 2021/2

Y1 - 2021/2

N2 - This paper describes outcomes of the 2019 Cryo-EM Model Challenge. The goals were to (1) assess the quality of models that can be produced from cryogenic electron microscopy (cryo-EM) maps using current modeling software, (2) evaluate reproducibility of modeling results from different software developers and users and (3) compare performance of current metrics used for model evaluation, particularly Fit-to-Map metrics, with focus on near-atomic resolution. Our findings demonstrate the relatively high accuracy and reproducibility of cryo-EM models derived by 13 participating teams from four benchmark maps, including three forming a resolution series (1.8 to 3.1 Å). The results permit specific recommendations to be made about validating near-atomic cryo-EM structures both in the context of individual experiments and structure data archives such as the Protein Data Bank. We recommend the adoption of multiple scoring parameters to provide full and objective annotation and assessment of the model, reflective of the observed cryo-EM map density.

AB - This paper describes outcomes of the 2019 Cryo-EM Model Challenge. The goals were to (1) assess the quality of models that can be produced from cryogenic electron microscopy (cryo-EM) maps using current modeling software, (2) evaluate reproducibility of modeling results from different software developers and users and (3) compare performance of current metrics used for model evaluation, particularly Fit-to-Map metrics, with focus on near-atomic resolution. Our findings demonstrate the relatively high accuracy and reproducibility of cryo-EM models derived by 13 participating teams from four benchmark maps, including three forming a resolution series (1.8 to 3.1 Å). The results permit specific recommendations to be made about validating near-atomic cryo-EM structures both in the context of individual experiments and structure data archives such as the Protein Data Bank. We recommend the adoption of multiple scoring parameters to provide full and objective annotation and assessment of the model, reflective of the observed cryo-EM map density.

UR - http://www.scopus.com/inward/record.url?scp=85101049839&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85101049839&partnerID=8YFLogxK

U2 - 10.1038/s41592-020-01051-w

DO - 10.1038/s41592-020-01051-w

M3 - Article

C2 - 33542514

AN - SCOPUS:85101049839

SN - 1548-7091

VL - 18

SP - 156

EP - 164

JO - Nature Methods

JF - Nature Methods

IS - 2

ER -

Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge

Abstract

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this