<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Publishing DTD v1.0 20120330//EN" "JATS-journalpublishing1.dtd">
<article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="research-article">
<front>
<journal-meta>
<journal-id journal-id-type="publisher-id">JDS</journal-id>
<journal-title-group><journal-title>Journal of Data Science</journal-title></journal-title-group>
<issn pub-type="epub">1683-8602</issn><issn pub-type="ppub">1680-743X</issn><issn-l>1680-743X</issn-l>
<publisher>
<publisher-name>School of Statistics, Renmin University of China</publisher-name>
</publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="publisher-id">JDS1029</article-id>
<article-id pub-id-type="doi">10.6339/21-JDS1029</article-id>
<article-categories><subj-group subj-group-type="heading">
<subject>Philosophies of Data Science</subject></subj-group></article-categories>
<title-group>
<article-title>So You Developed a Clinical Prediction Model, Now What?</article-title>
</title-group>
<contrib-group>
<contrib contrib-type="author">
<contrib-id contrib-id-type="orcid">https://orcid.org/0000-0003-0679-8730</contrib-id>
<name><surname>Speiser</surname><given-names>Jaime Lynn</given-names></name><email xlink:href="mailto:jspeiser@wakehealth.edu">jspeiser@wakehealth.edu</email><xref ref-type="aff" rid="j_jds1029_aff_001">1</xref><xref ref-type="fn" rid="cor1">∗</xref>
</contrib>
<aff id="j_jds1029_aff_001"><label>1</label>Department of Biostatistics and Data Science, <institution>Wake Forest School of Medicine</institution>, Winston-Salem NC, <country>USA</country></aff>
</contrib-group>
<author-notes>
<corresp id="cor1"><label>∗</label>Email: <ext-link ext-link-type="uri" xlink:href="mailto:jspeiser@wakehealth.edu">jspeiser@wakehealth.edu</ext-link>.</corresp>
</author-notes>
<pub-date pub-type="ppub"><year>2021</year></pub-date><pub-date pub-type="epub"><day>4</day><month>11</month><year>2021</year></pub-date><volume>19</volume><issue>4</issue><fpage>519</fpage><lpage>527</lpage><history><date date-type="received"><day>14</day><month>10</month><year>2021</year></date><date date-type="accepted"><day>25</day><month>10</month><year>2021</year></date></history>
<permissions><copyright-statement>2021 The Author(s). Published by the School of Statistics and the Center for Applied Statistics, Renmin University of China.</copyright-statement><copyright-year>2021</copyright-year>
<license license-type="open-access" xlink:href="https://creativecommons.org/licenses/by/4.0/">
<license-p>Open access article under the <ext-link ext-link-type="uri" xlink:href="https://creativecommons.org/licenses/by/4.0/">CC BY</ext-link> license.</license-p></license></permissions>
<abstract>
<p>A recent trend in medical research is to develop prediction models aiming to improve patient care and health outcomes. While statisticians and data scientists are well-trained in the methods and process of developing a prediction model, their role post-model-development is less clear. This paper covers the critical scientific reasoning step in the prediction pipeline after a model is developed. Working collaboratively with domain experts, statisticians and data scientists should critically evaluate models, carefully implement models into practice, and assess the model’s impact in real world settings. Constructs from implementation science are discussed in the context of prediction modeling. The paper focuses on clinical prediction models, but these ideas apply to other domains as well.</p>
</abstract>
<funding-group><award-group><funding-source xlink:href="https://doi.org/10.13039/100000049">National Institute on Aging</funding-source><award-id>K25AG068253</award-id></award-group><funding-statement>This work was supported by a K25 Career Development Grant from the National Institute on Aging (K25AG068253). The views expressed are those of the author, not of the funding agency. </funding-statement></funding-group>
</article-meta>
</front>
<body/>
<back>
<ref-list id="j_jds1029_reflist_001">
<title>References</title>
<ref id="j_jds1029_ref_001">
<mixed-citation publication-type="journal"> <string-name><surname>Royston</surname> <given-names>P</given-names></string-name>, <string-name><surname>Moons</surname> <given-names>KG</given-names></string-name>, <string-name><surname>Altman</surname> <given-names>DG</given-names></string-name>, <string-name><surname>Vergouwe</surname> <given-names>Y</given-names></string-name> (<year>2009</year>). <article-title>Prognosis and prognostic research: developing a prognostic model</article-title>. <source>BMJ</source>, <volume>338</volume>:<elocation-id>b604</elocation-id>.</mixed-citation>
</ref>
<ref id="j_jds1029_ref_002">
<mixed-citation publication-type="journal"> <string-name><surname>Altman</surname> <given-names>DG</given-names></string-name>, <string-name><surname>Vergouwe</surname> <given-names>Y</given-names></string-name>, <string-name><surname>Royston</surname> <given-names>P</given-names></string-name>, <string-name><surname>Moons</surname> <given-names>KG</given-names></string-name> (<year>2009</year>). <article-title>Prognosis and prognostic research: validating a prognostic model</article-title>. <source>BMJ</source>, <volume>338</volume>:<elocation-id>b605</elocation-id>.</mixed-citation>
</ref>
<ref id="j_jds1029_ref_003">
<mixed-citation publication-type="journal"> <string-name><surname>Speiser</surname> <given-names>JL</given-names></string-name>, <string-name><surname>Callahan</surname> <given-names>KE</given-names></string-name>, <string-name><surname>Ip</surname> <given-names>EH</given-names></string-name>, et al. (<year>2021</year>). <article-title>Predicting future mobility limitation in older adults: A machine learning analysis of health ABC study data</article-title>. <source>The Journals of Gerontology: Series A</source>.</mixed-citation>
</ref>
<ref id="j_jds1029_ref_004">
<mixed-citation publication-type="journal"> <string-name><surname>Speiser</surname> <given-names>JL</given-names></string-name>, <string-name><surname>Lee</surname> <given-names>WM</given-names></string-name>, <string-name><surname>Karvellas</surname> <given-names>CJ</given-names></string-name>, <string-name><surname>Group</surname> <given-names>USALFS</given-names></string-name> (<year>2015</year>). <article-title>Predicting outcome on admission and post-admission for acetaminophen-induced acute liver failure using classification and regression tree models</article-title>. <source>PLoS One</source>, <volume>10</volume>(<issue>4</issue>): <elocation-id>e0122929</elocation-id>.</mixed-citation>
</ref>
<ref id="j_jds1029_ref_005">
<mixed-citation publication-type="journal"> <string-name><surname>Moons</surname> <given-names>KG</given-names></string-name>, <string-name><surname>Kengne</surname> <given-names>AP</given-names></string-name>, <string-name><surname>Grobbee</surname> <given-names>DE</given-names></string-name>, <etal>et al.</etal> (<year>2012</year>). <article-title>Risk prediction models: II. External validation, model updating, and impact assessment</article-title>. <source>Heart</source>, <volume>98</volume>(<issue>9</issue>): <fpage>691</fpage>–<lpage>698</lpage>.</mixed-citation>
</ref>
<ref id="j_jds1029_ref_006">
<mixed-citation publication-type="journal"> <string-name><surname>Salazar de Pablo</surname> <given-names>G</given-names></string-name>, <string-name><surname>Studerus</surname> <given-names>E</given-names></string-name>, <string-name><surname>Vaquerizo-Serrano</surname> <given-names>J</given-names></string-name>, <etal>et al.</etal> (<year>2021</year>). <article-title>Implementing precision psychiatry: A systematic review of individualized prediction models for clinical practice</article-title>. <source>Schizophrenia Bulletin</source>, <volume>47</volume>(<issue>2</issue>): <fpage>284</fpage>–<lpage>297</lpage>.</mixed-citation>
</ref>
<ref id="j_jds1029_ref_007">
<mixed-citation publication-type="journal"> <string-name><surname>Kansagara</surname> <given-names>D</given-names></string-name>, <string-name><surname>Englander</surname> <given-names>H</given-names></string-name>, <string-name><surname>Salanitro</surname> <given-names>A</given-names></string-name>, <etal>et al.</etal> (<year>2011</year>). <article-title>Risk prediction models for hospital readmission: A systematic review</article-title>. <source>JAMA</source>, <volume>306</volume>(<issue>15</issue>): <fpage>1688</fpage>–<lpage>1698</lpage>.</mixed-citation>
</ref>
<ref id="j_jds1029_ref_008">
<mixed-citation publication-type="journal"> <string-name><surname>Belsher</surname> <given-names>BE</given-names></string-name>, <string-name><surname>Smolenski</surname> <given-names>DJ</given-names></string-name>, <string-name><surname>Pruitt</surname> <given-names>LD</given-names></string-name>, <etal>et al.</etal> (<year>2019</year>). <article-title>Prediction models for suicide attempts and deaths: A systematic review and simulation</article-title>. <source>JAMA Psychiatry</source>, <volume>76</volume>(<issue>6</issue>): <fpage>642</fpage>–<lpage>651</lpage>.</mixed-citation>
</ref>
<ref id="j_jds1029_ref_009">
<mixed-citation publication-type="journal"> <string-name><surname>Glasgow</surname> <given-names>RE</given-names></string-name>, <string-name><surname>Vinson</surname> <given-names>C</given-names></string-name>, <string-name><surname>Chambers</surname> <given-names>D</given-names></string-name>, <string-name><surname>Khoury</surname> <given-names>MJ</given-names></string-name>, <string-name><surname>Kaplan</surname> <given-names>RM</given-names></string-name>, <string-name><surname>National</surname> <given-names>HC</given-names></string-name> (<year>2012</year>). <article-title>Institutes of Health approaches to dissemination and implementation science: Current and future directions</article-title>. <source>American Journal of Public Health</source>, <volume>102</volume>(<issue>7</issue>): <fpage>1274</fpage>–<lpage>1281</lpage>.</mixed-citation>
</ref>
<ref id="j_jds1029_ref_010">
<mixed-citation publication-type="journal"> <string-name><surname>Rapport</surname> <given-names>F</given-names></string-name>, <string-name><surname>Clay-Williams</surname> <given-names>R</given-names></string-name>, <string-name><surname>Churruca</surname> <given-names>K</given-names></string-name>, <string-name><surname>Shih</surname> <given-names>P</given-names></string-name>, <string-name><surname>Hogden</surname> <given-names>A</given-names></string-name>, <string-name><surname>Braithwaite</surname> <given-names>J</given-names></string-name> (<year>2018</year>). <article-title>The struggle of translating science into action: Foundational concepts of implementation science</article-title>. <source>Journal of Evaluation in Clinical Practice</source>, <volume>24</volume>(<issue>1</issue>): <fpage>117</fpage>–<lpage>126</lpage>.</mixed-citation>
</ref>
<ref id="j_jds1029_ref_011">
<mixed-citation publication-type="journal"> <string-name><surname>Bauer</surname> <given-names>MS</given-names></string-name>, <string-name><surname>Kirchner</surname> <given-names>J</given-names></string-name> (<year>2020</year>). <article-title>Implementation science: What is it and why should I care?</article-title> <source>Psychiatry Research</source>, <volume>283</volume>: <fpage>112376</fpage>.</mixed-citation>
</ref>
<ref id="j_jds1029_ref_012">
<mixed-citation publication-type="journal"> <string-name><surname>Kappen</surname> <given-names>TH</given-names></string-name>, <string-name><surname>van Klei</surname> <given-names>WA</given-names></string-name>, <string-name><surname>van Wolfswinkel</surname> <given-names>L</given-names></string-name>, <string-name><surname>Kalkman</surname> <given-names>CJ</given-names></string-name>, <string-name><surname>Vergouwe</surname> <given-names>Y</given-names></string-name>, <string-name><surname>Moons</surname> <given-names>KG</given-names></string-name> (<year>2018</year>). <article-title>Evaluating the impact of prediction models: Lessons learned, challenges, and recommendations</article-title>. <source>Diagnostic and Prognostic Research</source>, <volume>2</volume>(<issue>1</issue>): <fpage>1</fpage>–<lpage>11</lpage>.</mixed-citation>
</ref>
<ref id="j_jds1029_ref_013">
<mixed-citation publication-type="journal"> <string-name><surname>Knaus</surname> <given-names>WA</given-names></string-name>, <string-name><surname>Draper</surname> <given-names>EA</given-names></string-name>, <string-name><surname>Wagner</surname> <given-names>DP</given-names></string-name>, <string-name><surname>Zimmerman</surname> <given-names>JE</given-names></string-name> (<year>1985</year>). <article-title>APACHE II: A severity of disease classification system</article-title>. <source>Critical Care Medicine</source>, <volume>13</volume>(<issue>10</issue>): <fpage>818</fpage>–<lpage>829</lpage>.</mixed-citation>
</ref>
<ref id="j_jds1029_ref_014">
<mixed-citation publication-type="journal"> <string-name><surname>Kompa</surname> <given-names>B</given-names></string-name>, <string-name><surname>Snoek</surname> <given-names>J</given-names></string-name>, <string-name><surname>Beam</surname> <given-names>AL</given-names></string-name> (<year>2021</year>). <article-title>Second opinion needed: Communicating uncertainty in medical machine learning</article-title>. <source>NPJ Digital Medicine</source>, <volume>4</volume>(<issue>1</issue>): <fpage>1</fpage>–<lpage>6</lpage>.</mixed-citation>
</ref>
<ref id="j_jds1029_ref_015">
<mixed-citation publication-type="journal"> <string-name><surname>Moons</surname> <given-names>KG</given-names></string-name>, <string-name><surname>Altman</surname> <given-names>DG</given-names></string-name>, <string-name><surname>Vergouwe</surname> <given-names>Y</given-names></string-name>, <string-name><surname>Royston</surname> <given-names>P</given-names></string-name> (<year>2009</year>). <article-title>Prognosis and prognostic research: Application and impact of prognostic models in clinical practice</article-title>. <source>BMJ</source>, <volume>338</volume>: <elocation-id>b606</elocation-id>.</mixed-citation>
</ref>
</ref-list>
</back>
</article>
