<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Publishing DTD v1.0 20120330//EN" "JATS-journalpublishing1.dtd">
<article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="research-article">
  <front>
    <journal-meta>
      <journal-id journal-id-type="publisher-id">JDS</journal-id>
      <journal-title-group>
        <journal-title>Journal of Data Science</journal-title>
      </journal-title-group>
      <issn pub-type="epub">1680-743X</issn>
      <issn pub-type="ppub">1680-743X</issn>
      <publisher>
        <publisher-name>SOSRUC</publisher-name>
      </publisher>
    </journal-meta>
    <article-meta>
      <article-id pub-id-type="publisher-id">030105</article-id>
      <article-id pub-id-type="doi">10.6339/JDS.2005.03(1).181
</article-id>
      <article-categories>
        <subj-group subj-group-type="heading">
          <subject>Research Article</subject>
        </subj-group>
      </article-categories>
      <title-group>
        <article-title>Observer Variability: A New Approach in Evaluating Interobserver Agreement</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <name>
            <surname>Haber</surname>
            <given-names>Michael</given-names>
          </name>
          <xref ref-type="aff" rid="j_JDS_aff_000"/>
        </contrib>
        <aff id="j_JDS_aff_000">Emory University</aff>
        <contrib contrib-type="author">
          <name>
            <surname>Barnhart</surname>
            <given-names>Huiman X.</given-names>
          </name>
          <xref ref-type="aff" rid="j_JDS_aff_001"/>
        </contrib>
        <aff id="j_JDS_aff_001">Duke University</aff>
        <contrib contrib-type="author">
          <name>
            <surname>Song</surname>
            <given-names>Jingli</given-names>
          </name>
          <xref ref-type="aff" rid="j_JDS_aff_002"/>
        </contrib>
        <aff id="j_JDS_aff_002">Eli Lilly and Company</aff>
        <contrib contrib-type="author">
          <name>
            <surname>Gruden</surname>
            <given-names>James</given-names>
          </name>
          <xref ref-type="aff" rid="j_JDS_aff_003"/>
        </contrib>
        <aff id="j_JDS_aff_003">Emory University</aff>
      </contrib-group>
      <volume>3</volume>
      <issue>1</issue>
      <fpage>69</fpage>
      <lpage>83</lpage>
      <permissions>
        <ali:free_to_read xmlns:ali="http://www.niso.org/schemas/ali/1.0/"/>
      </permissions>
      <abstract>
        <p>Abstract: Existing indices of observer agreement for continuous data, such as the intraclass correlation coefficient or the concordance correlation coefficient, measure the total observer-related variability, which includes the variabilities between and within observers. This work introduces a new index that measures the interobserver variability, which is defined in terms of the distances among the ‘true values’ assigned by different observers on the same subject. The new coefficient of interobserver variability (CIV ) is defined as the ratio of the interobserver and the total observer variability. We show how to estimate the CIV and how to use bootstrap and ANOVAbased methods for inference. We also develop a coefficient of excess observer variability, which compares the total observer variability to the expected total observer variability when there are no differences among the observers. This coefficient is a simple function of the CIV . In addition, we show how the value of the CIV , estimated from an agreement study, can be used in the design of measurements studies. We illustrate the new concepts and methods by two examples, where (1) two radiologists used calcium scores to evaluate the severity of coronary artery arteriosclerosis, and (2) two methods were used to measure knee joint angle.</p>
      </abstract>
      <kwd-group>
        <label>Keywords</label>
        <kwd>Agreement studies</kwd>
        <kwd>ANOVA</kwd>
      </kwd-group>
    </article-meta>
  </front>
</article>
