Journal of Data Science

Advances in AI and automation are reshaping qualitative research workflows, making processes more efficient, accurate, consistent, and scalable. This paper presents innovations developed for the Illinois Needs Assessment project, a statewide initiative led by the Illinois State Board of Education and the American Institutes for Research to conduct comprehensive needs assessments for schools that need intensive or comprehensive support. To address the scale and tight timeline requirements of the project, the team designed three interconnected pipelines that work together to produce a finalized report. The first, an Audio Pipeline, uses Whisper and generative AI to automate transcription, text-based speaker role attribution, thematic coding, and insight generation from focus groups and interviews. The second, a Report Generation Pipeline, integrates Airtable automations with AWS infrastructure to produce customized school reports that merge AI-generated findings with survey data, school performance metrics, and contextual comparisons. Third, the Needs Assessment Summary Report automates the assembly of all quantitative and qualitative inputs into a polished, customizable deliverable that combines efficiency with expert review. Together, these pipelines replace ad hoc manual workflows with reproducible, consistent systems that enhance data quality, reduce error, and broaden access for non-technical users. The integrated design demonstrates how automation and generative AI can reduce manual burdens, shorten delivery timelines, and support timely, data-informed, and human-centered decision-making in education.

The Data Science Consulting Program at North Carolina State University Libraries, in partnership with the Data Science and AI Academy, provides comprehensive support for a wide range of tools and software, including R, Python, MATLAB, ArcGIS, and more, to assist students, faculty, and staff with their data-related needs. This paper explores the integration of generative AI, specifically ChatGPT, into our consultation services, demonstrating how it enhances the efficiency and effectiveness of addressing numerous and diverse requests. ChatGPT has been instrumental in tasks such as data visualization, statistical analysis, and code generation, allowing consultants to quickly resolve complex queries. The paper also discusses the program’s structured approach to consultations, highlighting the iterative process from initial request to resolution. We address challenges like prompt engineering and response variability, offering best practices to maximize the tool’s potential. As AI technology continues to evolve, its role in our data science consultations is expected to expand, improving service quality and the consultant’s ability to handle increasingly complex tasks. The study concludes that ChatGPT is a valuable asset in academic data science, significantly streamlining workflows and broadening the scope of support provided by our program.

Many believe that use of generative AI as a private tutor has the potential to shrink access and achievement gaps between students and schools with abundant resources versus those with fewer resources. Shrinking the gap is possible only if paid and free versions of the platforms perform with the same accuracy. In this experiment, we investigate the performance of GPT versions 3.5, 4.0, and 4o-mini on the same 16-question statistics exam given to a class of first-year graduate students. While we do not advocate using any generative AI platform to complete an exam, the use of exam questions allows us to explore aspects of ChatGPT’s responses to typical questions that students might encounter in a statistics course. Results on accuracy indicate that GPT 3.5 would fail the exam, GPT4 would perform well, and GPT4o-mini would perform somewhere in between. While we acknowledge the existence of other Generative AI/LLMs, our discussion concerns only ChatGPT because it is the most widely used platform on college campuses at this time. We further investigate differences among the AI platforms in the answers for each problem using methods developed for text analytics, such as reading level evaluation and topic modeling. Results indicate that GPT3.5 and 4o-mini have characteristics that are more similar than either of them have with GPT4.