# Evaluation of Text Cluster Naming with Generative Large Language Models

This repo contains code for the paper "Evaluation of Text Cluster Naming with Generative Large Language Models" by Alexander Preiss, Caren Arbeit, Anthony Berghammer, John Bollenbacher, John McCarthy, Madeline Brom, Mike Enger, Nicholas Rios Villacorta, and Shaquavia Straughn. 

## Replication
Here we briefly document instructions for how to use this code to reproduce our results. 

### A word of caution
Many of the steps in this study are difficult to replicate. The clustering, cluster sampling, and cluster naming processes include an element of randomness, so clusters and cluster names may not be identical to those described in the manuscript. The manual processes - naming clusters and evaluating cluster names against a rubric - are subjective, so different annotators may produce different results. 

The results of most intermediate steps are documented in the outputs of jupyter notebooks and Excel spreadsheets. A "lightweight" approach to replication might review these intermediate outputs for accuracy, then simply rerun the final analysis step. 

Steps for a more thorough replication are outlined below.

### Replication steps

1. Begin by downloading the PMC and NSF datasets and place the data and metadata in the correct folders (instructions in respective folders in [./data](data)).
1. Create a virtual Python 3.9 environment and install the dependencies in `requirements.txt`.
1. To produce clean, aggregated data files for use in the rest of the analysis pipeline, run the data preprocessing scripts (`./data/process_nsf.py`, `./data/dedupe_nsf.py`, `./data/process_pmc.py`, in this order). These will produce `./data/NSF/nsf_2022.parquet`, `./data/NSF/nsf_2022_deduped.parquet`, and `./data/PMC-Patients/pmc_patients.parquet`, respectively.
1. To produce text clusters for both datasets, run `./clustering/cluster.ipynb`. This will produce cluster tags for each text and cluster metadata, including total number of texts per cluser and represenative keywords for each cluster. The cluster tags for each text are contained in output files `./clustering/nsf_2022_clustered.parquet` and `./clustering/pmc_patients_clustered.parquet`, while the cluster metadata is in `./clustering/nsf_2022_cluster_labels.parquet` and `./clustering/pmc_patients_cluster_labels.parquet`.
1. To sample 50 clusters from each dataset for further analysis, run `./clustering/sample.ipynb`.
1. To produce cluster names by multiple methods, run each the notebook for each.
   - Document based cluster naming
        - Code: `./naming/long_doc_based.ipynb`
        - Outputs: `./output/nsf_long_doc_based_names.parquet`, `./output/pmc_long_doc_based_names.parquet`

    - Keyword based cluster naming
        - Code: `./naming/JMB_keyword_naming.ipynb`
        - Outputs: `./output/nsf_keyword_names.parquet`, `./output/pmc_keyword_names.parquet`,
   
    - Chained resampling cluster naming
        - Code: `./naming/calare/ab_calare_addProbs.ipynb`
        - Outputs: `./outputs/nsf_resample_names.csv`, `./outputs/nsf_resample_names.csv`
   
    - Manual naming: run `./evaluation/make_input_excels.ipynb`, and then have annotators name clusters in the resulting Excel spreadsheets, per instructions in the paper and annotation guide.
   

1. To produce the Excel spreadsheets used by human labelers to manually evaluate cluster names, run `./evaluation/make_eval_excels.ipynb`.
1. Have annotators evaluate the names produced by each naming method, using the evaluation methods documented in the paper and annotation guide. Deposit resulting, completed spreadsheets back in `./evaluation`.
1. To produce paper figures and analysis results, run `./analysis/analyze_eval_results.ipynb`. This file contains detailed documentation of each analytic step and previews of intermediate outputs; most of this methodological information is also contained in the paper. If you want to rerun the sensitivity analysis too, run `./analysis/sensitivity_analysis.ipynb`.








