# Data

## Datasets

We use two datasets to test the cluster naming process.

- **NSF award abstracts from 2022** 
    - All NSF awards from 2022 (nearly 10k awards)
    - Downloaded from https://nsf.gov/awardsearch/download.jsp
- **PMC-Patients dataset**
    - 167k patient notes from case reports in PubMed Central by recognizing certain section patterns
    - [Paper](https://arxiv.org/ftp/arxiv/papers/2202/2202.13876.pdf), [Code](https://github.com/zhao-zy15/PMC-Patients), [Data](https://drive.google.com/file/d/1vFCLy_CF8fxPDZvDtHPR6Dl6x9l0TyvW/view?usp=sharing)
    - Authors released the data publicly under Creative Commons BY-NC-SA license. This gives us freedom to copy and redistribute the material in any medium or format. 
    - Access to data on Google Drive does require sending an access request to the authors

## Data Processing

The processed datasets for use in clustering and cluster naming are `data/NSF/nsf_2022.parquet` and `data/PMC-Patients/pmc_patients.parquet`.

Raw data is stored in the `data/NSF` and `data/PMC-Patients` directories.

Data was processed using `data/process_nsf.py` and `data/process_pmc.py`. These can be rerun to regenerate processed data from raw data as necessary.

The NSF dataset included some duplicates and near duplicates. To remove these, we added an additional processing step in `data/dedupe_nsf.pf`.

## Getting the data using Git LFS

Some of the data files are too big to store on GitHub, so we use Git LFS.

When you clone this branch locally, the files stored on Git LFS will be pointers rather than the actual data files.

If you don't have Git LFS installed on your user account already, first follow the instructions to download and install it [here](https://git-lfs.com/).

To get the data locally, run `git lfs pull`.
