# Clustering

Clustering is done in `clustering/cluster.ipynb`. The notebook contains notes on the BERTopic hyperparameters chosen, additional preprocessing steps, etc. 

Note that documents with outlier lengths are not included in the clustered datasets. This is intended to improve cluster quality. It implies that not all docs in the raw datasets are clustered. All cluster naming analyses should start from the clustered datasets.

## Files and contents

For each dataset, there are two files.

- `x_clustered.parquet` contains one row per document. 
    - Each document's cluster is indicated.
    - **Representative documents** are documents which BERTopic thinks represent the overall cluster well. They are identified by comparing CTFIDF representations of a sample of documents from the cluster to the cluster's CTFIDF representation. The documents which are closest to the cluster's representation are chosen. By default, only three docs are chosen per cluster. Note that by default, 500 docs are sampled from each cluster, since this is computationally expensive. Most of our clusters are smaller than that, but there could be some randomness to the chosen docs in larger clusters.
    - This file will be useful for cluster naming using cluster full texts/summaries.
- `x_cluster_labels.parquet` contains one row per cluster. 
    - Each cluster's keywords are included as a comma-separated column
    - This file will be useful for cluster naming using cluster keywords.

## Sampling

We will evaluate a sample of 50 cluster names from each dataset to save time and money. Sampling is done after clustering is complete in `clustering/sample.ipynb`. This notebook adds a `Sampled` column to the cluster label datasets, where 1 indicates that the cluster was selected for evaluation. The document-level dataset is not filtered to the sampled clusters, nor does it indicate whether a document is from a sampled cluster. To identify documents from sampled clusters, you'll need to use the `...cluster_labels` dataset, e.g., join the cluster-level table to the document-level table and filter.

## Getting the data using Git LFS

As in the `data/` directory, we use Git LFS to store the clustered data files. 

When you pull the contents of `cluster/` locally, the files stored on Git LFS will be pointers rather than the actual data files.

If you don't have Git LFS installed on your user account already, first follow the instructions to download and install it [here](https://git-lfs.com/).

To get the data locally, run `git lfs pull`.
