Contact Information

Contact Information

PhD Candidate, Demographer, Sociologist.

Coding Open-Ended Survey Text Without Leaving Stata: the catllm Package for LLM-Assisted Content Coding

Date:

Open-ended survey items capture what closed-ended items cannot, and are routinely dropped from analysis because hand-coding them does not scale. catllm puts the whole coding loop inside Stata: extract/explore discover categories directly from the data, classify applies a category scheme and returns ordinary Stata indicator variables ready for tabulate or regress. It supports OpenAI, Anthropic, and Google models as well as local open-weight models through Ollama, so restricted-use data need never leave the analyst’s machine.

Validated against human coders on multi-label, multi-class survey free text, proprietary models agree with human annotators on 97% of straightforward items and 88–91% of complex interpretive ones, with open-weight models 1–2 points behind; a three-model ensemble reaches 98% agreement with human consensus. Models tend to over-classify; writing categories as descriptive sentences rather than one-word labels helps on every model tested, and unanimous voting across diverse models corrects the rest. Across eight models and 25,664 classifications, cost for an identical job varied by a factor of 73 and wall-clock time by a factor of 18, with accuracy only loosely coupled to price.

Poster PDF

Install catllm for Stata

Requires Stata 16+ with Python integration (python query to verify) and Python 3.8+.

net install catllm, ///
    from("https://raw.githubusercontent.com/chrissoria/cat-llm/main/stata-package/") ///
    replace

catllm setup                  // one-time: install the Python backend
catllm setup, check           // probe what's installed

You’ll also need an API key from a supported provider (OpenAI, Anthropic, or Google), or a local Ollama install if you’d rather not send data to an API. Set the key once with, e.g., global ANTHROPIC_API_KEY : env ANTHROPIC_API_KEY, then run:

catllm classify response,                           ///
    categories(                                     ///
      "Housing cost: Moved over rent, a mortgage, or affordability." ///
      "Employment: Moved for a job, a transfer, or school."          ///
      "Family: Moved for caregiving or to be near relatives."        ///
      "Other: Does not fit any of the above categories.")            ///
    apikey($ANTHROPIC_API_KEY)                      ///
    model("claude-fable-5-1")                       ///
    generate(reason)

tab1 reason_*
regress reason_Housing_cost age female

Learn more