Studying the History of Concepts with Embeddings
Project summary
How do we follow the history of a concept (rationality, inflation, freedom, climate) in millions of sentences, without deciding in advance what its exact meanings are? Language-model embeddings make this possible at a scale historians of ideas could not reach by reading. Embeddings place every sentence in a multidimensional space where similar sentences are close. It is thus possible to identify different meaning of a term and their evolution over time.
But such approach also comes with a long series of choices (which sentences and texts stand for the concept? how do we cut and cluster them? how do we follow clusters through time?), and each choice shapes the story the historian ends up telling.
This project with Thomas Delcey is about that method. The goal is to develop, test and document a toolbox for the history of disciplinary concepts that makes these choices explicit, lets vary the choices one at a time, and reports how much the results depend on them.
First step: One Sentence at a Time
The project starts from a paper written with Thomas Delcey and Alexandre Truc, “One Sentence at a Time: A Quantitative History of Rationality in Economic Thought” (published version; preprint). In this article, we follow the concept of rationality in nearly 260,000 economics articles from 1900 to 2009. The paper is the first full implementation of a method of “semantic drift” or “semantic change” (Periti and Montanelli 2024) in the history of economics:
- Sentence embeddings. Every sentence of the full texts is represented by a vector (all-mpnet-base-v2, 768 dimensions).
- A representative vector of the term. For each year, the sentences containing “rational” or “rationality” are averaged into a centroid, and the centroids of an 11-year window are averaged again. This vector summarises where the term lives in the space of sentences, for one specific year.
- A sentences-corpus. For each year, the 1% of sentences closest to this vector are kept, whether they contain the word or not: “agents maximise their utility” is about rationality without saying so.
- Clusters of discussions. Each decade is clustered on its own (UMAP and HDBSCAN), then the clusters of different decades are linked by the similarity of their centroids into intertemporal clusters: the emergence of a social view of rationality, the circulation of bounded rationality, the rise of behavioural economics.
- Bibliometric anchoring. The clusters are confronted with bibliographic-coupling communities, to tell who was discussing rationality, in which literature, in which way.
We have created an interactive application to explore the clusters (the app can also be downloaded from Zenodo).
Resources:
- Methodological appendix (Zenodo), which details each step.
- Replication code (the data are under licence and cannot be shared).
Making the method the object of study
Writing the paper showed that each step is a decision among several defensible choices, and that these decisions are rarely compared. The project organises them in two questions that can be combined freely:
- Which texts represent the concept? Token embeddings of the word [clusters are “senses”; Periti et al. (2022)], sentences containing the word (clusters are contexts), or a sentences-corpus selected from a representative vector (clusters are contexts of the concept, with or without the word).
- How do we follow clusters through time? Cluster each period then link the clusters, or incremental approaches in which clusters carry a memory from one period to the next, as in the WiDiD method of Periti and co-authors.
We want to work on the following methodological questions:
- Robustness. How much do the clusters change with the random seed, the dimensionality reduction, the share of sentences kept, the window? Which results recur across runs and can be read, and which cannot?
- Presentism. Models are trained mostly on recent texts and corpora grow exponentially: how do these biases distort the picture of the past, and how can the design limit them?
- Comparison. What do the different strategies reveal and hide when applied to the same corpus, and how should a historian read their disagreements?
- Validation. How can a historian check that a cluster is what the algorithm says, by reading sentences, by examples, by distinctive words, by comparison with traditional history of the concept, etc.?
- Other concepts and corpora. Do the choices made for rationality in economics travel to other concepts, disciplines and languages?
A companion software package
The methods developed in this project are being implemented in an R package, embedscope, still in development and not yet released. It starts from embeddings computed elsewhere and aims to read them without loading them in memory (corpora of hundreds of gigabytes on a laptop), build representative vectors and sentences-corpora, reduce and cluster, follow clusters through time with several strategies, and measure the stability of the results. Each function documents its formulas, defaults and the literature behind them, so that the package serves transparency as well as convenience. The longer-term ambition is a general package for manipulating embeddings in R, of which the study of conceptual change is the first use case.