Sociolinguistic auto-coding has fairness problems too: measuring and mitigating bias

1 Introduction

As research in language variation and change involves ever-larger corpora, researchers have increasingly turned to computational techniques to relieve pinch-points in the methodological workflow (e.g., Barreda 2021; McAuliffe et al. 2017; Wassink et al. 2018). Before sociolinguistic variation can be analyzed, for example, researchers must perform tedious and time-consuming coding: assigning variants to tokens. Thus, several teams of researchers have in recent years independently explored sociolinguistic auto-coding (SLAC) for phonological variables (Kendall et al. 2021; McLarty et al. 2019; Villarreal et al. 2020). In a typical phonological^[1] SLAC use case, humans hand-code a fraction of the available tokens, rather than the entire data set; these hand-coded tokens (and their acoustic characteristics) comprise the training set from which a machine learning (ML) model attempts to discern the acoustic patterns characteristic of different variants. This model (also known as an auto-coder) then applies these patterns to a test set of uncoded tokens to predict how humans would have coded them, essentially replicating human coding with substantial time savings. These early investigations have found auto-coding performance comparable to human inter-coder reliability.

Research on ML fairness has found that predictive algorithms can reproduce intergroup biases in the data they are trained on (e.g., Field et al. 2021; Koenecke et al. 2020), with concrete costs to potential users (Mengesha et al. 2021). For example, journalists have raised concerns that an algorithm used to assess the risk of a pretrial defendant, COMPAS, inadvertently uses defendants’ race as a decision criterion (Angwin et al. 2016). COMPAS erroneously classifies lower-risk Black defendants as higher-risk, suggesting that its risk classifications are based (partially, at least) on the defendant’s race. COMPAS’s training set does not explicitly include race, but it does include items from which COMPAS can implicitly recover racial identification (e.g., parents’ criminal record, peers’ drug use). This predictive bias thus stems from a phenomenon I call “overlearning”: when an algorithm’s predictions are inadvertently based at least in part on group membership.^[2] As a rule, ML fairness is generally an afterthought in an algorithm’s development (e.g., Bender et al. 2021; Field et al. 2021); several US states had been using COMPAS risk scores for years before its fairness issues came to light (Angwin et al. 2016). Making matters more complicated, there is no universally applicable way to define ML fairness (e.g., Berk et al. 2021; Corbett-Davies et al. 2017; Kleinberg et al. 2017).

As an ML application, SLAC is potentially subject to biased predictions as a result of overlearning group-level characteristics rather than (or in addition to) legitimate acoustic markers of different variants. Whereas the cost of COMPAS’s biased predictions is wrongful detention, the cost of biased predictions in SLAC would be erroneous empirical observations. Given the central importance in sociolinguistics of correlating speaker groups to differences in variable usage, an auto-coder that under- or over-represented the strength of a speaker group constraint would be highly problematic. Fortunately, SLAC is in its infancy; unlike COMPAS and other biased algorithms, it is not too late to reverse the typical “unleash the algorithm first, ask questions later” pattern. The present study thus seeks to answer the following research questions (RQs):

1. How should (un)fairness be defined and measured for SLAC?

2. Is SLAC prone to unfairness? If so, then …

3. How well do unfairness mitigation strategies work for SLAC?

I investigate these research questions with speaker gender as the group-level characteristic, and English non-prevocalic /r/ as the variable to be auto-coded. Both gender and /r/ are good “test cases” for SLAC fairness. Gender fairness is relevant since gender commonly influences variation (e.g., Cheshire 2004; Labov 1990, 2001), including /r/ (e.g., Bartlett 2002; Nagy and Irwin 2010; Villarreal et al. 2021). /r/ is a prototypical variable for SLAC since it is acoustically complex (Heselwood 2009; Lawson et al. 2014, 2018; Stuart-Smith 2007; Stuart-Smith et al. 2014; Zhou et al. 2008), difficult to code reliably (Fosler-Lussier et al. 2007; Hall-Lew and Fix 2012; Irwin 1970; Lawson et al. 2014; Pitt et al. 2005; Yaeger-Dror et al. 2009),^[3] usually treated as a binary with present and absent (also known as rhotic and non-rhotic) variants, and variable within and across speech communities (for these two final points, see, e.g., Bartlett 2002; Becker 2009; Gordon et al. 2004; Labov et al. 2006; Nagy and Irwin 2010). Despite these challenges, /r/ auto-coders have demonstrated performance comparable to human inter-coder reliability (Kendall et al. 2021; Villarreal et al. 2020).

This paper investigates these questions from the perspective of researchers who want to use SLAC on data where they suspect gender correlates with variable rhoticity. First, I choose definitions that make sense for SLAC and translate these definitions to quantifiable metrics. Second, I re-analyze Villarreal et al.’s (2019, 2020 /r/ auto-coder, finding that it indeed makes unfair predictions by gender, possibly as a result of overlearning speaker gender. Third, I attempt multiple strategies to mitigate gender unfairness, finding that it is possible to produce a fair auto-coder, albeit at the expense of overall auto-coding performance; one such strategy was used to create a fair /r/ auto-coder in a separate article (Villarreal et al. 2021). I close by discussing what SLAC fairness means not just for auto-coding, but more broadly for how we conceptualize variation as an object of study.

2 RQ1: Defining and measuring fairness for SLAC

Algorithms that attempt to sort observations into two or more categories are known as “classifiers” (e.g., SLAC, spam filtering, cancer detection), with the categories known as “classes” (e.g., present vs. absent /r/, spam vs. not spam, malignant vs. benign; Hastie et al. 2009). Classifier performance is assessed by comparing the classifier’s predictions to the so-called ground truth. For two-class classifiers, such as /r/ auto-coders, this comparison can be represented in a confusion matrix, such as Table 1; a “true present”, for example, is an /r/ token that is present in reality and was correctly auto-coded “present”. These concepts are similar to notions like “true positive” and “false positive” (which are common in ML, medical testing, and other domains); but “positive/negative” does not make sense for auto-coding because it implies that only one class (“positive”) is the real object of detection, as with classifiers that attempt to detect spam email or malignant tumors. Arguably, in auto-coding “no class is more important for purposes of detection” (Villarreal et al. 2020: 11).

The confusion matrix is the building block for calculating performance metrics (quantifiable measurements of a classifier’s success at sorting observations into classes). Table 2 displays some common performance metrics. Analysts typically use performance metrics to compare classifiers to one another; for example, Kendall et al. (2021) find that overall accuracy for -ing classifiers changes based on the type of training data, and Villarreal et al. (2020) find better overall accuracy for binary /t/ than binary /r/. However, these metrics can also be used to assess fairness, by breaking down performance by group (Berk et al. 2021; Corbett-Davies et al. 2017). Thus, just as we can quantify performance, we can quantify fairness for the purposes of choosing a fair auto-coder.

There are multiple possible definitions of fairness, and if groups have different base rates (see Table 2), it is impossible for a classifier to satisfy all fairness definitions at once (Berk et al. 2021; Corbett-Davies et al. 2017; Kleinberg et al. 2017). Applied to SLAC and gender fairness, an auto-coder cannot satisfy all definitions if women and men have different community-wide rhoticity rates – exactly the sort of external factor sociolinguists attempt to detect (This is in contrast to classification use cases like pretrial risk assessment, where it is reasonable to assume a priori that potential Black and White defendants are equally risky). As a result, analysts must choose the fairness definition or definitions that make the most sense for them (Kleinberg et al. 2017).

For SLAC, I argue that “fairness” is best captured by two definitions – overall accuracy equality and class accuracy equality – corresponding to three metrics (Table 3). These definitions make sense because of what separates SLAC from other classification problems. Because there is no “positive” class (as discussed above), it makes sense to measure both overall accuracy equality, which “assumes that true negatives are as desirable as true positives” (Berk et al. 2021: 15), and class accuracy equality (for both absent tokens and present tokens). Overall accuracy equality is also useful over and above class accuracy equality because it weights each class by its prevalence in the data, thus giving the clearest picture of the total number of (in)accurately coded tokens. Because we cannot assume equal base rates a priori, it makes no sense to pursue (conditional) statistical parity (Berk et al. 2021: 16; Corbett-Davies et al. 2017: 798), which would penalize auto-coders for predicting different rates of rhoticity for women and men. These metrics may not capture “fairness” for all potential SLAC use cases, however, so Section 5 revisits alternative fairness definitions and metrics.

3 RQ2: Assessing fairness for SLAC

Using the fairness definitions and metrics established in the previous section, this section re-analyzes Villarreal et al.’s (2019, 2020) /r/ auto-coder to assess gender fairness.

4 RQ3: Mitigating SLAC unfairness

The unfair predictions in Villarreal et al.’s (2019, 2020 auto-coder potentially compromise SLAC’s usefulness in sociolinguistic methodology, as cross-group comparisons are a cornerstone of sociolinguistic research. Fortunately, ML researchers have found numerous ways to mitigate unfairness in classification problems like SLAC (e.g., Corbett-Davies et al. 2017).

4.2 Results

The majority of unfairness mitigation strategies were fairer than the no-UMS baseline (Figure 1). In particular, ten UMSs did not yield any significant differences between women’s and men’s performance (with χ ² [1] ≤ 2.30, p ≥ 0.13). This fairness did not come for free, however – all strategies that improved fairness worsened performance on at least one metric, and some UMSs improved fairness or performance on one metric but worsened fairness or performance on others. For example, UMS 1.3.2 reduced the overall accuracy difference between women and men but yielded worse present class accuracy regardless of gender (see Figure 2 below). Regardless of the strategy, a loss of performance makes intuitive sense, as the UMSs ultimately boil down to “give the auto-coder less information”.

Figure 1:

Comparison of the fairness and performance of unfairness mitigation strategies (UMSs). Fairness increases from left to right; performance increases from bottom to top. The dotted line is the no-UMS baseline. “Optimal” denotes the UMS used by Villarreal et al. (2021). The numerical data for this figure can be found in Appendix B.

Figure 2:

Comparison of the fairness and performance for the optimal unfairness mitigation strategy (UMS) and several similar UMSs. Fairness increases (gender difference decreases) from left to right; performance increases from bottom to top. The dotted line is the no-UMS baseline. UMS labels are as in Appendix A. The numerical data for this figure can be found in Appendix B.

Given this trade-off, the UMS that has the greatest fairness may be undesirable because its performance cost is too high. So how do we choose a UMS for auto-coding our data? One technique for winnowing down the space of options is to find the UMSs for which any other UMS that is better in fairness is worse in performance, or vice versa; in ML, observations that have this property are known as Pareto-optimal. Thus, researchers may choose a UMS that is neither the fairest nor the best performing, but for which the fairness–performance trade-off is acceptable.

Villarreal et al. (2021)’s Southland /r/ auto-coder used UMS 4.2.1, which not only is Pareto-optimal in all three facets of Figure 1 but also ranks highest for all three fairness metrics.^[6] This UMS is a combination of UMS 1.3.1 (downsampling: removing female absent to achieve equal rhoticity rates by gender) and 2.2 (valid predictor selection: removing F0 measures). UMS 4.2.1 had near-zero unfairness (Table 6). It is important to note that not all auto-coders will necessarily have one UMS that is Pareto-optimal across the board; as a result, SLAC users may face a judgment call in terms of choosing the UMS that is right for their application.

Table 6:

Gender fairness metrics for optimal UMS. Values in parentheses indicate the change from the unfair auto-coder results seen in Table 4; positive changes indicate improvement in accuracy (female and male columns) or greater unfairness (difference column). The χ ² column reports test of homogeneity comparing female and male columns.

Metric	Female	Male	Difference	χ 2
Overall accuracy	0.8044 (−0.0871)	0.8070 (−0.0152)	−0.0026 (−0.0667)	χ 2 [1] = 0.02, p = 0.90
Absent class accuracy	0.9236 (−0.0398)	0.9239 (+0.0130)	−0.0002 (−0.0523)	χ 2 [1] < 0.01, p > 0.99
Present class accuracy	0.5670 (+0.0526)	0.5718 (−0.0745)	−0.0048 (−0.1270)	χ 2 [1] = 0.01, p = 0.93

Returning to the hypothesis that overlearning is the underlying cause of unfairness, two pieces of evidence support this hypothesis. First, the auto-coder’s class accuracies by gender (absent higher for women, present higher for men) mirror the gender/rhoticity distribution in the training set (men are more rhotic than women). This suggests that, in some cases, the auto-coder attends to acoustic measures that cue gender rather than “legitimate” cues of rhoticity. Second, the unfairness mitigation analysis found acoustic measures (pitch) that, when removed, substantially reduced unfairness.

However, a small comparison set of UMSs tells a different story (Figure 2). If overlearning were the primary cause of unfairness, then removing pitch measures (UMS 2.2) or normalizing pitch by speaker (UMS 3.1) would substantially improve fairness. Both only resulted in modest fairness gains. In other words, overlearning appears to be a cause of unfairness, but not (contra my original hypothesis) the primary cause of unfairness.

5 Discussion

Sociolinguistic auto-coding was developed under the premise that a methodological challenge could be met with a technological solution. Early work in phonological SLAC has demonstrated both its promise for meeting this methodological challenge and that questions remain to be answered (Kendall et al. 2021; McLarty et al. 2019; Villarreal et al. 2020). In short, it is crucial to frame SLAC, COMPAS (Angwin et al. 2016), or any other algorithm not as a “magic bullet”; engaging in uncritical “tech solutionism” prevents us from clearly seeing algorithms’ limitations (Bender and Grissom 2024). So too should we avoid the trap of seeing bias successfully mitigated in a single SLAC use case and think “we solved SLAC bias!”.^[7] This paper used a SLAC use case that I argue is fairly typical: researchers want to deploy an auto-coder of /r/ on large-scale corpus data in a community where they suspect gender will be part of the analysis. But a single use case cannot cover all possible outcomes. Of course, unfair predictions would be problematic anytime speaker groups are in play – since unfair predictions distort correlations between speaker groups and variable usage – but different circumstances may yield different outcomes or even necessitate different fairness definitions. I discuss these different circumstances in reverse order of the research questions.

First, for Southland /r/, unfairness was partially caused by the auto-coder overlearning a group characteristic based on several acoustic measures in the feature set. Other SLAC cases may exhibit fairness metrics that suggest overlearning, but without an acoustic measure (or other feature) that is readily identifiable as the locus of overlearning; in these cases, mitigating unfairness could be more challenging. In other cases, unfairness may not be caused by overlearning at all (e.g., if groups have equal base rates), but inadequate representation in the training set (e.g., Koenecke et al. 2020), necessitating other unfairness mitigation strategies. In still other cases, the UMS that maximizes fairness may negatively impact performance so much that it is untenable to deploy on the test set.

Second, other SLAC use cases may also differ with respect to the degree of unfairness; there is no guarantee that all auto-coders will be unfair. Just as Villarreal et al. (2020) find better overall accuracy for binary /t/ than /r/, some variables are easier to code than others (both by hand and by auto-coder); conceivably, these variables may be less prone to unfairness. Likewise, auto-coders that achieve balanced class accuracies (unlike /r/) may be less prone to unfairness. On the other hand, whereas the unfair /r/ auto-coder overpredicted the majority class for both groups, other auto-coders could overpredict different classes for different groups.

Third, I argued that the most appropriate fairness definitions for SLAC were overall accuracy equality and class accuracy equality, but future SLAC use cases may demand alternative fairness definitions. For some variables in some communities, researchers might discard the “no positive class” assumption that arguably sets SLAC apart from classification problems like spam detection; for example, it may be more important to detect a variant that is incipient or dying out than its better-established alternative(s). Not all categorical variables are binary. In addition, even for traditionally categorical sociolinguistic variables like /r/, sociolinguistic reality may be better represented by continuous acoustic variation (e.g., not absent/present but degree of rhoticity), a view supported by a growing body of sociophonetic research (Duncan 2021; Holliday and Villarreal 2020; Podesva 2007; Purse 2019). When auto-coders are used to generate probabilistic rather than categorical predictions of variant-hood (McLarty et al. 2019; Villarreal et al. 2020), researchers should use fairness definitions that account for probabilistic predictions, such as calibration (Kleinberg et al. 2017). Finally, extending this approach beyond a binary group characteristic would require rethinking how the fairness definitions are translated to quantifiable metrics (e.g., taking the standard deviation of overall accuracies rather than the difference).

Given this discussion, it seems prudent to revisit the initial framing of SLAC as essentially replicating human coding. Instead, these findings support the idea that auto-coders “should not seek simply to replace human analysts but rather that they reflect alternative approaches to coding that have advantages and appropriateness for some applications and disadvantages and inappropriateness for others” (Kendall et al. 2021: 15). In other words, auto-coding (and human coding) is not “searching for one right answer” – different analyses may necessitate different approaches to auto-coding. In the case of Southland /r/, there was previous sociolinguistic and sociological evidence to hypothesize the importance of speaker gender (Bartlett 2002; Jackson et al. 2009), so achieving gender fairness was well worth the performance loss compared to the unfair auto-coder. If a researcher were to perform a future analysis of the same data that only considered men, however, they should use auto-codes that maximize performance. A different future analysis that attempted to cluster speakers based on rhoticity patterns (e.g., Brand et al. 2021) would need to ensure fairness across speakers. In other words, the “correct” code for any given token in the test set may change based on the needs of the analysis. As a result, SLAC is potentially appropriate only for corpus sociolinguistic approaches, rather than micro-sociolinguistic analyses. This may be an unsettling idea for researchers who are used to hand-codes representing the “ground truth”, but as corpus sociolinguistics matures, we will need to recognize the inherent uncertainty that “big data” implies. These observations add a new dimension to Villarreal et al.’s (2019) observation that auto-coding is “not a one-size-fits-all process”.

These differences notwithstanding, the present study represents two “proofs of concept” – SLAC is prone to predictive bias, and SLAC bias can be mitigated. To be clear, I am not suggesting that SLAC is too inherently flawed to use; the findings of RQ3 clearly show otherwise, and this bias-mitigation approach to auto-coding has already been used in sociolinguistic research (Villarreal et al. 2021). Rather, I argue that users of computational methodologies like SLAC deserve a “warts and all” awareness of their drawbacks before they have the chance to unleash harm. Algorithms are never neutral – they are the result of choices by human designers, so they reflect our priorities, biases, and mistakes.