Thoughts and Theory

Language Invariant Properties in Natural Language Processing

Are text properties kept when you transform (e.g., translate) a message in another language?

When you transform a text (i.e., translate, paraphrase, summarize it), you expect the general properties of your message to not change from input to output. For example, you expect your message to be perceived as happy even when translated to another language.

Issues can arise when this does not happen. For example, a Palestinian man was arrested when his post on Facebook was incorrectly translated from “good morning!” to “attack them”.

This blog post describes our new paper on the topic:

Federico Bianchi, Debora Nozza, Dirk Hovy (2022). Language Invariant Properties in Natural Language Processing. In the Proceedings of The First Workshop on Efficient Benchmarking in NLP.

Language Invariant Properties (LIPs, for short) are properties of language that should not be affected by a transformation. As another example, paraphrasing text should result in text with the same emotional valence: we don’t want to make a happy message sad. Thus, we need to consider and control this property when developing text transformation algorithms.
We introduce this concept of properties that should not be affected by any transformation and a python package that can be used to evaluate this problem.

Language Invariant Properties

Translation Makes You Sound Older and More Male

Let’s start with an actual example from one of our recent papers: “You Sound Just Like Your Father” Commercial Machine Translation Systems Include Stylistic Biases. (ACL 2020).

We found that after translation (e.g., from German to English), a lot of messages were predicted as male (more than what we saw in the original data). We found a similar effect for age: the messages sounded older.

Translation should not change speaker demographics. Translation System 1 makes you sound older. Image by Author.

Language Invariant Properties

Speaker demographics should not be affected by translation. So now we talk about our new paper: “Language Invariant Properties in Natural Language”.

The discussion above brings us to the necessity of better defining what should not be affected by all those transformation methods that change the surface form of text (e.g., paraphrase, summarization).

We call those properties that should not change after a transformation Language Invariant Properties (LIPs) (as mentioned before, this can be either a translation, a paraphrase, or a summarization).

There are potentially many LIPs, but significant ones are meaning and speaker demographics. Transforming should not affect the message, and if you are 26 years old, you don’t want your message to “sound” like it has been written by an 87-year-old.

More Formal Definition

Assume the existence of a set S of all the possible utterable sentences. Let’s define A and B as subsets of S. For our purpose, these two sets can be in the same or different languages. Let’s now introduce the following mapping function:

i.e., we can think of t as a transformation that changes the surface form of a text A into B (if the transformation is a translation, the function will translate a sentence in one language into another).

A language property p is a function that maps elements of S to a set P of property values. p is invariant if and only if

Meaning that applying the function t to a shouldn’t change the value of property p.

Evaluating LIPs

How do we evaluate this? Well, it’s easier than what you might think:

Let’s start again from the example above. We started from a dataset in which users self-identified themselves with male-female binary gender and age in English, Dutch, Italian, French, German. We trained language-specific age and gender classifiers, meaning that for each text in each language, we can now predict age and gender.

Then, we can use a bunch of commercial machine translation tools to translate the test portion (let’s call this O) of our dataset (e.g., we translate test text from Dutch to English). In particular, we used Google Translate, Microsft Translator and DeepL.

Now that we have translated the data, we use our classifiers (Dutch and English) on the original O data and the translated dataset. We thus obtain the predictions on the original data PO and on the translated data PT.
Studying the differences between these two prediction sets (PO and PT) and the original dataset (O), we can understand if age and gender are kept after translation. We might have found some bias if PO has a distribution similar to O, but PT does not.

The setting described above can be used to evaluate many different LIPs. As long as there is enough data to train language-specific classifiers, it is possible to use in-house tools to apply the transformation and check if there is any significant difference between the distributions.

LIPs Framework

We have built a framework to test and verify if LIPs are affected by transformations. You can find it on GitHub.

The logo of our framework.

The API, as for now, is still a bit shaky, but we are working on it to make it more stable. But the general frontend API should not change much.

We tried to make this as easy as possible to use. As long as you have your transformation method (e.g., a translation model), you can easily get the scores from your model (in terms of KL divergence and significance effects).

Code required to test gender LIP. Image from Carbon, made by Author.

The framework also gives you the raw predicted distributions to plot. Here is an experiment we run applying paraphrasing on a dataset for which we had self-annotated gender values.

O is the original test data, PO is the predictions of the classifier on the original data. PT is the predictions of the classifier on the transformed data. We can see that after transformation, a good number of messages have been recognized as coming from male users, suggesting that the transformation has introduced a bias in the distribution.

Experiment with a paraphrasing algorithm. O is the original test data, PO is the predictions of the classifier on the original data. PT is the predictions of the classifier on the transformed data. We can see that after transformation, a good number of messages have been recognized as coming from male users, suggesting that the transformation has introduced a bias in the distribution. Image by Author.

Limitations

LIPs are an interesting theoretical concept, but they might not always generalize to tasks. For example, translating from Spanish to Japanese has to consider the cultural differences of the two countries, and it might not be possible to reduce these to a set of LIPs.

The tool we implemented comes with some limitations. We cannot altogether remove the learned bias in the classifiers. We always assume that when there are two classifiers, these two perform reliably well on both languages to compare the output. The same goes for paraphrasing: we assume that the LIP classifier performs equally well on both the original and the transformed text.

Conclusion

We believe LIPs can be beneficial to better understand how well transformation methods work. We have reached a point in the field in which these methods are reliable enough to transform the general message (i.e., Google Translate is good enough for many use-cases); however, not keeping LIPs fixed during transformation can create misunderstandings.

Thank you for reading our work! You can find more information about this in the paper! Feel free to contact us if you have any questions! Feel free to reach out on Twitter.

Acknowledgements

Thanks to Dirk and Debora for edits and comments on this!