Semantics Intro
Semantics is how we know how close words’ meanings are. Just from writing, it’s not always possible. For example, take the words “cat”, “dog”, and “catastrophe”. If you didn’t know English, you might guess that perhaps “cat” and “catastrophe” are close in meaning, but that’s not true, relative to “dog” (dog-lovers, I know you’ll strongly object and point out that cats are a form of catastrophes). In fact, “dog” and “cat” are actually more closely related to each other than to “catastrophe”, because they both are pets.
In this blog post, we’ll talk about the origins of semantics and modern semantics, and the second part will be all about how LLMs understand meanings.
Philosophy of Language
The first people who pondered about how meaning comes from words were actually not linguists, but philosophers! Some of the ideas presented by them are now core concepts in semantics.
The ancient Greek philosopher Plato believed that a word represents a perfect abstract concept. For example, the word “dog” represents some idea called “dog” that doesn’t exist in the real world. All instances of a dog that you actually see are just imperfect imitations of that concept. Plato also thought that individual phonemes had some universal, objective, inherent meaning to them, but as you know, this is false, because sound has nothing to do with meaning. This is why Plato isn’t the most trustworthy source, but you can still take that first theory into consideration if you want.
Sense and Reference
Gottlob Frege first created the distinction between sense and reference. Reference is the actual thing (or set of things) that the word refers to, hence the name. Meanwhile, sense is the ideas and concepts behind what you express that reference as, and how you perceive it.
Multiple senses can have the same reference - for example, a 2D plane with evenly spaced, perpendicular gridlines of unit length can be thought of as either the Cartesian plane where each point is of the form \((x, y)\), the complex plane where each point is of the form \(x + i y\), or a vector space with basis vectors of \([1, 0]\) and \([0, 1]\), where each point is a vector of the form \([x, y]\) starting from the origin.
One sense can also point to multiple references - for example, if you think of a “person”, then that reference points to over 8 billion people. You could potentionally solve this problem by saying that the set of all people is still a single set.
Family Resemblence
Traditionally, meanings of words were given through their dictionary definitions. However, Ludwig Wittgenstein thought that maybe words can’t be directly defined, but instead speakers can tell what concepts match a word well. This is called family resemblence. For example, what is a game? There’s not a single answer that can always indicate whether something is a game or not. Chess, FizzBuzz, Monopoly, and soccer are all considered games, but you can’t really find a defining feature that is common to every single example of a game. All examples of games have links to other examples, but not necessarily to every single example, forming a family, hence that name.
Necessary and Sufficient Conditions
Necessary and sufficient conditions are also very important in semantics. A condition is necessary when a statement is true if and only if that condition holds, while it’s sufficient when if the condition holds, then the statement must be true, but the statement can also be true otherwise.
You can also think of it as the sufficient condition being a subset of the necessary condition, because if the sufficient condition holds, then the necessary condition holds too.
An example of this is when it comes to defining life. In biology, an organism is living if and only if it has cells and DNA (among five other conditions) - if it doesn’t have them, then it’s not living. This is a necessary condition of life.
On the other side, if an organism is an animal, then that is a sufficient condition to say that it is living. However, not all living organisms are animals. Again, you can view it like this: animals are a subset of anything with cells and DNA, and there aren’t any animals without cells and DNA - then it wouldn’t be sufficient to say that if something it an animal, then it is living.
Compositionality
A major concept in semantics is the Principle of Compositionality. Basically, since there are infinite possible meanings in the world but you can’t have a brand new word for EVERY single new concept due to the finite amount of mental capacity you have, more complex meanings must be derived from simpler ones.
Of course, if you’re going to use a concept very often, and it’s tedious to describe it using many words, it can either shorten or take a loanword.
For example, English took “déja vu” from French, which literally translates to “already seen”, which, in grammatically correct English (for most people’s internal grammar), would have to be expressed as “the feeling of having experienced something before” if not for this loanword. This is obviously much more concise.
Words can also shorten. The full name of “soccer” originated from “association football” (which obviously no one wants to say every time), so that became “assoc”, and then “soccer”.
However, these cases don’t take away from the main point that meanings as a whole are derived from smaller parts - that’s how you understand this sentence, for example. Syntax might allow you to arrange these meanings in the correct way, but compositional semantics allows you to get the overall idea as to what the sentence is actually saying. If you understand each smaller part, then you will understand the whole sentence, even if you haven’t seen this specific syntactical structure before.
Non-Compositional Meaning
One complication that arises from the idea of compositionality is that sometimes, the meanings of whole phrases can’t be derived from their smaller units.
Imagine if someone says, “That problem was a piece of cake”. Are you REALLY going to interpret this as something absurd like “ah yes, the problem they’re talking about was secretly part of a cake in disguise”? I think not. The meaning of “piece of cake”, even if it was derived from this image, just means “easy”.
.
Hyponyms and Hypernyms
Now, let’s get into how words relate to each other in terms of how specific they are. A hyponym is a more specific meaning, and a hypernym is more general. So, “cat” is a hyponym of “animal”, and “animal” is a hypernym of “cat”.
To be precise, the hyponym is a subset of the hypernym. This is why every cat is an animal, but not every animal is a cat.
The same concept of unions and intersections from set theory also applies here. “green cat” is just the intersection of anything that is a cat and anything that is green.
You could represent the sets of everything that words or phrases refer to and how they interact using Venn Diagrams, but this stuff is mostly just common sense, not some deep concept in linguistics. There is a NACLO problem on this.
Negations
One problem in semantics is about how negation works. You might think it’s relatively straightforward, until you realize there’s actually not a single way to “negate” something.
Imagine we have a spectrum (with integer values for simplicity) ranging from -10 (cold) to 10 (hot), and we wish to find the numerical value of “not hot”.
If you do computer science or logic, you might think of negation as the not/! operator, that switches a boolean. You might have a variable for the temperature, and “hot” means “temperature == 10”. With this logic, “not hot” means “temperature != 10”, which means it is any number from -10 to 9. This is the same line of reasoning from set theory, where negation might be the complement of a set (“hot” is a set with a single element, 10), so “not hot” means anything but 10.
However, “negate” literally means to multiply by -1, that’s why if you negate 3, you get -3, not every number but 3. So, the traditional view might say “not hot” is exactly “cold”, or “-10”.
Maybe “hot” just means anything that is leaning towards 10 (so it’s more hot than cold), so “not hot” is all the negative numbers.
You must also consider the pragmatic perspective (future blog post). Why would someone say “not hot” as opposed to “cold”? Depending on their intention, the range of what it represents on the spectrum can vary a lot.
If you think this is just a silly thought experiment, think again. One focus of semantics is to find out how languages convey negation. Here’s a more practical example of an English sentence. “I can not do it” could either mean “I am incapable of doing it”, or it could mean “I am capable of not doing it”. You can probably see the difference yourself.
Antonyms
Adding on to the idea of negation, we have the problem of antonyms, or pairs of opposite words. However, the concept of “opposite” is not really a single thing. There are different types of antonyms.
- Gradable: Two ends of a spectrum, allowing for stuff in between. For example, “hot” and “cold” are extremes, but you can also have “room temperature”, “somewhat chilly”, etc.
- Complementary: Exist as a binary, nothing in betwen. For example, someone can be “alive” or “dead”, but there is a clear cutoff distinguishing the two.
- Converse: Different ends of a relationship, and one can’t exist without another. For example, you can’t “buy” something without someone else there to “sell” it.
- Reversible: One undoes the other. For example, you can “turn on” a light and then “turn off” the same light, which has the same result as not doing anything (in other words, it has an orbit size of 2).
Again, it brings up the question of what “opposite” really is.
Questions
There are actually a few different types of questions, if you think about it.
The first is a question with a binary yes or no answer, which would usually start with “is/are” or “do/does”, like “do you eat food” or “is the sky blue”.
The second is a question where there is a set of possible answers, possibly of infinite size, and your answer is a subset of that (which can be a single element), like “where do you live” or “who are all in your class”, but even something with a really long answer like “what is the meaning of life”. Semantically, the first type can be thought of as a special case of this type.
Then, there are questions where you list all the options, like “what do you want, tea or coffee”, which assumes you will pick one of them.
Time
Another topic in semantics is how languages express time through tense and aspect, as well as time words. If you want to mark when some verb took place, different languages mark the boundaries differently.
English has past, present, and future, obviously, but it also accounts for whether something happens and it’s completed, or it’s reoccurring, or it’s completed after reoccurring, or it’s just general.
Thai doesn’t require that you distinguish tense or aspect, especially if you have context surrounding it.
Zulu has an extra tense, for distinguishing recent and distant past, but it’s also asymmetric because it only has a single future tense.
Hindi has something weird, where “yesterday” and “tomorrow” have the same word, “kal”.
All this goes to show that distinguishing events based on time can work very differently based on the language.
Kinship
One more interesting observation is that languages can distinguish words for family members differently. You may or may not have realized this problem, but English’s inventory for family members is kind of limited for relatively prominent stuff.
“Uncle” and “aunt”, for example, can mean both someone who shares a very recent common ancestor with you (their parents are the same as your grandparents) or it can mean someone who was married in to your family. Also, it doesn’t encode whether they are your father or your mother’s siblings / sibling in laws.
Tamil has a LOT of distinctions, meanwhile. Its vocabulary for “uncle” and “aunt” encodes for additional features, such as whether they are an older or younger sibling, whether they are a blood relative or not, etc.
In this NACLO problem, you see just how different other languages express kinship.
Spoiler: Explains the main idea of the problem
Basically, the way talking about siblings works in Manam is quite different from “brother” or “sister”. There is one word for “younger sibling of opposite gender”, another word for “younger sibling of same gender”, and a third for “older sibling” (apart from “I” in the problem, which should be obvious). However, “younger” and “older” are also symmetrical for the purposes of this problem, so you could get the same solution by making the younger sibling the one that merges genders.
Number
A final note on how different languages perceive meanings is for number. As you know, English counts numbers using base 10 (decimal numbers). However, different languages use other bases, though they are less rare.
The most common example is French. Although it mostly uses base 10, there are still remnants of vigesimal counting (base 20) from the previous Celtic languages in modern-day France before Latin arrived, like Gaulish (if you’re interested, Asterix and Obelix is all about the Gauls resisting the Romans). To observe this, look at these numbers:
- 19: dix-neuf (this is base 10, literally “ten nine”, similar to how English “nineteen” is connected to “nine ten”)
- 79: soixante dix-neuf (literally “sixty nineteen”)
- 80: quatre-vingt (literally “four twenty”, instead of English “eighty”, connected to “eight ten”, very vigesimal)
- 99: quatre-vingt dix-neuf (literally “four twenty nineteen”, this is full-on base 20)
For larger digit numbers, it goes back to base 10 (hundreds, thousands, etc), and base 20 is only used for the last two digits sometimes.
Base 10 likely originated because people have 10 fingers to count, so base 20 means they used their toes to count as well.
There are other bases too (such as base 60 from the Babylonians, which is where we get 60 in time and angles from), but they are much more uncommon. Even base 20 is rarely seen anymore.
Semantic Drift
Semantic drift is when the meaning of a word shifts over time. Looking at some common examples might illustrate this better.
“Incredible”
“Incredible” means something like “amazing” or “awesome”. You might realize that “credible” means “believable”, so “incredible” means “unbelievable”… right? Well, it used to!
Friend: “Yo bro, I just got a 5 on all my 31 AP exams this year!” You: “Really? That’s incredible!” Friend: “HOW DARE YOU SUGGEST THAT I CHEATED??? WE’RE NOT FRIENDS ANYMORE!!!”
Obviously, that won’t actually happen. Why? Because “incredible” has semantically drifted to mean something closer to “amazing” than “unbelievable”. It was used for things that were so good that it was almost hard to believe, but eventually it was only associated with those good things. What’s very cool is that this causes non-compositional meaning, because the meaning of “incredible” is no longer derived from “in” and “credible”.
Just for the record, I think the friend did cheat.
Semantic Bleaching with “lol”
Semantic drift can cause something similar to push or pull chains in sound changes, except for word meanings.
The abbreviation “lol” on the Internet technically means “laughing out loud”, but nowadays, no one uses it for extreme humor. It has evolved to mean very mild laughter. If someone says a very hilarious joke, and you simply reply “lol”, that might even be offensive because you didn’t find it that funny.
Therefore, people started adding profanity to intensify their humor. This is why “lol” got replaced with “lmao” (which I’m not gonna write out). However, after being used so much, even this got less intense in meaning. So, they just added even more bad stuff to humorous reactions to make it less mild.
This is why now people might elongate the “o” or they might do all caps or even add even worse curse words in the middle (like the voiceless labiodental fricative one) or just shift to other things entirely, like the skull emoji / “I’M DEAD” (which itself is quite dead now), “son i’m crine”, or who knows what else these days.
Semantic bleaching is when a word’s meaning gets less intense or “bleached” over time, so it causes replacements to be necessary.
Euphemism with “bear”
In PIE, the word for “bear” was *h₂ŕ̥tḱos. This is the ancestor to “ṛ́kṣa” in Sanskrit and “ursus” in Latin. Checks out so far, let’s try Greek - “arktos”. Cool, now, uh, let’s try the next common Indo-European language that most people use as examples - English. Wait, “bear”?
Yes, that’s right. Turns out, h₂ŕ̥tḱos was one of the world’s oldest no-no words. In Proto-Germanic, the word for “bear” was berô, meaning “the brown one”. Early Germanic people must have been terrified of bears, and they thought even saying the word for it would summon one. So, they used this euphemism instead.
Slavic languages have the same story, except their euphemism is “honey-eater” instead of “the brown one” (“medvedj” in Russian).
Mathematical Representation
One way for linguists to avoid ambiguity is to represent stuff using mathematical notation. Note in advance that most of this is common sense.
For example, take the “I can not do it” example. If we use “¬” to mean negation (as in flipping a boolean value), then the two meanings can be represented as can(¬do(x)) or ¬can(do(x)). Modal verbs like “can” in this case are kind of like functions. Here, x is a variable that can stand for any element that we’re interested in, and it behaves like the parameter for the inner function.
Feature Vectors
Just like how you can make a vector out of features for phonemes, you can do the same for meanings. For example, “king” can be represented as [+ human, + male, + royal, + adult], while “queen” can be represented as [+ human, - male, + royal, + adult], and “prince” can be represented as [+ human, + male, + royal, - adult]. The specific features depend on what you’re interesting in representing and contrasting, but this is the main idea.
Truth Conditions, Logic, and Set Theory
Truth conditions are basically a way to decompose a sentence into what conditions must make it true (kind of like necessary conditions). In the Minecraft Movie’s famous line “I… am STEVE”, the truth condition is that the speaker is Steve, so the set of all worlds where this speaker is Steve make this sentence true.
Now, if someone says “I like cats and dogs”, then the truth condition, because of the “and”, is the intersection of the set of all worlds where the speaker likes cats and the set of all worlds where the speaker likes dogs. If it was “or” instead, it would be the union of those two sets, but it could also be the union minus the intersection if it’s intended to not be both (basically “xor”).
Quantifiers are symbols for how many elements in a set are being talked about. For example, the universal quantifier (∀) means “all” or “every”, and the existential quantifier (∃) means “at least one” or “a”.
Truth conditions are kind of like checking if-statements in programming. Building on this, constructing the set of all worlds is a lot like list comprehensions, where you can filter out invalid worlds.
“Every student passed” would be written as ∀x (student(x) -> passed(x)). Basically, x can be anything, and if it is a student, then it passed, due to the universal quantifier. This should hopefully illustrate how formal semantics works without a bunch of yapping.
Conclusion
Now that you know how humans traditionally understand meanings, let’s learn about how machines understand them. Stay tuned for part 2!