Why Do Hinge Loss and Softmax Look the Way They Do?

This is a side note to Part 6, not the next step in the series. Skipping it and going straight to Part 7 (gradient descent) leaves out nothing the later parts depend on. It’s here because Part 6 took both loss formulas as given without saying where they come from.

Why the max? Why add 1? Why raise e to a power instead of just taking an absolute value or squaring the score? Without those answers, the formulas can look like they were pulled out of a hat.

Each one is built backward from a goal. Every piece does a specific job. Here’s what each piece is for.

What any loss function has to do

Before any specific formula, a loss has to satisfy three things:

1. It’s small (ideally zero) when the prediction is good.
2. It grows when the prediction is bad.
3. You can push it downward by nudging the weights.

A loss exists to be made smaller by the training process (that’s what Part 7 covers). So the formula has to respond to weight changes in a way that points toward better weights.

Lots of formulas satisfy these three requirements. Hinge and softmax are two of those, but they aren’t the only possible choices. They’re just two well-worn options, each derived from a goal.

Hinge loss, one symbol at a time

The goal behind hinge loss: make the correct label’s score beat the wrong label’s score by a comfortable buffer.

loss = max(0, 1 + wrong_score - correct_score)

Here’s what each part does.

(With more than two labels, like the cat/dog/bird examples below, wrong_score is the highest-scoring wrong label. Some versions instead sum a penalty over every wrong label; using just the single highest keeps this simple.)

Why a difference (wrong_score - correct_score) instead of the raw scores? Because the absolute size of the scores is meaningless; only the gap between them matters. A classifier scoring cat = 1000, dog = 999 is barely better than a coin flip. One scoring cat = 5, dog = -5 is more confident. What separates them is the gap, so the loss depends on the gap, not the raw numbers.

Why wrong_score - correct_score and not correct_score - wrong_score? Because a loss has to get bigger when the prediction gets worse, and wrong_score - correct_score does exactly that, while correct_score - wrong_score decreases instead. wrong_score - correct_score is positive exactly when a wrong label beats the correct one, which is the bad case, so it works as a penalty: large and positive when things are bad, zero or negative when they’re fine.

The reverse, correct_score - wrong_score, is large when things are good, since it measures how far the correct label is ahead. That quantity has a name, the margin, and it’s the natural thing to plot on a graph showing how confidently correct or wrong a prediction is, but it’s the wrong sign to hand to a penalty. The formula flips it back anyway: 1 + wrong_score - correct_score is the same as 1 - (correct_score - wrong_score), or 1 - margin. The two orderings are just negatives of each other, and hinge loss uses the one that already grows in the bad direction.

Scenario C from Part 6 (a wrong guess, cat = 3, dog = 10) shows why the sign matters:

wrong_score - correct_score = 10 - 3 = +7   (positive: works as a penalty)
correct_score - wrong_score = 3 - 10 = -7   (negative: useless as a penalty)

Why max(0, ...)? This is the piece that gives hinge loss its name and its behavior. Suppose you dropped the max and just used 1 + wrong_score - correct_score. Then a prediction that’s already correct by a wide margin would produce a large negative loss, and the training process would keep trying to drive that margin even wider, spending effort on examples that are already fine. The max(0, ...) says: “once you clear the required margin, your loss is exactly zero.” It caps the demand at good-enough. That threshold, past which an example is satisfied and contributes nothing, is what the max is for.

Scenario B from Part 6 (a confident, correct guess, cat = 100, dog = 1) shows what the max does:

without max:  1 + 1 - 100 = -98   (a big negative "reward" for being right)
with max:     max(0, -98)  = 0    (no reward, no penalty: nothing left to do)

Why the 1? That’s the size of the buffer. Without it, the loss would drop to zero the moment the correct score edged ahead of the wrong score by any amount at all, even 0.0001. That’s fragile: a speck of noise flips the answer. The 1 demands the correct score win by a clear margin, not barely.

Scenario A from Part 6 (a close call, cat = 10, dog = 9.5) shows the difference:

without the 1:  max(0, 9.5 - 10)     = 0     (a hair ahead already counts as fine)
with the 1:     max(0, 1 + 9.5 - 10) = 0.5   (still penalized until the gap reaches 1)

But why 1 specifically, and not 3 or 0.5? Because the scores have no fixed scale. The classifier can multiply all its weights up or down, which scales all its scores up or down, so it can make a margin of “1” mean whatever it needs. Pick any positive number and you get an equivalent classifier with rescaled weights. The 1 is just a convention that pins down the scale. Its exact value doesn’t matter, because the weights adjust around it.

So max(0, 1 + wrong_score - correct_score) reads, in full: “zero penalty if the correct score beats the wrong score by at least 1; otherwise the penalty grows in proportion to how far short you fell.”

Softmax loss, one symbol at a time

Different goal: turn the scores into a probability for each label. Then penalize the model by how little probability it assigned to the correct label. The less probability on the correct label, the bigger the penalty.

probability_cat = exp(score_cat) / (exp(score_cat) + exp(score_dog) + exp(score_bird))

# cat is the correct label in this example, so the loss uses probability_cat
loss = -log(probability_cat)

Why exponentiate (exp(score))? A natural way to turn a set of numbers into probabilities is to divide each by the total, the way the chance of drawing a red ball from a bucket is (red balls) / (all balls). That’s the divide step below, and it works for ball counts because counts are never negative. Scores can be negative, though, and dividing them directly breaks: with cat = 10, dog = -5, bird = 1 (sum 6), you’d get cat = 1.67 (above 1) and dog = -0.83 (negative), which aren’t probabilities. So the scores have to be made positive first. exp(x) means the fixed number e (about 2.718, a constant the way π is about 3.14159) raised to the power x, and it turns any number into a positive one while keeping the ranking. Then dividing by the total works.

Why not just take the absolute value, then? Absolute value turns numbers positive too, but it treats +5 and -5 the same, so a label the classifier strongly rejected (a big negative score) would come out with a high probability, scrambling the ranking. exp always grows as its input grows, so it keeps the ranking intact. The section “Why exp and not just absolute value, or squaring?” further down works through the numbers.

Why divide by the sum of all the exp values? Dividing each positive number by their total forces the results to add up to 1, which is the defining property of a set of probabilities. That step just turns a handful of positive numbers into proportions.

With a probability for each label in hand, the loss itself is -log(probability of the correct label). The penalty rises as the correct answer’s probability falls.

When the probability of a correct guess is 1, that means -log(1) is 0, which means the penalty is zero. That’s what we want: a penalty of zero for a “correct-guess” probability of 1.

Conversely, as the probability of our guess drops toward 0, -log(probability_of_our_guess) climbs toward infinity. So confidently correct costs nothing and confidently wrong costs a lot.

The expression has two parts, the log and the minus sign in front. Here’s what each is for.

Why take the log? A single guess gives one probability for the correct label, and -log of it is one number. The reason for the log shows up only when training combines many guesses across the whole training set.

To fit the whole set, you want the model to get image 1 right and image 2 right and so on, and probabilities of independent events multiply. So the quantity you want to be high is the product of every guess’s correct-label probability. Say the model gives the correct label a probability of 0.7 on one image and 0.2 on another:

product of both = 0.7 × 0.2 = 0.14

loss on image 1 = -log(0.7) = 0.357
loss on image 2 = -log(0.2) = 1.609
total loss      = 0.357 + 1.609 = 1.966

That total, 1.966, is exactly -log(0.14), the log of the product, because log(a × b) = log(a) + log(b). So adding the per-guess losses gives the same answer as taking the log of the whole product. That equivalence is why the formula uses log:

• Without it, combining guesses means multiplying their probabilities, and multiplying thousands of numbers below 1 underflows to zero on a computer, wiping out the information.
• With it, combining guesses means adding one number per guess, which stays a normal, representable number (a thousand values around 2.3 sum to about 2,300, which a computer stores fine).

So the log is computed per guess, but adding those per-guess losses is the same as taking the log of all the raw probabilities multiplied together.

Why the minus sign? Training makes the loss as small as possible, so a worse prediction needs to produce a bigger loss. log(probability) does the opposite. It’s 0 for a perfect prediction and goes more and more negative as the correct label’s probability shrinks:

• correct label at 1.0: log(1.0) = 0
• correct label at 0.2: log(0.2) is about -1.6
• correct label at 0.1: log(0.1) is about -2.3

Flipping the sign turns that around: -log(probability) is 0 for a perfect prediction and grows as the prediction gets worse. Now a worse prediction means a bigger loss, and shrinking the loss means improving the prediction.

Why exp and not just absolute value, or squaring?

If all you want is to turn negative numbers positive, absolute value does that. |-5| = 5. And if you want bigger scores to count for more, squaring does that. (-5)² = 25. So why reach for exp, which needs a special constant and a power?

The answer is that “make it positive” is too weak a description of what softmax needs. The full requirement is positive and order-preserving, including across negative numbers. A probability assignment has to keep the classifier’s ranking intact: if the classifier scored cat higher than dog, then cat must come out with more probability than dog. Always.

Test the candidates against that with real numbers. Say the scores are:

cat  =  3     # the correct label
dog  = -5
bird =  1

The classifier clearly likes cat most and dog least.

exp:

exp(3) = 20.09,  exp(-5) = 0.0067,  exp(1) = 2.72     (sum = 22.82)

P(cat)  = 20.09 / 22.82 = 0.880
P(dog)  = 0.0067 / 22.82 = 0.0003
P(bird) = 2.72 / 22.82 = 0.119

Cat keeps the highest probability and dog the lowest, matching the score ranking.

The next two aren’t used in any real classifier; they’re here only to show why keeping the ranking matters when scores can be negative.

absolute value:

|3| = 3,  |-5| = 5,  |1| = 1     (sum = 9)

P(cat)  = 3/9 = 0.333
P(dog)  = 5/9 = 0.556
P(bird) = 1/9 = 0.111

Dog now has the highest probability, even though the classifier scored it lowest. Absolute value threw away the sign, so the strongly-rejected class (score -5) became the strongly-preferred one. Broken.

squaring:

3² = 9,  (-5)² = 25,  1² = 1     (sum = 35)

P(cat)  = 9/35  = 0.257
P(dog)  = 25/35 = 0.714
P(bird) = 1/35  = 0.029

Same failure, worse. Dog wins again, because squaring also ignores the sign.

Both absolute value and squaring are symmetric around zero: they treat +5 and -5 as the same. But in a classifier, +5 means “probably this class” and -5 means “probably not this class.” Any function that maps those to the same output is unusable. exp is the natural function that always increases as its input increases, over every number including negative ones, so it never scrambles the ranking. That rules out both alternatives.

exp turns score gaps into probability ratios. If cat scores 2 higher than dog, cat ends up about 7.4 times more probable (because exp(2) is about 7.4), no matter what the baseline scores are. A useful consequence is that adding the same amount to every score changes nothing, because the shift cancels out in the division. That matches the fact that only the gaps between scores carry meaning, not their absolute values.

The structural reason: adding scores is the same as multiplying probabilities. This is the part that makes exp the natural choice, not just any positive-making function. It comes down to a mismatch between how the classifier builds a score and how probabilities combine.

A linear classifier builds each score by adding: it adds up a weighted amount from every pixel. Adding is its whole operation.

Probabilities combine by multiplying. If one thing has a 1-in-2 chance, and a separate, independent thing also has a 1-in-2 chance, the chance of both happening is 1/2 × 1/2 = 1/4. You multiply the chances, you don’t add them.

So the classifier speaks in adding and probability speaks in multiplying, and exp and log are the translators between the two. They can be, because of one fact: adding the logs of two numbers gives the same answer as multiplying the numbers themselves, and exp reverses log. Since the score was built by adding, exp is exactly the tool that turns it back into the multiplying world of probabilities. Any other positive-making function (like absolute value) would break that match. exp is the one that fits how the score was made.

One last footnote: why e, and not 2? The digits of e (2.718…) look arbitrary, and 2 looks clean, so it’s natural to think 2 would be the less arbitrary choice. It’s the other way around. Any base bigger than 1 gives a valid ranking (2^score works fine; switching bases just rescales the scores by a fixed amount the weights absorb during training), so the requirement is only “an exponential.” What makes e special is how it behaves, not how it’s spelled.

Training constantly measures how fast a quantity changes as you nudge a weight (its slope, covered in Part 7). e^x has a unique property: its slope at any point equals its own value, with nothing extra attached. 2^x doesn’t; its slope is 2^x times a fixed but awkward factor (about 0.693) that then tags along through every later calculation. e is defined as exactly the base that makes that factor 1.

So 2 buys you nice-looking digits but a messier slope, while e buys you ugly digits but the cleanest slope. Since the math cares about the slope and not about how pretty the constant looks, e wins. “Raise e to the power” is convention; “use an exponential” is the actual necessity.

The takeaway

Neither formula is the one right answer. Each is the written-down form of a goal:

• Hinge loss = “give me the widest buffer between the correct score and the wrong ones.” The max caps the demand at good-enough; the 1 sets the buffer.
• Softmax loss = “make the correct label as probable as possible.” The exp turns scores into positive, correctly-ordered probability weights; the log turns a product of probabilities into a sum you can add up and minimize.

People do invent other valid losses for other purposes. But once you know the goal each of these two starts from, the formulas look less arbitrary.

Back to the series: Part 7 covers gradient descent, the method that actually reads the loss and adjusts the weights to make it smaller.