LogSumExp explained

The LogSumExp (LSE) (also called RealSoftMax^[1] or multivariable softplus) function is a smooth maximum – a smooth approximation to the maximum function, mainly used by machine learning algorithms.^[2] It is defined as the logarithm of the sum of the exponentials of the arguments:

$$\backslash mathrm(x\_1,\; \backslash dots,\; x\_n)\; =\; \backslash log\backslash left(\backslash exp(x\_1)\; +\; \backslash cdots\; +\; \backslash exp(x\_n)\; \backslash right).$$

Properties

The LogSumExp function domain is

, the real coordinate space, and its codomain is , the real line. It is an approximation to the maximum with the following bounds$$\backslash max\; \backslash leq\; \backslash mathrm(x\_1,\; \backslash dots,\; x\_n)\; \backslash leq\; \backslash max\; +\; \backslash log(n).$$The first inequality is strict unless . The second inequality is strict unless all arguments are equal.(Proof: Let . Then . Applying the logarithm to the inequality gives the result.)

In addition, we can scale the function to make the bounds tighter. Consider the function

. Then (Proof: Replace each with for some in the inequalities above, to giveand, since finally, dividing by gives the result.)

Also, if we multiply by a negative number instead, we of course find a comparison to the

function:

The LogSumExp function is convex, and is strictly increasing everywhere in its domain.^[3] It is not strictly convex, since it is affine (linear plus a constant) on the diagonal and parallel lines:^[4]

Other than this direction, it is strictly convex (the Hessian has rank), so for example restricting to a hyperplane that is transverse to the diagonal results in a strictly convex function. See , below.

Writing

the partial derivatives are:$$\backslash frac\; =\; \backslash frac,$$which means the gradient of LogSumExp is the softmax function.

The convex conjugate of LogSumExp is the negative entropy.

log-sum-exp trick for log-domain calculations

The LSE function is often encountered when the usual arithmetic computations are performed on a logarithmic scale, as in log probability.^[5]

Similar to multiplication operations in linear-scale becoming simple additions in log-scale, an addition operation inlinear-scale becomes the LSE in log-scale:

$$\backslash mathrm(\backslash log(x\_1),\; ...,\; \backslash log(x\_n))\; =\; \backslash log(x\_1\; +\; \backslash dots\; +\; x\_n)$$A common purpose of using log-domain computations is to increase accuracy and avoid underflow and overflow problemswhen very small or very large numbers are represented directly (i.e. in a linear domain) using limited-precisionfloating point numbers.^[6]

Unfortunately, the use of LSE directly in this case can again cause overflow/underflow problems. Therefore, thefollowing equivalent must be used instead (especially when the accuracy of the above 'max' approximation is not sufficient).

$$\backslash mathrm(x\_1,\; \backslash dots,\; x\_n)\; =\; x^*\; +\; \backslash log\backslash left(\backslash exp(x\_1-x^*)+\; \backslash cdots\; +\; \backslash exp(x\_n-x^*)\; \backslash right)$$where

Many math libraries such as IT++ provide a default routine of LSE and use this formula internally.

A strictly convex log-sum-exp type function

LSE is convex but not strictly convex.We can define a strictly convex log-sum-exp type function^[7] by adding an extra argument set to zero:

$$\backslash mathrm\_0^+(x\_1,...,x\_n)\; =\; \backslash mathrm(0,x\_1,...,x\_n)$$This function is a proper Bregman generator (strictly convex and differentiable). It is encountered in machine learning, for example, as the cumulant of the multinomial/binomial family.

In tropical analysis, this is the sum in the log semiring.

See also

• Logarithmic mean
• Log semiring
• Smooth maximum
• Softmax function

Notes and References

1. Web site: Zhang . Aston . Lipton . Zack . Li . Mu . Smola . Alex . Dive into Deep Learning, Chapter 3 Exercises . www.d2l.ai . 27 June 2020 . d2lch3ex.
2. Frank . Nielsen . Ke . Sun . 2016 . Guaranteed bounds on the Kullback-Leibler divergence of univariate mixtures using piecewise log-sum-exp inequalities . 1606.05850 . 10.3390/e18120442 . 18 . Entropy . 12 . 442 . 2016Entrp..18..442N . 17259055 . free .
3. Book: El Ghaoui, Laurent. Optimization Models and Applications. 2017.
4. Web site: convex analysis - About the strictly convexity of log-sum-exp function - Mathematics Stack Exchange. stackexchange.com.
5. Book: McElreath, Richard . . 1107423386.
6. Web site: Practical issues: Numeric stability.. CS231n Convolutional Neural Networks for Visual Recognition.
7. Frank . Nielsen . Gaetan . Hadjeres . 2018 . Monte Carlo Information Geometry: The dually flat case . cs.LG . 1803.07225.