Micro-example — solving a Chebyshev bound for \(k\) My addition

Chebyshev's inequality (§2.2, Thm 2.40) gives \(\P(|X-\E[X]|\ge k)\le \Var[X]/k^2\), where \(\Var[X]\) is the variance and \(k>0\) is a deviation you get to choose. Suppose \(\Var[X]=2.5\) and you want the probability to be at most \(0.1\). Solve

Multiply both sides by \(k^2>0\) (positive, so no flip): \(2.5\le 0.1\,k^2\). Divide by \(0.1>0\) (no flip): \(25\le k^2\). Both sides are \(\ge0\), so take square roots (equivalence): \(5\le k\). Check: \(2.5/5^2 = 2.5/25 = 0.1\) exactly, so \(k=5\) is the smallest \(k\) that works.

One more line of algebra that turns up in the same tasks: because both sides are \(\ge0\),

which is exactly how the proof of Chebyshev turns a two-sided deviation into Markov's inequality applied to a square.

The double inequality — turning a \(z\)-bound into a confidence interval My addition — the algebra Script §3.4, Ex 3.56 leaves out

Every confidence interval [Konfidenzintervall] in §3.4 starts from a statement about the standardised sample mean \(Z=\dfrac{\bar X_n-\mu}{\sigma/\sqrt n}\) — here \(\bar X_n\) is the average of the \(n\) observations, \(\mu\) the unknown mean, \(\sigma\) the known standard deviation — and the task is to get \(\mu\) alone in the middle. The script writes "rearranging the inequality for \(\mu\) yields" and prints the answer; these are the four lines it skips. Start from

1. Multiply all three parts by \(\sigma/\sqrt n\), which is positive, so nothing flips: \(-z\tfrac{\sigma}{\sqrt n}\le \bar X_n-\mu\le z\tfrac{\sigma}{\sqrt n}\).
2. Subtract \(\bar X_n\) from all three parts (adding never flips): \(-\bar X_n-z\tfrac{\sigma}{\sqrt n}\le -\mu\le -\bar X_n+z\tfrac{\sigma}{\sqrt n}\).
3. Multiply by \(-1\). This is the one step that flips both \(\le\) signs, and because the two outer expressions swap ends, the left-hand end of the interval is built from the \(+z\) line and the right-hand end from the \(-z\) line: \(\bar X_n+z\tfrac{\sigma}{\sqrt n}\ \ge\ \mu\ \ge\ \bar X_n-z\tfrac{\sigma}{\sqrt n}\).
4. Read it left to right: \(\bar X_n-z\tfrac{\sigma}{\sqrt n}\ \le\ \mu\ \le\ \bar X_n+z\tfrac{\sigma}{\sqrt n}\), i.e. \(\bar X_n\pm z\tfrac{\sigma}{\sqrt n}\).

Check with numbers (\(\bar x_n=12\), \(\sigma=3\), \(n=36\), \(z=1.96\)): \(\sigma/\sqrt n=0.5\), so the interval is \([11.02,\,12.98]\). Putting \(\mu=11.02\) back into \((\bar x_n-\mu)/(\sigma/\sqrt n)\) gives \(+1.96\) and \(\mu=12.98\) gives \(-1.96\) — the two ends really did swap. ✓

Micro-example and the splitting trick My addition

With \(a=3\) and \(b=-5\): \(|3+(-5)|=|-2|=2\), while \(|3|+|-5|=8\). Indeed \(2\le 8\). The inequality is far from tight when the signs disagree, and is an equality when they agree.

The splitting trick. Suppose \(|a+b|\ge\varepsilon\) for some \(\varepsilon>0\). Then at least one of \(|a|\ge\varepsilon/2\) and \(|b|\ge\varepsilon/2\) must hold. Why: if both failed, then \(|a|<\varepsilon/2\) and \(|b|<\varepsilon/2\), so by the triangle inequality \(|a+b|\le|a|+|b|<\varepsilon/2+\varepsilon/2=\varepsilon\), contradicting the assumption. In event language this is

and then you bound each of the two pieces separately and add the bounds.

Micro-example — sup without max My addition

\(A=\{1-\tfrac1n : n=1,2,3,\dots\}=\{0,\ 0.5,\ 0.6\overline{6},\ 0.75,\ 0.8,\ \dots\}\). Every element is \(<1\), so \(1\) is an upper bound; and no number below \(1\) is an upper bound, because \(1-\tfrac1n\) eventually exceeds it. So \(\sup A=1\), but \(1\notin A\), so \(A\) has no maximum. On the other side \(\inf A = 0 = \min A\), attained at \(n=1\).

Micro-example — the median of a three-point distribution My addition

Let \(\P(X=1)=0.3\), \(\P(X=2)=0.5\), \(\P(X=3)=0.2\). Then \(F_X(x)=0\) for \(x<1\), \(=0.3\) on \([1,2)\), \(=0.8\) on \([2,3)\), \(=1\) from \(3\) on. For the median, \(\alpha=0.5\):

Note that \(F_X\) never takes the value \(0.5\) — it jumps from \(0.3\) straight to \(0.8\). Solving "\(F_X(m)=0.5\)" would have no solution; the infimum definition still returns the sensible answer \(m=2\).

Micro-example — \(\dfrac{n}{n+1}\to 1\) My addition

You cannot apply the quotient rule directly: numerator and denominator both run to \(\infty\). The standard repair is to divide top and bottom by the fastest-growing term, here \(n\):

Now \(\tfrac1n\to0\), so the denominator \(\to 1+0=1\) by the sum rule, and the quotient rule gives \(\tfrac{1}{1}=1\). Numerically at \(n=100\): \(100/101=0.990099\). The same one-line trick gives \(\tfrac{n}{n-1}\to1\) and \(\tfrac{2n-1}{n^2}\to0\).

Warm-up before the rule — just multiply it out My addition

Set \(x=1\) and compute \(\bigl(1+\tfrac1n\bigr)^{n}\) with a calculator: \(n=1\) gives \(2^{1}=2\); \(n=2\) gives \(1.5^{2}=2.25\); \(n=4\) gives \(1.25^{4}=2.44140625\); \(n=12\) gives \(2.6130353\). The numbers climb towards \(2.7182818\) and never pass it — and that number is \(e\). The rule below is that observation, with \(x\) in place of \(1\).

Micro-example — the numbers My addition

\(\lambda=2\), \(n=1000\): \((1-0.002)^{1000}=0.135065\), against \(e^{-2}=0.135335\) — close, and the gap shrinks like \(1/n\).

Warm-up before the DPT version — halves My addition

Add \(\tfrac12+\tfrac14+\tfrac18=\tfrac78=0.875\). Each new term covers half of what is still missing, so after three terms \(\tfrac18\) is missing, after four \(\tfrac1{16}\), and the running total creeps up to \(1\) without ever reaching it.

That is the shifted formula above with \(q=\tfrac12\): \(\sum_{j\ge1}2^{-j}=\tfrac{q}{1-q}=\tfrac{1/2}{1/2}=1\). ✓

Micro-example My addition

\(q=\tfrac12\): \(\sum_{j\ge0}2^{-j}=1+\tfrac12+\tfrac14+\cdots=\tfrac{1}{1-1/2}=2\) (check: the partial sums \(1,1.5,1.75,1.875,\dots\) climb to \(2\)); and \(\sum_{j\ge1}2^{-j}=\tfrac{1/2}{1/2}=1\), which is the previous answer minus the \(j=0\) term. Finite case \(q=\tfrac13\), \(n=4\): \(1+\tfrac13+\tfrac19+\tfrac1{27}=\tfrac{40}{27}=1.4815\), and \(\tfrac{1-(1/3)^4}{1-1/3}=\tfrac{80/81}{2/3}=1.4815\). ✓

Warm-up before the DPT version — four terms of \(e\) My addition

Take \(x=1\) and add the first four terms by hand: \(1+1+\tfrac12+\tfrac16=2.6666667\), against \(e=2.7182818\).

The next term \(\tfrac1{24}\) lifts it to \(2.7083333\), the one after that \(\tfrac1{120}\) to \(2.7166667\). The terms shrink fast because the denominator \(k!\) grows faster than anything else in sight. The formula below is this sum carried out forever, with \(x\) in place of \(1\).

Micro-example — why the Poisson pmf sums to \(1\) My addition

The \(\Poi(\lambda)\) probability mass function is \(p(k)=e^{-\lambda}\lambda^{k}/k!\) for \(k=0,1,2,\dots\). Summing:

The constant \(e^{-\lambda}\) came out of the sum because it does not depend on \(k\); then the remaining sum is the exponential series with \(x=\lambda\). Numerical check at \(\lambda=3\), summing \(k=0\) to \(59\): the total is \(1.0000000\).

Micro-example — the three rules in one line each My addition

• \(8^{1/3}=e^{\frac13\ln 8}=e^{\frac13\cdot2.0794}=e^{0.6931}=2\). ✓
• \(\bigl((2\pi s^2)^{-1/2}\bigr)^{3}=(2\pi s^2)^{-3/2}\): with \(s^2=4\), the left side is \(0.19947^{3}=0.0079367\) and the right side is \((25.13)^{-1.5}=0.0079367\). ✓ This is exactly the step that turns \(n\) copies of a normal density into the factor \((2\pi\sigma^2)^{-n/2}\) in a likelihood.
• \(\dfrac{e^{-\lambda(t+s)}}{e^{-\lambda s}}=e^{-\lambda(t+s)+\lambda s}=e^{-\lambda t}\).

DPT micro-example 1 — memorylessness WiSe25 · P3b

\(X\sim\Exp(\lambda)\) has survival function \(\P(X\ge t)=e^{-\lambda t}\) for \(t\ge0\). Asked for \(\P(X\ge 10\mid X\ge 5)\), the conditional probability \(\P(A\mid B)=\P(A\cap B)/\P(B)\) (Def 1.69). Since \(\{X\ge10\}\subseteq\{X\ge5\}\), the intersection is \(\{X\ge10\}\), so

The whole computation is the single rule \(e^{a}/e^{b}=e^{a-b}\). Check with \(\lambda=0.2\): \(e^{-2}/e^{-1}=0.13534/0.36788=0.36788=e^{-1}\). ✓ The answer is \(p\), the number you were given — which is the point of the task: the exponential forgets how long it has already waited (§1.4).

DPT micro-example 2 — simplifying a likelihood's logarithm SoSe25 · P4a

\(X_1,\dots,X_n\) are iid \(\Poi(\lambda)\) ("iid" = independent and identically distributed). The likelihood is the joint pmf read as a function of \(\lambda\); independence turns it into a product:

Step (1) is three exponent rules at once: \(n\) copies of \(e^{-\lambda}\) multiply to \(e^{-\lambda}\cdots e^{-\lambda}=e^{-n\lambda}\) by \(e^{a}e^{b}=e^{a+b}\); the powers of \(\lambda\) add their exponents; the factorials just collect. Now take \(\ln\), using \(\ln(xy)=\ln x+\ln y\), \(\ln(x/y)=\ln x-\ln y\) and \(\ln(x^{\alpha})=\alpha\ln x\):

The product has become a sum, and the last term does not contain \(\lambda\) at all, so it will vanish when you differentiate. That is the whole reason for taking the logarithm. Check: maximising \(\ell\) numerically over \(\lambda\) for the data \(2,3,5,4,1\) gives \(\hat\lambda=3.0000\), which is \(\bar x=15/5=3\). ✓ Continue at finding the maximum and §3.2.

DPT micro-example 3 — a log-likelihood with a parameter in the denominator WiSe25 · P4a

Rainfall model: \(p_\theta(x)=\theta^{-2}\,x\,e^{-x/\theta}\) for \(x>0\), \(\theta>0\). For iid data \(x_1,\dots,x_n\),

Three rules were used inside the logarithm: \(\ln(abc)=\ln a+\ln b+\ln c\), \(\ln(\theta^{-2})=-2\ln\theta\), and \(\ln(e^{-x/\theta})=-x/\theta\). Differentiating (next section) and setting the result to zero gives \(-\tfrac{2n}{\theta}+\tfrac{1}{\theta^{2}}\sum_i x_i=0\), hence \(\hat\theta=\tfrac{1}{2n}\sum_i x_i\). Check with data \(1.2,\,3.4,\,2.0,\,5.1\): a numerical search over \(\theta\) maximises \(\ell\) at \(1.46249\), and the formula gives \(11.7/8=1.4625\). ✓

DPT micro-example 4 — the median of \(\Exp(\lambda)\) Script §2.1, Ex 2.21

\(X\sim\Exp(\lambda)\) has cdf \(F(x)=1-e^{-\lambda x}\) for \(x\ge0\). The median \(m\) solves \(F(m)=\tfrac12\):

Line 2 → line 3 took \(\ln\) of both sides and used \(\ln(e^{a})=a\) on the left; line 3 → line 4 divided by \(-\lambda\), which is a negative number, but this is an equation, not an inequality, so nothing flips. Check at \(\lambda=2\): \(m=0.693/2=0.34657\), and \(F(0.34657)=1-e^{-0.69315}=0.5\) exactly. ✓ At \(\lambda=\tfrac12\): \(m=1.38629\), and \(F(m)=0.5\). ✓

Worth memorising as a fact: the median of \(\Exp(\lambda)\) is \(\ln 2/\lambda\), which is smaller than the mean \(1/\lambda\) because \(\ln 2=0.693<1\). The distribution is right-skewed, so its mean sits to the right of its median.

DPT micro-example 5 — inverting a cdf (the quantile transform) WiSe25 · P3a

\(U\sim\Unif(0,1)\), and \(X:=-\tfrac1\lambda\ln U\). Find the distribution of \(X\). Compute the cdf directly, for \(x\ge0\):

Step (1): multiply both sides of the inequality inside by \(-\lambda<0\), so the inequality flips — this is the one place marks are lost. Step (2): apply \(\exp\), which is strictly increasing, so the direction is preserved, and \(e^{\ln U}=U\). Step (3): for \(U\sim\Unif(0,1)\), \(\P(U\ge u)=1-u\) whenever \(u\in[0,1]\), and \(e^{-\lambda x}\in(0,1]\) for \(x\ge0\). The answer \(1-e^{-\lambda x}\) is the cdf of \(\Exp(\lambda)\), so \(X\sim\Exp(\lambda)\). Differentiating gives the density \(p_X(x)=\lambda e^{-\lambda x}\) on \((0,\infty)\) and \(0\) elsewhere — and stating the support is part of the answer on that exam task.

Check with numbers (\(\lambda=1.5\), \(u=0.3\)): \(x=-\ln(0.3)/1.5=0.80265\), and \(F_X(0.80265)=1-e^{-1.5\cdot0.80265}=1-0.3=0.7\). That is exactly \(\P(U\ge0.3)\). ✓

DPT micro-example 6 — an integral you get for free WiSe25 · P3c

With \(X=-\tfrac1\lambda\ln U\) as above and \(\E[X]=1/\lambda\) known for \(\Exp(\lambda)\), the change-of-variables formula for expectations (LOTUS, Lemma 2.2) says \(\E[g(U)]=\int_0^1 g(u)\,\mathrm du\) for \(U\sim\Unif(0,1)\). With \(g(u)=-\tfrac1\lambda\ln u\):

Check: the antiderivative of \(\ln u\) is \(u\ln u-u\), so \([u\ln u-u]_0^1=(0-1)-(0-0)=-1\); numerical integration gives \(-1.0000000\). ✓ The task is a probability fact used backwards to evaluate an integral (§2.1).

Two worked instances Sheet 5 · Ex 1(iv),(v); Sheet 7 · Ex 4(i)

(a) The 5 % value at risk. A return \(R\) has \(F_R(r)=(r+3)^{2}/15\) on \([-3,0)\). Solve \((r+3)^{2}/15=0.05\), so \((r+3)^{2}=0.75\) and \(r=-3\pm\sqrt{0.75}\): the roots are \(-2.1339746\) and \(-3.8660254\), and the second is outside \([-3,0)\). So \(r_{0.05}=-2.1339746\) and the loss is \(2.134\). Check: \((-2.1339746+3)^{2}/15=0.0500000\) ✓. The median is the same with \(0.5\): \(r_{0.5}=-3+\sqrt{7.5}=-0.2613872\), \(F_R=0.5000000\) ✓.

(b) Inverting \(F(x)=2x-x^{2}\) on \([0,1]\). Setting \(F(x)=u\) gives \(x^{2}-2x+u=0\). Complete the square (\(b=-2\), so \(b/2=-1\)): \((x-1)^{2}-1+u=0\), i.e. \((x-1)^{2}=1-u\), so \(x=1\pm\sqrt{1-u}\). The \(+\) root lies in \((1,2)\), outside \([0,1]\), so \(x=1-\sqrt{1-u}\). Check at \(u=0.36\): \(x=0.2000000\) and \(F(0.2)=0.3600000\) ✓.

Two habits these drill. Decide which branch you are in before solving: in (a), \(F_R(0)=9/15=0.6>0.05\), so the quantile is in the first branch. And when a cdf has several branches, each starts from the value already accumulated: on \([0,2)\) it is \(F_R(0)+\int_0^{r}\tfrac15(2-t)\mathrm dt=\tfrac{6+4r-r^{2}}{10}\), not just the integral. Check the joins — at \(r=0\) both branches give \(0.6000000\) ✓ and at \(r=2\) the last gives \(1.0000000\) ✓ (Sheet 5 Ex 1(ii)). Inverse transform sampling: Sheet 7 Ex 4, §1.3; quantiles: §2.1.

Micro-example 1 — differentiate the exponential cdf Script §1.3, Ex 1.61(iv)

\(F(x)=1-e^{-\lambda x}\) for \(x\ge0\), with \(\lambda>0\) a fixed parameter. Differentiate term by term (linearity): the \(1\) gives \(0\), and \(-e^{-\lambda x}\) is \(-1\) times \(e^{g(x)}\) with \(g(x)=-\lambda x\), \(g'(x)=-\lambda\). So

That is exactly the \(\Exp(\lambda)\) density. The two minus signs cancelling is the whole content of the calculation, and dropping one of them gives a "density" that is negative everywhere. Check (\(\lambda=1.4\), \(x=0.7\)): the numerical slope of \(F\) there is \(0.5254355\), and \(1.4\,e^{-0.98}=0.5254355\). ✓

Micro-example 2 — the product rule on \(x\mapsto x e^{-x}\) My addition

This is the \(\mathrm{Gamma}(2,1)\) density shape, and it is also the function whose maximum you find when you locate the mode of that distribution. Set \(f(x)=x\) and \(g(x)=e^{-x}\), so \(f'=1\) and \(g'=-e^{-x}\):

Factoring out \(e^{-x}\) in the last step is legitimate because \(e^{-x}\) is never \(0\). Since \(e^{-x}>0\) always, the sign of the derivative is the sign of \(1-x\): positive for \(x<1\), negative for \(x>1\). So the function rises, then falls, and its highest point is at \(x=1\). Check: the numerical slope at \(x=0.4\) is \(0.4021920\), and \((1-0.4)e^{-0.4}=0.4021920\); a numerical maximiser puts the peak at \(x=1.000001\). ✓ That sign argument is the same one you will use for maximum likelihood, at finding a maximum.

Micro-example 3 — the quotient rule My addition

\(h(x)=\dfrac{x}{1+x^{2}}\) — up to a constant this is the Cauchy density's cousin, and the same shape turns up whenever a density has a polynomial on top and a polynomial underneath. With \(f(x)=x\) (\(f'=1\)) and \(g(x)=1+x^{2}\) (\(g'=2x\)):

Check at \(x=2\): the formula gives \((1-4)/25=-0.12\), and the numerical slope is \(-0.1200000\). ✓

Micro-example 4 — the normal density Script §1.3, Ex 1.61(ii)

Write \(\varphi(z)=\tfrac{1}{\sqrt{2\pi}}e^{-z^{2}/2}\) for the standard normal density (the \(\mathcal N(0,1)\) bell curve). The factor \(\tfrac1{\sqrt{2\pi}}\) is a constant, so it just rides along. Chain rule with \(g(z)=-z^{2}/2\) and \(g'(z)=-z\):

Two facts you get for free. \(\varphi'(z)=0\) exactly at \(z=0\), which is the peak; and \(\varphi'\) changes sign from \(+\) to \(-\) there, so the peak is a maximum. Differentiating once more (product rule on \(-z\cdot\varphi(z)\)) gives \(\varphi''(z)=(z^{2}-1)\varphi(z)\), which is \(0\) at \(z=\pm1\): the two inflection points of the bell curve sit one standard deviation from the centre. For \(\mathcal N(\mu,\sigma^{2})\) they sit at \(\mu\pm\sigma\) (§1.3, the zoo).

Micro-example 5 — differentiating a log-likelihood in \(\theta\) WiSe25 · P4a

Here the variable you differentiate with respect to is the parameter \(\theta\), not the data. The data \(x_1,\dots,x_n\) are fixed numbers while you differentiate. From the logarithm rules, the log-likelihood of the rainfall model was

Differentiate term by term. The first term: \(\tfrac{\mathrm d}{\mathrm d\theta}\ln\theta=\tfrac1\theta\), so it contributes \(-\tfrac{2n}{\theta}\). The middle term contains no \(\theta\), so it contributes \(0\) — this is why dropping it early is safe and why you say so in the exam. The last term is \(-s\cdot\theta^{-1}\) with \(s:=\sum_i x_i\) a constant, and \((\theta^{-1})'=-\theta^{-2}\), so it contributes \(+s\theta^{-2}\). Together:

The last step put both terms over the common denominator \(\theta^{2}\), which is the form you want: the denominator is positive for every \(\theta>0\), so the sign of \(\ell'\) is the sign of \(s-2n\theta\). That single observation is what turns a local maximum into a global one at the next section. Check with data \(1.2,\,3.4,\,2.0,\,5.1\) (\(n=4\), \(s=11.7\)): a numerical maximiser of \(\ell\) returns \(\theta=1.4625003\), and \(s/(2n)=11.7/8=1.4625\). ✓

Micro-example 6 — reading a moment off a characteristic function SoSe26 · P4c

A characteristic function is \(\varphi_X(t)=\E[e^{itX}]\); the fact you need here is \(\E[X^{n}]=i^{-n}\varphi_X^{(n)}(0)\), where \(\varphi^{(n)}\) is the \(n\)-th derivative and \(i^{2}=-1\) (Rem 2.47(iii), §2.3). On that exam \(\varphi_{Z_n}(t)=\cos^{n}(t/\sqrt n)\); take \(n=1\), so \(\varphi(t)=\cos t\). Using the two new table rows,

Hence \(\E[X]=\varphi'(0)/i=0/i=0\) and \(\E[X^{2}]=\varphi''(0)/i^{2}=(-1)/(-1)=1\), so \(\Var[X]=1-0^{2}=1\). That is the alternative route the official solution accepted for full marks. Check: the numerical second difference of \(\cos\) at \(0\) gives \(-1.0000001\) ✓; the numerical slope of \(\cos\) at \(0.9\) is \(-0.7833269=-\sin(0.9)\) ✓.

Warm-up before the exam versions — a straight line on a uniform My addition

\(X\sim\Unif(0,1)\): density \(p_X(x)=1\) for \(x\) between \(0\) and \(1\), and \(0\) outside. Let \(Y=g(X)=3X+2\). Run the four steps on numbers small enough to check by eye.

1. Invert: \(y=3x+2\). Subtract \(2\): \(y-2=3x\). Divide by \(3\): \(x=\tfrac{y-2}{3}\). So \(g^{-1}(y)=\tfrac{y-2}{3}\).
2. Differentiate in \(y\): \(\tfrac{\mathrm d}{\mathrm dy}\tfrac{y-2}{3}=\tfrac13\) — the slope of a straight line is the same everywhere.
3. Absolute value: \(\bigl|\tfrac13\bigr|=\tfrac13\).
4. Support: \(x=0\) gives \(y=2\), \(x=1\) gives \(y=5\), and \(3x+2\) increases, so the whole of \([0,1]\) lands on \([2,5]\).

Put the pieces together: \(p_Y(y)=p_X\bigl(\tfrac{y-2}{3}\bigr)\cdot\tfrac13=1\cdot\tfrac13=\tfrac13\) for \(y\) in \([2,5]\), and \(0\) outside. Check without any formula: stretching an interval of length \(1\) into one of length \(3\) must divide the height by \(3\) to keep the area at \(1\), and \(3\cdot\tfrac13=1\) ✓. That division by the stretch factor is the \(|(g^{-1})'|\) in the formula — the two exam versions below only replace the straight line by a curve, so that the stretch factor changes from place to place.

Worked, \(g\) increasing — \(Y=\ln X\) SoSe25 · P3d

\(X\) has the density \(p_X(x)=x/2\) on \([0,2]\) and \(0\) elsewhere (that is part (a) of the same problem, worked in the integration section). Let \(Y=g(X)=\ln X\).

1. Invert: \(y=\ln x \iff x=e^{y}\), so \(g^{-1}(y)=e^{y}\).
2. Differentiate: \(\tfrac{\mathrm d}{\mathrm dy}e^{y}=e^{y}\).
3. Absolute value: \(|e^{y}|=e^{y}\), since \(\exp\) is positive everywhere.
4. Support: \(\ln\) is strictly increasing, so it maps \(x\in(0,2]\) to \(y\in(-\infty,\ln 2]\); the left end goes to \(-\infty\) because \(\ln x\to-\infty\) as \(x\downarrow0\).

Substituting \(x=e^{y}\) into \(p_X(x)=x/2\) gives \(p_X(e^{y})=e^{y}/2\), and the formula is

Checks: \(\int_{-\infty}^{\ln 2}\tfrac12 e^{2y}\,\mathrm dy=1.0000000\) ✓ (a density must integrate to \(1\), and this is the fastest way to catch a lost factor). Simulating \(400\,000\) draws of \(X\) and taking logarithms gives \(\P(Y\le-0.5)=0.0916\) against the formula's \(\int_{-\infty}^{-0.5}\tfrac12e^{2y}\mathrm dy=0.0919699\) ✓, and a sample mean \(0.1936\) against \(\int y\,p_Y(y)\,\mathrm dy=0.1931472=\ln2-\tfrac12\) ✓.

Worked, \(g\) decreasing — \(X=-\tfrac1\lambda\ln U\) WiSe25 · P3a

\(U\sim\Unif(0,1)\), so \(p_U(u)=1\) on \((0,1)\) and \(0\) elsewhere. The transformation is \(x=g(u)=-\tfrac1\lambda\ln u\), which is strictly decreasing in \(u\).

1. Invert: \(x=-\tfrac1\lambda\ln u \iff -\lambda x=\ln u \iff u=e^{-\lambda x}\). So \(g^{-1}(x)=e^{-\lambda x}\).
2. Differentiate: \(\tfrac{\mathrm d}{\mathrm dx}e^{-\lambda x}=-\lambda e^{-\lambda x}\) (chain rule, inner derivative \(-\lambda\)) — negative, exactly because \(g\) decreases.
3. Absolute value: \(\bigl|-\lambda e^{-\lambda x}\bigr|=\lambda e^{-\lambda x}\). Without the bars you would report a negative density.
4. Support: \(u\in(0,1)\) gives \(\ln u\in(-\infty,0)\), and multiplying by \(-\tfrac1\lambda<0\) flips that to \(x\in(0,\infty)\). So the new support is \(x\ge0\).

And \(p_U(g^{-1}(x))=p_U(e^{-\lambda x})=1\), because \(e^{-\lambda x}\in(0,1)\) for \(x>0\). So

— the \(\Exp(\lambda)\) density, which matches the cdf route done at micro-example 5. Check (\(\lambda=1.5\), \(x=0.8\)): \(\lambda e^{-\lambda x}=0.4517913\), and scipy's \(\Exp(1.5)\) density at \(0.8\) is \(0.4517913\) ✓.

Warm-up before the likelihoods — a parabola, and where its slope is \(0\) My addition

Forget probability for a moment and maximise \(f(x)=-x^{2}+6x-4\), a downward parabola. The derivative \(f'(x)\) is the slope of the graph at \(x\); differentiating term by term with the power rule \((x^{n})'=nx^{n-1}\) gives \(f'(x)=-2x+6\).

Set it to zero: \(-2x+6=0\), add \(2x\) to both sides to get \(6=2x\), divide by \(2\) to get \(x=3\). Is it a maximum? Differentiate once more: \(f''(x)=-2\), which is negative for every \(x\), so the graph bends downwards everywhere and \(x=3\) is the top. The value there is \(f(3)=-9+18-4=5\).

Check by hand at neighbouring points: \(f(2)=-4+12-4=4\) and \(f(4)=-16+24-4=4\) — both below \(5\), with the slope \(f'(2)=+2\) (still climbing) on the left and \(f'(4)=-2\) (already falling) on the right. That sign change from \(+\) to \(-\) is the sign-change test of step 5, run on a function small enough to see. Everything below is this same calculation with \(\ell\) in place of \(f\) and \(\theta\) in place of \(x\).

Worked example 1 (kind A) — the rainfall mle, with a global argument WiSe25 · P4a

Model: \(p_\theta(x)=\theta^{-2}x e^{-x/\theta}\) for \(x>0\), parameter \(\theta>0\). From micro-example 5, with \(s:=\sum_{i=1}^{n}x_i\),

Set to zero. A fraction is \(0\) exactly when its numerator is \(0\) and its denominator is not. Here \(\theta^{2}>0\) for every \(\theta>0\), so \(\ell'(\theta)=0 \iff s-2n\theta=0 \iff \hat\theta=\dfrac{s}{2n}=\dfrac{\bar x_n}{2}\), where \(\bar x_n=\tfrac1n\sum_i x_i\) is the sample mean.

Maximum, locally. Differentiate \(\ell'(\theta)=-2n\theta^{-1}+s\theta^{-2}\) once more, using \((\theta^{-1})'=-\theta^{-2}\) and \((\theta^{-2})'=-2\theta^{-3}\): \(\ell''(\theta)=2n\theta^{-2}-2s\theta^{-3}=\dfrac{2n\theta-2s}{\theta^{3}}\). At \(\theta=\hat\theta=s/(2n)\) the numerator is \(2n\cdot\tfrac{s}{2n}-2s=s-2s=-s<0\) and the denominator is positive, so \(\ell''(\hat\theta)<0\). That is a local maximum.

Maximum, globally — the extra line the official solution omits. \(\ell'(\theta)\) has the positive denominator \(\theta^{2}\), so its sign is the sign of \(s-2n\theta\), which is \(>0\) for \(\theta<\hat\theta\) and \(<0\) for \(\theta>\hat\theta\). So \(\ell\) increases on \((0,\hat\theta)\) and decreases on \((\hat\theta,\infty)\): the maximum is global.

Write that sentence. It is one line, it costs nothing, and the exam's own solution has the gap — see TT5, step 4. Check (data \(1.2,3.4,2.0,5.1\)): numerical maximisation gives \(1.4625003\); the formula gives \(11.7/8=1.4625\). ✓

Worked example 2 (kind B) — a maximum at the boundary Sheet 10 · Ex 1 and Ex 3

The uniform model. \(X_1,\dots,X_n\) independent and \(\Unif(0,\theta)\)-distributed: the density is \(p_\theta(x)=\tfrac1\theta\) for \(x\in[0,\theta]\) and \(0\) outside. The likelihood is

Where the case split comes from. If \(\theta<\max_i x_i\), then at least one observation lies outside \([0,\theta]\), its density is \(0\), and one zero factor kills the whole product.

Why the derivative recipe fails here. On the region where \(L\) is not \(0\), \(L(\theta)=\theta^{-n}\) and \(L'(\theta)=-n\theta^{-n-1}\), which is strictly negative for every \(\theta>0\) — it is never \(0\), so there is no critical point at all. Taking logarithms does not help either, because \(\ln L=-n\ln\theta\) is undefined on the branch where \(L=0\).

The argument that works. \(L\) is \(0\) to the left of \(\max_i x_i\) and strictly decreasing to the right of it. A function that is strictly decreasing on \([\,m,\infty)\) takes its largest value at the left endpoint \(m\). Hence

Check (data \(0.3,\,1.7,\,0.9,\,1.55\)): scanning \(L\) on a fine grid over \([0.5,5]\) peaks at \(\theta=1.70002\), and \(\max_i x_i=1.7\). ✓

The mirror image, Sheet 10 Ex 3. \(p_X(x)=2c^{2}/x^{3}\) for \(x\ge c\), \(c>0\). Now \(L(c)=\bigl(\prod_i 2/x_i^{3}\bigr)\,c^{2n}\) whenever \(c\le\min_i x_i\), and \(0\) otherwise. The factor \(c^{2n}\) is strictly increasing in \(c\), so \(L\) rises until it hits the wall at \(\min_i x_i\) and drops to \(0\) after it: \(\hat c=\min\{x_1,\dots,x_n\}\). The exercise's own hint — "consider the case where a sample value is less than \(c\)" — is pointing at exactly this wall. Check (data \(3.1,\,2.6,\,4.8,\,2.9\)): the grid maximum is at \(c=2.59999\), and \(\min_i x_i=2.6\). ✓

Warm-up before the DPT example — a rectangle My addition

Take the flat function \(p(x)=c\) on \([0,2]\). Its graph encloses a rectangle of width \(2\) and height \(c\), so the area is \(\int_0^2 c\,\mathrm dx=2c\) — no calculus needed, it is width times height.

"Choose \(c\) so that the density integrates to \(1\)" then reads \(2c=1\), so \(c=\tfrac12\), which is the \(\Unif(0,2)\) density. The example below is the same question with a sloping roof instead of a flat one, so the area needs the power rule instead of multiplication.

Worked example — a whole density problem, start to finish SoSe25 · P3a–c

\(p_X(x)=\dfrac{x}{c}\) for \(x\in[0,2]\) and \(0\) otherwise, for some constant \(c\in\R\).

(a) Find \(c\). Impose "total area \(=1\)", integrating over the support \([0,2]\) only, and pull the constant \(1/c\) out by linearity:

Then add the non-negativity line, which the official solution writes out as well: with \(c=2>0\) and \(x\ge0\) on \([0,2]\), \(p_X(x)=x/2\ge0\), so it is a valid density. Check: \(\int_0^2 x/2\,\mathrm dx=1.0000000\). ✓

(b) The cdf, all three branches. \(F_X(x)=\int_{-\infty}^{x}p_X(t)\,\mathrm dt\). Below the support there is nothing to collect; above it everything has been collected; in between, integrate from the left end of the support:

All three branches are required for full credit. Two checks that cost five seconds each: \(F_X(2)=4/4=1\) ✓, and differentiating the middle branch gives \(2x/4=x/2=p_X(x)\) ✓.

(c) Mean and median. The mean weights each \(x\) by its density:

The median \(m\) solves \(F_X(m)=\tfrac12\), so \(m^{2}/4=\tfrac12\), \(m^{2}=2\), \(m=\sqrt2\approx1.4142\) — and the root \(-\sqrt2\) is discarded because it is outside the support \([0,2]\). Checks: numerical integration gives \(\E[X]=1.3333333\) ✓, and \(F_X(\sqrt2)=2/4=0.5\) ✓. Note \(\E[X]=1.333<1.414=m\): this density leans to the right, so its balance point sits to the left of its middle.

Convergent case — a heavy-ish tail that still normalises Sheet 10 · Ex 3(i)

\(p_X(x)=\dfrac{2c^{2}}{x^{3}}\) for \(x\ge c\), with \(c>0\). The tail is a power, so use Rule 1 with \(p=3>1\):

The step \(\int x^{-3}=\tfrac{x^{-2}}{-2}\) is the power rule with \(a=-3\), and \(b^{-2}\to0\) is Rule 1 doing its work. Check at \(c=2.5\): the numerical value is \(1.0000000\) ✓, and the mean \(\int_c^{\infty}x\,p_X(x)\,\mathrm dx=5.0000000=2c\) ✓ (here the integrand behaves like \(1/x^{2}\), still \(p>1\), so the mean exists).

Warm-up before the DPT substitutions — small numbers My addition

Compute \(\int_0^1(2x+1)^{2}\,\mathrm dx\) with \(u=2x+1\). Then \(\mathrm du=2\,\mathrm dx\), so \(\mathrm dx=\tfrac12\,\mathrm du\), and both limits move: \(x=0\mapsto u=1\) and \(x=1\mapsto u=3\).

\(\displaystyle\int_0^1(2x+1)^{2}\mathrm dx=\frac12\int_1^{3}u^{2}\,\mathrm du =\frac12\Bigl[\frac{u^{3}}{3}\Bigr]_1^{3}=\frac12\cdot\frac{27-1}{3}=\frac{13}{3}\). Check by multiplying out instead: \(\int_0^1(4x^{2}+4x+1)\mathrm dx=\tfrac43+2+1=\tfrac{13}3 =4.3333333\) ✓ — same answer, more work.

Substitution examples My addition

S1 — the exponential normalisation, \(u=\lambda x\). With \(\mathrm du=\lambda\,\mathrm dx\), the limits \(0\mapsto0\) and \(\infty\mapsto\infty\):

The factor \(\lambda\) in front of the density is precisely the \(\mathrm du/\mathrm dx\) the substitution needs — which is a good way to remember why \(\Exp(\lambda)\) has that constant. Check (\(\lambda=1.7\)): \(1.0000000\). ✓

S2 — standardising a normal, \(z=\dfrac{x-\mu}{\sigma}\). This is the substitution of the second half of the course: it turns any \(\mathcal N(\mu,\sigma^{2})\) into \(\mathcal N(0,1)\), so that a single table of \(\Phi\)-values serves every normal distribution. Here \(\mathrm dz=\tfrac{1}{\sigma}\mathrm dx\), so \(\mathrm dx=\sigma\,\mathrm dz\), and the limits \(-\infty,\infty\) are unchanged:

The \(\sigma\) from \(\mathrm dx\) cancels the \(\sigma\) in the constant — that cancellation is the whole reason the density is written with \(1/\sigma\) in front. Check (\(\mu=3.2\), \(\sigma=1.8\)): the integral is \(1.0000000\) ✓ and \(\int x\,p(x)\,\mathrm dx=3.2000000=\mu\) ✓. The probability version of the same move is \(\P(X\le x)=\Phi\!\bigl(\tfrac{x-\mu}{\sigma}\bigr)\), used in every clt task (TT8).

S3 — \(\displaystyle\int x e^{-x^{2}/2}\,\mathrm dx\), \(u=x^{2}/2\). The inside of the exponential is \(x^{2}/2\); its derivative is \(x\); and an \(x\) is sitting right there as a factor. So \(\mathrm du=x\,\mathrm dx\) and

Over a symmetric interval this integral is \(0\), because \(x e^{-x^{2}/2}\) is an odd function: \(\int_{-\infty}^{\infty}z e^{-z^{2}/2}\mathrm dz=0.0000000\) ✓, which says \(\E[\mathcal N(0,1)]=0\). Keep the antiderivative anyway — example P2 below needs it.

S4 — turning an exponential into a uniform. Let \(X\sim\Exp(\lambda)\) and set \(U:=e^{-\lambda X}\). For \(u\in(0,1)\), using that \(t\mapsto e^{-\lambda t}\) is decreasing so the inequality flips,

which is the cdf of \(\Unif(0,1)\). So \(U\sim\Unif(0,1)\). This is the inverse of the transformation in micro-example 5 of part 1, and the same substitution written at the level of integrals reads \(\int_0^{\infty}h(e^{-\lambda x})\,\lambda e^{-\lambda x}\,\mathrm dx =\int_0^{1}h(u)\,\mathrm du\), with \(u=e^{-\lambda x}\) and \(\mathrm du=-\lambda e^{-\lambda x}\mathrm dx\) absorbing the sign into the swapped limits. Check: simulating \(400\,000\) draws with \(\lambda=1.3\) gives \(U\) a sample mean \(0.5005\) and sample variance \(0.0833\), against \(\tfrac12=0.5\) and \(\tfrac1{12}=0.0833\) for \(\Unif(0,1)\). ✓

Integration-by-parts examples My addition

P1 — the mean of \(\Exp(\lambda)\). Take \(u=x\) (so \(u'=1\)) and \(v'=\lambda e^{-\lambda x}\) (so \(v=-e^{-\lambda x}\)):

The boundary term vanishes at the top because the exponential beats the factor \(x\), and at the bottom because of the factor \(x=0\). The sign of the second term became \(+\) because the rule's minus met the minus inside \(v\). Check (\(\lambda=2.2\)): \(0.4545455=1/2.2\). ✓

P2 — the variance of \(\mathcal N(0,1)\) Sheet 8 · Ex 5. The task is \(\int_{-\infty}^{\infty}x^{2}e^{-x^{2}/2}\,\mathrm dx\). Split the \(x^{2}\): take \(u=x\) and \(v'=x e^{-x^{2}/2}\), whose antiderivative is \(v=-e^{-x^{2}/2}\) by example S3 above. Then

Dividing by \(\sqrt{2\pi}\) gives \(\E[Z^{2}]=1\), and since \(\E[Z]=0\) that is \(\Var[Z]=1\). Check: \(\int_\R x^{2}e^{-x^{2}/2}\mathrm dx=2.5066283=\sqrt{2\pi}\) ✓ and \(\int_\R z^{2}\varphi(z)\,\mathrm dz=1.0000000\) ✓. Choosing \(u=x^{2}\), \(v'=e^{-x^{2}/2}\) instead leads nowhere, because \(e^{-x^{2}/2}\) has no antiderivative in elementary functions — that choice is the trap.

P3 — a conditional expectation Sheet 7 · Ex 1. With \(u=y\), \(v'=e^{-y}\), \(v=-e^{-y}\):

The boundary term at the lower limit contributes \(+x e^{-x}\) because \(-(-x e^{-x})=x e^{-x}\). Checks: at \(x=0\) the value is \(1.0000000\) versus \((0+1)e^{0}=1\) ✓; at \(x=1\), \(0.7357589\) versus \(2e^{-1}=0.7357589\) ✓; at \(x=2.5\), \(0.2872975\) versus \(3.5\,e^{-2.5}\) ✓.

P4 — the tail formula \(\E[X]=\int_0^{\infty}\P(X>x)\,\mathrm dx\), for \(X\ge0\) with density \(p\) and cdf \(F\). Take \(u=x\) and \(v'=p(x)\). The clever choice is \(v(x)=-\bigl(1-F(x)\bigr)=-\P(X>x)\): this is a legitimate antiderivative of \(p\), because \(\tfrac{\mathrm d}{\mathrm dx}\bigl[-(1-F(x))\bigr]=F'(x)=p(x)\), and any antiderivative may be used. Then

The discrete twin, \(\E[X]=\sum_{k\ge1}\P(X\ge k)\) for \(X\) with values in \(\N_0\), is the same identity with a sum in place of the integral, and it is the fast route to \(\E[X]\) for the geometric distribution (§2.1).

P5 — \(\int\ln x\,\mathrm dx\), the "multiply by 1" trick. There is only one factor, so invent a second: \(u=\ln x\) (\(u'=1/x\)) and \(v'=1\) (\(v=x\)). Then

so the antiderivative of \(\ln x\) is \(x\ln x-x+C\). Checks: \(\int_1^{e}\ln x\,\mathrm dx =1.0000000\) ✓, and \(\int_0^{1}\ln u\,\mathrm du=-1.0000000\) ✓ — the second of these is the integral that WiSe25 · P3c obtains by a probabilistic argument instead (§2.1).

Warm-up before the exam line — what \(2\Phi(-1)\) is, and why the halves are equal My addition

Take \(Z\sim\mathcal N(0,1)\) and \(a=1\). The number \(\Phi(-1)\) is, by the definition of a cdf, the probability \(\P(Z\le-1)\): the area under the bell curve to the left of \(-1\). A table (or scipy) gives \(\Phi(-1)=0.1586553\).

Why the right tail has the same area. The bell curve \(\varphi(z)=\tfrac1{\sqrt{2\pi}}e^{-z^{2}/2}\) contains \(z\) only through \(z^{2}\), and \((-z)^{2}=z^{2}\), so \(\varphi(-z)=\varphi(z)\): the picture is the mirror image of itself in the vertical axis. Mirroring turns "everything left of \(-1\)" into "everything right of \(+1\)" and does not change areas. So \(\P(Z\ge1)=\P(Z\le-1)=0.1586553\) too.

The event \(\{|Z|\ge1\}\) means "\(Z\le-1\) or \(Z\ge1\)". Those two pieces do not overlap, so their probabilities add:

Check the other way round: \(1-\Phi(1)=1-0.8413447=0.1586553\), so \(2\bigl(1-\Phi(1)\bigr)=0.3173105\) — the same number, which is the identity \(\Phi(-a)=1-\Phi(a)\) at \(a=1\). And the middle piece is the leftover: \(\P(|Z|\le1)=1-0.3173105=0.6826895\), the "about 68% within one standard deviation" fact.

Worked — the exam's own line WiSe25 · P4c

There \(\hat\theta_{\mathrm{mle}}\) has \(\E[\hat\theta]=\theta\) and \(\Var[\hat\theta]=\theta^{2}/(2n)\), so its standard deviation is \(\theta/\sqrt{2n}\). The central limit theorem says the standardised estimator is approximately \(\mathcal N(0,1)\). Asked for \(\P(|\hat\theta-\theta|\ge\varepsilon)\), standardise inside the absolute value — dividing both \(|\hat\theta-\theta|\) and \(\varepsilon\) by the same positive number \(\theta/\sqrt{2n}\) — and then fold:

Check (\(n=200\), \(\varepsilon=0.2\), \(\theta=1\)): the argument is \(-4\), so the approximation is \(2\Phi(-4)=6.334\cdot10^{-5}\), against the Chebyshev bound \(\theta^{2}/(2n\varepsilon^{2})=0.0625000\) from part (b) — the same quantity, three orders of magnitude apart, which is exactly what part (d) asks you to comment on. The identical move with \(\hat\theta-\theta\) replaced by \(Z_n\) is SoSe26 · P4d: \(\P(|Z_n|\ge a)\approx2(1-\Phi(a))=2\Phi(-a)\).

D1 — a rectangle, done both ways My addition

Compute \(\displaystyle\int_0^1\!\!\int_0^2 xy\,\mathrm dy\,\mathrm dx\). Inner first, with \(x\) frozen so that it comes out of the \(y\)-integral as a constant:

By Rule 1 the same thing in one line: \(\bigl(\int_0^1x\,\mathrm dx\bigr)\bigl(\int_0^2y\,\mathrm dy\bigr) =\tfrac12\cdot2=1\). Check: numerical double integration gives \(1.0000000\). ✓

Warm-up before D2 — one slice of the triangle, by hand My addition

Take the triangle \(B=\{(x,y) : 0\le x\le y\le 1\}\) — the half of the unit square above the diagonal \(y=x\) — and integrate the constant \(1\) over it, which just measures its area. Choose \(x\) as the outer variable, so the inner integral runs over \(y\).

One slice. Fix \(x=0.3\) and ask the only question there is: for this \(x\), which \(y\) are in \(B\)? The conditions are \(y\ge x=0.3\) and \(y\le1\), so \(y\) runs from \(0.3\) to \(1\) and the slice has length \(1-0.3=0.7\). Written as an integral, \(\int_{0.3}^{1}1\,\mathrm dy=1-0.3=0.7\).

Every slice. The same two conditions at a general \(x\) give \(y\) from \(x\) to \(1\), of length \(1-x\). Now integrate those lengths over all \(x\) between \(0\) and \(1\):

Check two ways. The triangle is half of a unit square, so its area is \(\tfrac12\) ✓. And in the other order, with \(y\) outside: for a fixed \(y\), \(x\) runs from \(0\) to \(y\), length \(y\), and \(\int_0^1 y\,\mathrm dy=\tfrac12\) ✓ — the same answer, which is Rule 2. Notice the one thing that changed between the two orders: the inner limits, \(x\) to \(1\) in one, \(0\) to \(y\) in the other. Everything in D2 to D5 below is this slice question asked about a different region.

D2 — the triangle \(\{x\le y\}\): "which of the two happens first?" My addition

\(X\sim\Exp(\lambda)\) and \(Y\sim\Exp(\mu)\) independent, with \(\lambda=2\), \(\mu=3\). Independence makes the joint density the product \(p(x,y)=\lambda e^{-\lambda x}\cdot\mu e^{-\mu y}\) on \(x,y\ge0\). Then

Each step: the inner integral is the exponential table row \(\int_x^{\infty}\mu e^{-\mu y}\mathrm dy =\bigl[-e^{-\mu y}\bigr]_x^{\infty}=0-(-e^{-\mu x})=e^{-\mu x}\); then \(e^{-\lambda x}e^{-\mu x}=e^{-(\lambda+\mu)x}\) by the exponential rule \(e^{a}e^{b}=e^{a+b}\); then the same table row again with rate \(\lambda+\mu\), which gives \(\lambda\cdot\frac1{\lambda+\mu}\). Checks: \(\lambda=2,\mu=3\) gives \(0.4000000=2/5\) ✓, and doing the integral in the other order (\(\int_0^{\infty}\!\int_0^{y}\)) gives \(0.4000000\) as well ✓ — that is Rule 2 in action.

D3 — the half-plane \(\{x+y\le z\}\): the density of a sum My addition

\(U,V\) independent and \(\Unif(0,1)\) — that is, each has density \(1\) on \([0,1]\) and \(0\) elsewhere. What is \(\P(U+V\le z)\) for \(z=0.6\)? The region is the part of the unit square below the line \(u+v=0.6\), a right triangle with legs \(0.6\):

In general \(\P(U+V\le z)=z^{2}/2\) for \(0\le z\le1\); differentiating in \(z\) gives the density \(p_{U+V}(z)=z\) there. Checks: \(z=0.6\) gives \(0.1800000=0.6^{2}/2\) ✓; \(z=1.4\) gives \(0.8200000=1-(2-1.4)^{2}/2\) ✓ (above \(z=1\) the triangle is cut off by the square, so the formula changes — a reminder to draw the picture rather than trust one formula everywhere).

The general statement this produces is the convolution formula [Faltung]: differentiating \(\P(X+Y\le z)=\int_{-\infty}^{\infty}\int_{-\infty}^{z-x}p_X(x)p_Y(y)\,\mathrm dy\,\mathrm dx\) under the integral sign gives

Check on \(X,Y\) independent \(\Exp(1)\): the formula gives \(\int_0^{z}e^{-x}e^{-(z-x)}\mathrm dx=z e^{-z}\), and at \(z=2\) numerical integration returns \(0.2706706=2e^{-2}\) ✓ — the \(\Gamma(2,1)\) density, which is where the Gamma family comes from (§1.3).

D4 — the wedge \(\{x\le t\,y\}\): the density of a ratio SoSe25 · P1d

\(X,Y\) independent \(\Exp(1)\). For \(t>0\), the event \(\{X/Y\le t\}\) is \(\{X\le tY\}\) (multiplying by \(Y>0\) does not flip the inequality). So

The last step used the table row twice: \(\int_0^{\infty}e^{-y}\mathrm dy=1\) and \(\int_0^{\infty}e^{-(1+t)y}\mathrm dy=\frac1{1+t}\). Check at \(t=2\): numerical double integration gives \(0.6666667=2/3\) ✓. Differentiating in \(t\) gives the density of the ratio, \(p(t)=1/(1+t)^{2}\) on \(t>0\).

The whole recipe for a ratio, a sum or a maximum is the same three lines: (i) write the event as a region of the plane; (ii) integrate the joint density over it, inner limits first; (iii) differentiate the resulting cdf in the parameter to get a density. Nothing about it is specific to which function of \((X,Y)\) you took.

Worked — the uniform disc Sheet 8 · Ex 3(i)

\((X,Y)\) is uniform on the unit disc \(D=\{(x,y) : x^{2}+y^{2}\le1\}\). "Uniform" means constant density on \(D\); since the disc has area \(\pi\) and the total must be \(1\), that constant is \(1/\pi\): \(p_{X,Y}(x,y)=\tfrac1\pi\chi_D(x,y)\).

The slice. Fix an \(x\). The condition \(x^{2}+y^{2}\le1\) is \(y^{2}\le1-x^{2}\), i.e. \(|y|\le\sqrt{1-x^{2}}\). So \(y\) runs from \(-\sqrt{1-x^{2}}\) to \(+\sqrt{1-x^{2}}\) — and that requires \(1-x^{2}\ge0\), so there is a slice only for \(x\in[-1,1]\); outside that, \(p_X(x)=0\). Then

The integrand did not depend on \(y\), so the integral is just "constant × length of the interval", and the length is twice the half-width. By the symmetry between the two coordinates, \(p_Y(y)=\tfrac2\pi\sqrt{1-y^{2}}\) on \([-1,1]\). Checks: \(\int_{-1}^{1}p_X=1.0000000\) ✓ (a marginal is a density, so this is the free error-check); at \(x=0.6\) the slice integral gives \(0.5092958\) and the formula \(\tfrac2\pi\sqrt{1-0.36}=0.5092958\) ✓. The mean is \(0\) by the odd-function rule, and \(\E[X^{2}]=0.2500000=\tfrac14\), which is the trig-substitution box.

Why polar coordinates are the wrong tool here, even though the region is a disc: "integrate out \(y\)" is a statement about the coordinate \(y\), and after the change to \((r,\theta)\) there is no \(y\) left to integrate out. Polar coordinates are for Sheet 8 Ex 3(iii) — the covariance, where the integrand \(xy=r^{2}\cos\theta\sin\theta\) becomes \(\tfrac12 r^{2}\sin(2\theta)\) and the \(\theta\)-integral over a full turn is \(0\) (§2.1). Check: integrating \(xy/\pi\) over the disc numerically gives \(0.0000000\) ✓.

A second instance, with an unbounded slice Sheet 7 · Ex 1

\(p_{X,Y}(x,y)=e^{-y}\chi_A(x,y)\) with \(A=\{0\le x\le y\}\). For \(x\ge0\) the slice at that \(x\) is \(y\in[x,\infty)\), so \(p_X(x)=\int_x^{\infty}e^{-y}\mathrm dy=e^{-x}\). Slicing the other way, at a fixed \(y\ge0\) the variable \(x\) runs over \([0,y]\) and the integrand does not depend on \(x\), so \(p_Y(y)=\int_0^{y}e^{-y}\mathrm dx=y\,e^{-y}\). Checks: \(p_X(0.8)=0.4493290=e^{-0.8}\) ✓; \(p_Y(1.3)=0.3542913=1.3\,e^{-1.3}\) ✓. Both marginals are needed for what that exercise actually asks: the conditional densities \(p_{X\mid Y=y}=p_{X,Y}/p_Y\) and \(p_{Y\mid X=x}=p_{X,Y}/p_X\), from which it computes \(\E[X\mid Y=y]=y/2\) and \(\E[Y\mid X=x]=x+1\) (§2.1). A marginal is also the standard way to disprove independence, since \(p_{X,Y}=p_Xp_Y\) fails here — \(e^{-x}\cdot y e^{-y}\ne e^{-y}\chi_A(x,y)\) — and that is the move Sheet 8 · Ex 3(iii) makes for the disc.

Warm-up before Jensen — \(x^{2}\) at two points My addition

Take \(g(x)=x^{2}\) and the two points \(x_1=1\) and \(x_2=3\) on its graph, at heights \(g(1)=1\) and \(g(3)=9\). The chord is the straight segment joining \((1,1)\) to \((3,9)\).

Compare the two at the midpoint. The midpoint of \(1\) and \(3\) is \(\tfrac{1+3}{2}=2\). On the curve the height there is \(g(2)=4\). On the chord the height there is the midpoint of the two end heights, \(\tfrac{1+9}{2}=5\) — a straight line's value halfway along is the average of its two end values. And \(4\le5\): the chord is above the curve. That one inequality, for every pair of points and every point in between, is the definition of convex.

Now read it as probability. Let \(X\) be the random variable that takes the value \(1\) with probability \(\tfrac12\) and \(3\) with probability \(\tfrac12\). Then \(\E[X]=\tfrac12\cdot1+\tfrac12\cdot3=2\) and \(\E[g(X)]=\tfrac12\cdot1+\tfrac12\cdot9=5\), so

which is Jensen's inequality for this \(X\). The gap \(5-4=1\) is exactly \(\Var[X]\), because \(\E[X^{2}]-\E[X]^{2}\) is the variance, and \(\Var[X]=1\) for a variable sitting at \(1\) and \(3\) with equal weight ✓. Second-derivative check: \(g'(x)=2x\), \(g''(x)=2\), positive everywhere, so \(x^{2}\) is convex on all of \(\R\) — matching what the picture showed.

The list to memorise — six functions, with their second derivatives My addition

Numerical spot-checks of the second derivatives (difference quotient \(\frac{g(x+h)-2g(x)+g(x-h)}{h^{2}}\) with \(h=10^{-4}\)): \(x^{2}\) at \(x=1\): \(+2.000000\) versus \(2\) ✓; \(e^{x}\) at \(x=0.7\): \(+2.013753\) versus \(e^{0.7}=2.013753\) ✓; \(-\ln x\) at \(x=2\): \(+0.250000\) versus \(1/4\) ✓; \(\sqrt x\) at \(x=4\): \(-0.031250\) versus \(-\tfrac14\cdot4^{-3/2}=-1/32\) ✓; \(\ln x\) at \(x=3\): \(-0.111111\) versus \(-1/9\) ✓; \(1/x\) at \(x=2\): \(+0.250000\) versus \(2/8\) ✓.

The last row is the trap the exams like: a function can be convex on part of its domain and concave on another part. Jensen needs convexity on an interval that contains all the values \(X\) can take. Saying "on \(\R\)" when the function is only convex on \([0,\infty)\) loses the credit.

Watching the leftover shrink — a numeric example My addition

Take \(f(s)=\cos s\), whose expansion is \(1-s^{2}/2+o(s^{2})\). The table shows the true value, the parabola, the error, and — the column that matters — the error divided by \(s^{2}\).

The last column is the point. It is not enough that the error shrinks — the error divided by \(s^{2}\) shrinks too, by a factor of about \(100\) each time \(s\) shrinks by \(10\). That is what "\(o(s^{2})\)" asserts.

Examples with numbers

• \(A=[0,1]\). For \(x=0.3\) take \(r=0.3\): then \(B_{0.3}(0.3)=(0,0.6)\subseteq[0,1]\), so \(0.3\) is interior. In general for \(0<x<1\) take \(r=\min(x,1-x)\), the distance to the nearer end. For \(x=0\), every ball \((-r,r)\) contains \(-r/2\notin A\) and \(0\in A\), so \(0\) is a boundary point; the same for \(1\). Hence \([0,1]^\circ=(0,1)\) and \(\partial[0,1]=\{0,1\}\). \([0,1]\) is closed, not open.
• \(A=(0,1)\) is open: its interior is itself. It is not closed: \(\tfrac1n\in(0,1)\) but \(\tfrac1n\to0\notin(0,1)\).
• \(A=(0,\infty)\) (e.g. the possible values of a variance \(\sigma^2\)): open, since for \(x>0\) the ball \((x/2,\,3x/2)\) stays positive.
• In \(\R^2\): the parameter set \(\Theta=\R\times(0,\infty)\) of the Gaussian \((\mu,\sigma^2)\) is open — around any \((\mu,\sigma^2)\) with \(\sigma^2>0\), the disc of radius \(\sigma^2/2\) keeps the second coordinate positive. So every point is interior.
• A set can be neither open nor closed: \([0,1)\).

The median minimises \(\E|X-c|\) (Remark 2.22(v)), without the Leibniz rule

Let \(X\) have a continuous pdf \(p\), cdf \(F\), and \(\E|X|<\infty\). Split at \(c\) and pull \(c\) out of each integral (it is a constant for the integration variable \(x\)):

using \(\int_{-\infty}^cp=F(c)\) and \(\int_c^\infty p=1-F(c)\). Now only Case 1 and the product rule are needed: \(\bigl(cF(c)\bigr)'=F(c)+cp(c)\); \(\bigl(\int_{-\infty}^cxp\bigr)'=cp(c)\); \(\bigl(\int_c^\infty xp\bigr)'=-cp(c)\); \(\bigl(c(1-F(c))\bigr)'=(1-F(c))-cp(c)\). Adding with the signs above, the four \(cp(c)\) terms cancel:

So \(g'(c)=0\iff F(c)=\tfrac12\): \(c\) is a median. And \(g'<0\) to the left, \(g'>0\) to the right, so it is a minimum. Numbers: for \(X\sim\Unif(0,1)\) and \(0\le c\le1\), \(g(c)=\tfrac{c^2}2+\tfrac{(1-c)^2}{2}\), \(g'(c)=2c-1=2F(c)-1\) ✓, minimum at \(c=\tfrac12\).

Worked example with real sets My addition

Take \(\Omega=\{1,2,3,4,5,6\}\) (a die), \(A=\{2,4,6\}\) ("even"), \(B=\{4,5,6\}\) ("at least 4").

The last two rows give the same set. That is one instance of de Morgan's law (below). The general law says this is no coincidence of this die: it happens for all sets.

Infinite families My addition

Interval notation: \([a,b]\) contains both endpoints, \((a,b)\) neither, and \([a,b)\) only \(a\). \(\N_{>0}=\{1,2,3,\dots\}\).

• \(A_n=\bigl[0,1-\tfrac1n\bigr]\) for \(n\in\N_{>0}\). These grow: \(A_1=\{0\}\), \(A_2=[0,\tfrac12]\), \(A_3=[0,\tfrac23]\), … and \(\bigcup_n A_n=[0,1)\). Note the union is not \([0,1]\): no single \(A_n\) contains 1, and the union only collects what is in some member.
• \(B_n=\bigl(-\tfrac1n,\tfrac1n\bigr)\). These shrink, and \(\bigcap_n B_n=\{0\}\): a number \(x\ne0\) fails to be in \(B_n\) as soon as \(\tfrac1n<|x|\), which happens for some \(n\) by the Archimedean property.
• \(C_q=\{q\}\) for \(q\in\Q\). Here \(I=\Q\) is countably infinite and \(\bigcup_{q\in\Q}C_q=\Q\), where \(\Q\) is the set of fractions (rational numbers). This family is why \(\Q\) is a Borel set (§1.1): a countable union of single points.

The first two are exactly the shapes that appear in Proposition 1.19(iv) and (v) — increasing and decreasing families — and in Lemma 1.24, where \((-\infty,x_n]\) grows or shrinks.

Examples, including the ones that look strange Script A.4

• \(\pow(\{*,\odot\})=\bigl\{\emptyset,\{*\},\{\odot\},\{*,\odot\}\bigr\}\) — four elements.
• \(\pow(\emptyset)=\{\emptyset\}\). Read that carefully: the empty set has exactly one subset, namely itself, so its power set has one element and is not empty.
• \(\pow(\{\emptyset\})=\bigl\{\emptyset,\{\emptyset\}\bigr\}\) — two elements.
• \(|\pow(\Omega)|=2^{|\Omega|}\) for finite \(\Omega\), where \(|\Omega|\) is the number of elements of \(\Omega\). Reason: building a subset means one yes/no decision per element, so there are \(2\cdot2\cdots2\) possibilities by the product rule (page B3). For \(\Omega=\{1,2,3\}\) that is \(2^3=8\), matching the list above.

Worked example with numbers My addition

\(\Omega=\{1,\dots,6\}\) (a die), \(X(\omega)=\omega-3\), \(g(x)=x^2\). Then \(g\circ X\) sends \(1\mapsto4,\ 2\mapsto1,\ 3\mapsto0,\ 4\mapsto1,\ 5\mapsto4,\ 6\mapsto9\). Take \(B=\{1,4\}\).

• Backwards in two steps. \(g^{-1}(\{1,4\})=\{-2,-1,1,2\}\) (the numbers whose square is 1 or 4). Then \(X^{-1}(\{-2,-1,1,2\})=\{1,2,4,5\}\) (the \(\omega\) with \(\omega-3\) in that set).
• Directly. From the list, \(g(X(\omega))\in\{1,4\}\) exactly for \(\omega=1,2,4,5\) ✓.

A cdf-style example. With \(g(x)=x^2\) and \(B=(-\infty,y]\) for some \(y\ge0\): \(g^{-1}\bigl((-\infty,y]\bigr)=[-\sqrt y,\sqrt y]\), so \(\{X^2\le y\}=\{-\sqrt y\le X\le\sqrt y\}\). This is exactly the first step of the cdf method for the distribution of \(g(X)\).

Three quick applications My addition

• PINs. Four digits, each from \(\{0,\dots,9\}\), repeats allowed: \(10\cdot10\cdot10\cdot10=10^4=10\,000\). This is Sheet 0 Exercise 3.
• Two dice. \(6\cdot6=36\) ordered pairs — Sheet 0 Exercise 1. Note it is \(36\) and not \(21\): the 21 unordered pairs are not equally likely.
• Subsets. Building a subset of an \(n\)-element set means answering "in or out?" once per element: \(2^n\). That is why \(|\pow(\Omega)|=2^{|\Omega|}\).

Exactly one repeated digit in a 4-digit PIN Sheet 0 · Ex 3(iii)

Count PINs in which one digit appears exactly twice and the other two digits are distinct from each other and from the repeated one. Build such a PIN in four independent stages:

1. the repeated digit \(d\): 10 ways;
2. which two of the four positions carry \(d\): \(\binom42=\) 6 ways;
3. the digit in the first remaining position, from \(\{0,\dots,9\}\setminus\{d\}\): 9 ways;
4. the digit in the last remaining position, from the 8 still unused: 8 ways.

The subtlety: stages 3 and 4 fill two specific positions (the leftover ones, in left-to-right order), so each PIN is produced exactly once. Had you instead chosen an unordered pair of digits for those slots, you would have to divide by 2. Using \(9\cdot8\) already accounts for the order of the slots — so no correction is needed, and adding one is the classic error.

Derangements: nobody gets their own sheet back Sheet 2 · Ex 4

\(n\) students hand in sheets, and the sheets are handed back in random order. An outcome \(\omega\) is an arrangement \((\omega_1,\dots,\omega_n)\) of \(1,\dots,n\), where \(\omega_i\) is the sheet student \(i\) receives; so \(|\Omega|=n!\). Let \(A_i=\{\omega\mid\omega_i=i\}\) ("student \(i\) gets their own sheet"). The count uses inclusion–exclusion (above) plus one cancellation. Fixing \(k\) specified positions leaves \((n-k)!\) arrangements, so each \(k\)-fold intersection has probability \((n-k)!/n!\). There are \(\binom nk\) such intersections, and

So the whole inner sum of inclusion–exclusion collapses to the single number \(\tfrac1{k!}\). Hence \(\P(\text{someone gets their own sheet})=\sum_{k=1}^{n}\frac{(-1)^{k+1}}{k!}\), and \(\P(\text{nobody does})=1-\text{that}=\sum_{k=0}^{n}\frac{(-1)^k}{k!}\to e^{-1}\approx0.368\). Example \(n=3\): \(1-1+\tfrac12-\tfrac16=\tfrac13\); indeed, of the \(3!=6\) arrangements exactly two, \((2,3,1)\) and \((3,1,2)\), leave nobody with their own sheet. Full working on the §1.2 page.

Failure 1: functions that jump everywhere My addition

Take the Dirichlet function on \([0,1]\): \(\one_\Q(x)=1\) if \(x\) is rational, 0 otherwise. Every interval, however small, contains both rationals and irrationals (\(\Q\) is dense — Analysis 1.2). So every Riemann upper sum is 1 and every lower sum is 0, forever. The Riemann integral does not exist.

Probability suggests an answer anyway. Pick \(x\) uniformly from \([0,1]\). The chance of hitting a rational is 0: there are only countably many rationals, and each single point has probability 0. So the "right" value of \(\int_0^1\one_\Q\) is \(0\). The Lebesgue integral gives exactly that.

Failure 2: limits leave the class My addition

List the rationals of \([0,1]\) as \(q_1,q_2,\dots\) (possible, they are countable) and let \(f_n=\one_{\{q_1,\dots,q_n\}}\). Each \(f_n\) is zero except at \(n\) points, so it is Riemann integrable with \(\int f_n=0\). The sequence increases pointwise to \(\one_\Q\) — which is not Riemann integrable at all.

So "the integral of the limit is the limit of the integrals" cannot even be stated in the Riemann world. Probability needs it constantly: the laws of large numbers, the central limit theorem and the consistency arguments of Chapter 3 all describe what happens to a random quantity as \(n\to\infty\), and their proofs keep moving a limit past an expectation — that is, past an integral.

Two small examples My addition

1. All subsets. \(\Omega_1=\Omega_2=\{1,2\}\), both with the power set. Every single point is a rectangle, e.g. \(\{(1,2)\}=\{1\}\times\{2\}\). Every subset of \(\{1,2\}^2\) is a finite union of single points, so \(\Aa_1\otimes\Aa_2\) is the power set of \(\{1,2\}^2\): all \(2^4=16\) subsets.

But not every one of them is a rectangle. The diagonal \(\{(1,1),(2,2)\}\) ("both coordinates equal") is not of the form \(A\times B\): it would need \(1\in A\) (from \((1,1)\)) and \(2\in B\) (from \((2,2)\)), and then \((1,2)\in A\times B\) too, which is not in the diagonal. It is still in the product σ-algebra, as the union of the two rectangles \(\{1\}\times\{1\}\) and \(\{2\}\times\{2\}\). That is why the definition says "generated by" rectangles, not "consisting of" rectangles.

2. A coarse first factor. \(\Aa_1=\{\emptyset,\{1,2\}\}\) (you learn nothing about the first coordinate), \(\Aa_2=\pow(\{1,2\})\), where \(\pow(S)\) is the power set of \(S\), the collection of all its subsets — here \(\emptyset,\{1\},\{2\},\{1,2\}\). The rectangles are \(\emptyset\) and \(\{1,2\}\times B\), so \(\Aa_1\otimes\Aa_2=\bigl\{\emptyset,\ \{1,2\}\times\{1\},\ \{1,2\}\times\{2\},\ \{1,2\}^2\bigr\}\): only the 4 events that talk about the second coordinate alone.

A distribution with no mean My addition

The Cauchy distribution has density \(p(x)=\frac{1}{\pi(1+x^2)}\). Then \(\int_0^\infty x\,p(x)\,\mathrm dx=\frac1{2\pi}\bigl[\ln(1+x^2)\bigr]_0^\infty=\infty\), and the same on the negative side. So \(\int|x|p(x)\,\mathrm dx=\infty\): \(\E[X]\) does not exist, even though the density is symmetric about 0 and "the mean ought to be 0". Symmetry is not enough; integrability is the requirement. The script makes the same point in Remark 2.22(iii): "The mean \(\E[X]\) exists only if \(\E[|X|]<\infty\). For some distributions (e.g., Cauchy distribution), the mean is undefined." That is also why the Cauchy distribution is the standard counterexample to the law of large numbers on the §2.4 page: the law assumes a finite mean, and here there is none.

Example 1 — the triangle \(0\le x\le y\le1\), both orders My addition

\(D=\{(x,y)\mid 0\le x\le y\le1\}\) is the triangle above the diagonal \(y=x\) inside the unit square, with corners \((0,0)\), \((0,1)\), \((1,1)\). Integrate \(f(x,y)=x\).

\(x\) outside. \(x\) ranges over \([0,1]\). For fixed \(x\), the vertical line meets \(D\) where \(x\le y\le1\). So

(The inner integral: \(x\) does not depend on \(y\), so it is a constant, and a constant integrated over an interval of length \(1-x\) gives \(x(1-x)\).)

\(y\) outside. \(y\) ranges over \([0,1]\). For fixed \(y\), the horizontal line meets \(D\) where \(0\le x\le y\). So

Area check (\(f=1\)): \(\int_0^1\int_0^y1\,\mathrm dx\,\mathrm dy=\int_0^1y\,\mathrm dy=\tfrac12\), half of the unit square ✓.

The unbounded version \(0\le x\le y<\infty\) is Sheet 7 Ex 1 with \(f(x,y)=e^{-y}\). \(x\) outside: \(\int_0^\infty\int_x^\infty e^{-y}\,\mathrm dy\,\mathrm dx=\int_0^\infty e^{-x}\,\mathrm dx=1\). \(y\) outside: \(\int_0^\infty\int_0^ye^{-y}\,\mathrm dx\,\mathrm dy=\int_0^\infty ye^{-y}\,\mathrm dy=1\) (integration by parts). Same answer, and the second order directly gives the marginal density \(p_Y(y)=ye^{-y}\).

Example 2 — the half-plane \(x+y\le z\) (convolutions) My addition

For the distribution of a sum \(X+Y\) you need \(\P(X+Y\le z)=\iint_{A_z}p_{X,Y}\), where \(A_z=\{(x,y)\mid x+y\le z\}\) for a fixed number \(z\). The boundary is the line \(y=z-x\), and \(A_z\) is everything below it. With \(y\) outside: \(y\) ranges over all of \(\R\), and for fixed \(y\) the condition \(x+y\le z\) means \(x\le z-y\). So

which is the first step of the convolution formula in Sheet 7 Ex 2.

With non-negative variables the region becomes a triangle. If \(X,Y\ge0\), the density is zero unless \(x\ge0\) and \(y\ge0\), so only the triangle \(x\ge0,\ y\ge0,\ x+y\le z\) counts (for \(z\ge0\)): \(y\) from \(0\) to \(z\), and for fixed \(y\), \(x\) from \(0\) to \(z-y\).

• Numbers: \(X,Y\) independent and uniform on \((0,1)\) (joint density \(1\) on the unit square), \(z=\tfrac12\): \(\P(X+Y\le\tfrac12)=\int_0^{1/2}\int_0^{1/2-y}1\,\mathrm dx\,\mathrm dy=\int_0^{1/2}\bigl(\tfrac12-y\bigr)\mathrm dy=\tfrac14-\tfrac18=\tfrac18\). Check: a right triangle with both legs \(\tfrac12\) has area \(\tfrac12\cdot\tfrac12\cdot\tfrac12=\tfrac18\) ✓.
• Exponentials: \(X,Y\) independent \(\Exp(1)\), joint density \(e^{-x}e^{-y}\): \(\P(X+Y\le z)=\int_0^z\bigl(\int_0^{z-y}e^{-x}\,\mathrm dx\bigr)e^{-y}\,\mathrm dy=\int_0^z\bigl(1-e^{-(z-y)}\bigr)e^{-y}\,\mathrm dy =\int_0^z\bigl(e^{-y}-e^{-z}\bigr)\mathrm dy=1-e^{-z}-ze^{-z}\). Differentiating in \(z\) gives the density \(ze^{-z}\), the \(\Gamma(2,1)\) density found on the §2.3 page.

Example 3 — the unit disc, and why polar coordinates help My addition

\(D=\{(x,y)\mid x^2+y^2\le1\}\). In \(x\) and \(y\): \(x\) ranges over \([-1,1]\); for fixed \(x\), \(y^2\le1-x^2\) means \(-\sqrt{1-x^2}\le y\le\sqrt{1-x^2}\). The area is \(\int_{-1}^1 2\sqrt{1-x^2}\,\mathrm dx\) — correct, but awkward to integrate.

In polar coordinates \(x=r\cos\theta\), \(y=r\sin\theta\) (\(r\) = distance from the origin, \(\theta\) = angle), the disc is simply the rectangle \(0\le r\le1\), \(0\le\theta<2\pi\), and \(\mathrm dx\,\mathrm dy\) becomes \(r\,\mathrm dr\,\mathrm d\theta\) (the Jacobian factor \(r\), explained on B5). So

A probability. \((X,Y)\) uniform on the disc has density \(\tfrac1\pi\) on \(D\). Then \(\P(X^2+Y^2\le\tfrac14)=\int_0^{2\pi}\int_0^{1/2}\tfrac1\pi\,r\,\mathrm dr\,\mathrm d\theta=\tfrac1\pi\cdot2\pi\cdot\tfrac18=\tfrac14\): the inner disc of radius \(\tfrac12\) has a quarter of the area ✓. That uniform distribution on the disc is the setting of Sheet 8 Ex 3, which asks for the marginal densities, the means and variances, and \(\Cov[X,Y]\) — the covariance integral there is set up in exactly these polar coordinates.

Polar coordinates and the Gaussian integral

\(g(r,\theta)=(r\cos\theta,r\sin\theta)\) on \(U=(0,\infty)\times(0,2\pi)\). Jacobian \(\begin{pmatrix}\cos\theta&-r\sin\theta\\\sin\theta&r\cos\theta\end{pmatrix}\), determinant \(r\cos^2\theta+r\sin^2\theta=r\). So \(\mathrm dx\,\mathrm dy=r\,\mathrm dr\,\mathrm d\theta\) — the factor used in Sheet 8 Ex 3 and 4. (Worked integrals over a disc, and how to read off limits for other regions, are on B4.)

Application. With \(I=\int_\R e^{-x^2/2}\,\mathrm dx\), Tonelli — the integrand is non-negative, so no integrability has to be checked first (B4) — gives \(I^2=\iint e^{-(x^2+y^2)/2}\,\mathrm dx\,\mathrm dy=\int_0^{2\pi}\!\!\int_0^\infty e^{-r^2/2}\,r\,\mathrm dr\,\mathrm d\theta =2\pi\bigl[-e^{-r^2/2}\bigr]_0^\infty=2\pi\), so \(I=\sqrt{2\pi}\): the normalising constant of the normal density (also on §1.3). (Taking \(\theta\) in the open interval \((0,2\pi)\) and \(r>0\) is what makes \(g\) a bijection. The price is that the image \(g(U)\) is the plane with the non-negative \(x\)-axis \(\{(x,0)\mid x\ge0\}\) removed — a line, of area \(0\) — and leaving out a set of measure \(0\) does not change an integral (B4, null sets).)

Linear maps. For \(g(x)=Ax+b\) with invertible \(A\), every partial derivative \(\partial g_i/\partial x_j\) is just \(A_{ij}\), so \(J_g=A\) at every point — the constant \(b\) differentiates away. With \(U=\R^n\), so that \(g(U)=\R^n\) as well, the formula reads \(\int_{\R^n} f(y)\,\mathrm dy=|\det A|\int_{\R^n} f(Ax+b)\,\mathrm dx\). This is behind Example 2.52(ii). With numbers: \(A=\begin{pmatrix}2&0\\0&3\end{pmatrix}\), \(b=0\) maps the unit square \((0,1)^2\) onto the rectangle \((0,2)\times(0,3)\). Taking \(f=1\): area \(6\) on the left, \(|\det A|\cdot1=6\) on the right ✓.

Remark 0.1 — historical developments Script 0.1

Gambling is thousands of years old, but modern probability is usually dated to a 1654 letter from the French aristocrat and gambler Chevalier de Méré to Blaise Pascal. The question was the problem of points: if a repeated game of chance has to be abandoned before it ends, how should the pot be divided fairly, given the current score? Pascal discussed it with Pierre de Fermat, and the correspondence produced several of the ideas the subject still uses. In that same year Pascal reportedly had a religious vision and largely gave up scientific work.

Two later milestones: Pierre-Simon Laplace's Théorie Analytique des Probabilités (1812), which unified the field and introduced Bayes' theorem independently of Bayes; and the axiomatisation by Andrey Kolmogorov in 1933, which is the one this course builds on. The script adds the anecdote that Napoleon asked Laplace why God did not appear in his book, and Laplace answered: "I had no need of that hypothesis."

Remark 0.2 — why this course is called "discrete" Script 0.2

Friedrich L. Bauer started TUM's first degree programme in Informationsverarbeitung in 1967. For decades computer science was treated as inherently discrete, which produced core modules such as "Discrete Structures" and "Discrete Probability Theory". Randomised algorithms, machine learning and AI have since made continuous probability just as important for computer scientists. The content of this course has been adapted; the name has not.

Why this matters for you: despite the title, roughly half of this course is continuous (densities, Gaussians, the central limit theorem, t-tests). Do not expect the exam to stay discrete.

Remark 0.9 Script 0.9

The script goes through its own examples and asks, in each, where the randomness is supposed to live:

• In Example 0.4 it is "choosing blindly between the two cards", in 0.6 a coin toss, in 0.5 spinning the chamber. We assume each mechanism makes the different outcomes equally likely. But once the exact position and motion of the hand are fixed, the result could be predicted: these are deterministic, not random.
• Example 0.8 is about random number generators, and shows that a small change in ignorance — learning a seed — turns an unpredictable sequence completely deterministic. The script explicitly does not go into pseudo-random number generation.
• In Example 0.3 no randomising mechanism is invoked at all. The uncertainty comes from our imperfect understanding of a human being. If we knew everything about how a human works, we should be able to predict the outcome.
• In Example 0.7 it is a crowd-sourced, heavily biased, subjective collective belief.
• Quantum mechanics is often invoked as a source of "true" randomness. Measurement outcomes on well-prepared quantum states really cannot be predicted — yet the probabilities of the different outcomes can be predicted perfectly.
• In the models of this course (for example, of a coin toss), probabilities represent uncertainty or ignorance about a system. The aim is to express, consistently and with numbers, what we believe about uncertain things before we observe them. The script points ahead to Viewpoints 0.10, 1.31 and 1.32 for more on this.

The script's own summary sentence, which is worth memorising:

The script also recommends a podcast with the physicist Andrew Jaffe on how models, and the probabilities we give them, help us understand the world.

Viewpoint 0.10 — determinism, free will, compatibilism Script 0.10

Remark 0.9 tempts you to conclude: if only we knew enough (apart from quantum effects), nothing would be truly random, and uncertainty would disappear. In practice we cannot know enough, so we describe our ignorance with probabilities.

Determinism is the view that the universe really works like that at its core. Including quantum mechanics, it says: if you knew the wave function of the universe (its complete physical state) and the laws of physics perfectly, you could predict its evolution perfectly, and so also the probability of any future event.

Determinism leaves little room for real randomness, serendipity or luck. Many people also feel it clashes with their experience of free will. The script's point is that the two are not logically incompatible. A family of philosophical positions called compatibilism explains how a deterministic universe and free will can coexist. Randomness, determinism and free will are still debated in philosophy and physics.

Not examinable. The script marks this whole stretch "— Start / End Philosophical Interlude —".

Remark 1.5 — samples, events, event systems Script 1.5

The script includes an illustration (adapted from Georgii, 2015) showing three levels stacked on top of each other. Redrawn here:

Reading it from the bottom: a single outcome \(\omega\) is a point of \(\Omega\). An event \(A\) is a set of outcomes, so it is a point one level up, inside \(\pow(\Omega)\). An event system such as a \(\sigma\)-algebra \(\Aa\) is a set of events, so it is a point one level higher still, inside \(\pow(\pow(\Omega))\).

Example 1.9 \(\sigma\)-algebras Script 1.9

1. \(\{\emptyset,\Omega\}\) is a \(\sigma\)-algebra on \(\Omega\), called the trivial \(\sigma\)-algebra. Check: \(\Omega\) is in it; the complements are \(\emptyset^c = \Omega\) and \(\Omega^c = \emptyset\), both in it; any union of copies of \(\emptyset\) and \(\Omega\) is again \(\emptyset\) or \(\Omega\).
2. The power set \(\pow(\Omega)\) is a \(\sigma\)-algebra on \(\Omega\). Check: every subset of \(\Omega\) is in it, so any requirement of the form "this subset must be in it" holds automatically.
3. For every \(A \subseteq \Omega\), \(\sigma(\{A\}) = \{\emptyset, \Omega, A, A^c\}\). Check: this collection is closed under complements (the four complements are \(\Omega, \emptyset, A^c, A\)) and under unions (any union of these four is one of the four again — e.g. \(A \cup A^c = \Omega\)), and it contains \(A\). By Lemma 1.8 the generated \(\sigma\)-algebra is the smallest one containing \(A\); no member can be dropped, since each is forced by the axioms. Note the degenerate cases: if \(A = \emptyset\) or \(A = \Omega\) the list collapses to the trivial \(\sigma\)-algebra.

Overview 1.12 why the word "measure" Script 1.12

The idea behind measurability is literally to measure the size of mathematical objects. We have good intuition for the size of

1. subsets of \(\R\): the length of intervals,
2. subsets of \(\R^2\): the area of surfaces,
3. subsets of \(\R^3\): the volume of solids.

A rigorous formalism is needed for the corner cases: what volume should a line in 3D space have? What about the volume of an uncountable collection of points?

The script then makes the point that pays off later: we can also measure mass. To get the total mass of the earth's outer crust, take the density of the material and integrate it over the volume of the crust. "Integrate the density over a measure of volume to get a measure of mass" is the key idea behind densities. Another instance: integrate population density over an area to get a number of people.

Example 1.13 the Borel \(\sigma\)-algebra on \(\R^n\) Script 1.13

Let \(\Omega = \R^n\) for \(n \in \N_{>0}\). The \(\sigma\)-algebra generated by the open boxes

is called the Borel \(\sigma\)-algebra on \(\R^n\), written \(\mathcal B^n\). For \(n=1\) we simply write \(\mathcal B\).

Unpacking the notation. \((a,b)\) is the open interval \(\{x \in \R \mid a < x < b\}\); \(X \times Y := \{(x,y) \mid x \in X, y \in Y\}\) is the Cartesian product, so an open box in \(\R^2\) is an open rectangle and in \(\R^3\) an open brick. The generator is the set of all such boxes, for all choices of endpoints.

What \(\mathcal B\) contains. Far more than intervals. Since \(\sigma\)-algebras are closed under countable operations (Corollary 1.10): \([a,b] = \bigcap_{k\ge1}\bigl(a-\tfrac1k,\,b+\tfrac1k\bigr)\) is Borel; \(\{x\} = [x,x]\) is Borel; \(\Q = \bigcup_{q\in\Q}\{q\}\) is Borel, being a countable union of singletons. Every set you will ever write down explicitly is Borel. The sets that are not have to be conjured with the axiom of choice — see Theorem 1.28.

Remark 1.14 topological view on \(\mathcal B\) Script 1.14

If you know what a topology \(\mathcal O\) on a set \(\Omega\) is, you can try to prove that the Borel \(\sigma\)-algebra \(\mathcal B^n\) on \(\R^n\) is \(\sigma(\mathcal O^n)\), where \(\mathcal O^n\) is the standard topology on \(\R^n\).

Translation, for those who do not: a topology is a chosen collection of subsets declared "open"; the standard one on \(\R^n\) declares \(A\) open when every point of \(A\) has a small ball around it still inside \(A\). The remark says: generating from all open sets gives the same \(\sigma\)-algebra as generating from open boxes alone. The reason is that every open set is a countable union of boxes with rational corners. Sheet 2, Exercise 5 proves exactly this (with half-open rational intervals), and it is worked below, so you do not need any topology.

Overview 1.21 probability as measure Script 1.21

Intuitively, the fact that probability is essentially a measure is the idea that we "measure the likelihood" of something happening. The machinery developed for measuring subsets of arbitrary spaces is exactly what we need to assign probabilities consistently — to measure the "size of subsets" according to their probability.

The script's diagram (which is also on the title page of the script), redrawn:

Left: the experiment produces an outcome somewhere in \(\Omega\); three events \(A, B, C\) are marked. Middle: those events, and everything you can build from them with complements and countable unions, make up the \(\sigma\)-algebra \(\Aa\). Right: \(\P\) sends each event to a number in \([0,1]\).

Remark 1.29 how to interpret the two Vitali theorems Script 1.29

The invariance property in Theorem 1.27 states that every sequence of coin tosses should be exactly as probable as the one where the \(n\)-th flip came out differently. The invariance in Theorem 1.28 states that the probability of a set of locations in \([0,1)\) should not change if the set is shifted by a rational amount around the circle. Both are very natural symmetry principles.

The point of the theorems is not that these symmetries are unreasonable. The point is that demanding them on the full power set is too much. If we insist on using \(\pow(\Omega)\) for infinite spaces, then even very natural symmetry requirements become inconsistent with countable additivity. So while \(\pow(\Omega)\) is a \(\sigma\)-algebra, it is generally too large for a useful probability theory on infinite spaces, and we must work with a smaller \(\sigma\)-algebra of measurable sets.

The second construction makes this concrete: \(V\subseteq[0,1)\) cannot be assigned a translation-invariant probability compatible with interval length, and is a concrete example of a non-Borel subset of \([0,1)\). "This is one of the central reasons \(\sigma\)-algebras are not just technical decoration but genuinely necessary."

Remark 1.30 interpretations of probability Script 1.30

Connecting back to Remark 0.9: the probability space \((\Omega,\Aa,\P)\) does not necessarily represent a physically real source of randomness, but rather our ignorance or the limits of our knowledge. \(\P\) is our best description, and \(\Omega\) a convenient representation of the outcomes. Randomness is thus typically not "in nature" but "in our model". Probability theory is a quantitative framework for reasoning consistently about uncertainty, not an explanation of some intrinsic source of randomness. The script's working premise from here on: probability is useful.

In practice we call \(A\in\Aa\) an event and \(\P(A)\) the probability that \(A\) happens. An event may consist of several outcomes: "rolling a 1 or a 3" is a valid event, with probability \(\tfrac13\) for a fair die, even though only one of the two can occur in any given roll. The three common interpretations:

1. naive: "nature is not sure" what it does, and \(\P(A)\) is the degree of uncertainty with which it decides for \(A\);
2. frequentist: \(\P(A)\) is the relative frequency with which \(A\) tends to occur under the same external conditions;
3. subjective: \(\P(A)\) is the degree of certainty with which, based on my personal assessment of the situation, I am willing to bet on \(A\).

Viewpoint 1.31 probability from gambling: the Dutch book argument Script 1.31

We started directly with Kolmogorov's axioms. Besides "falling from the heavens", they can also be derived from other principles. The Dutch book arguments show that if a person's subjective probabilities violate the axioms of probability, a bettor can construct a list of bets that guarantees that person loses money. Consequently, rational belief assignments must obey the axioms of probability to avoid inconsistency.

A concrete instance, mine: suppose you claim \(\P(A)=0.6\) and \(\P(A^c)=0.6\), violating \(\P(A)+\P(A^c)=1\). I sell you a ticket paying 1 € if \(A\) happens, for 0.60 €, and another paying 1 € if \(A^c\) happens, for 0.60 €. You have paid 1.20 € and exactly one of the two tickets pays out 1 €. You lose 0.20 € no matter what happens. This is what "guaranteed to lose" means, and it is the reason the relative in Example 0.3 finally asks the doctor for betting odds: a bet is a way of forcing a vague degree of belief to become a number that obeys the axioms.

Viewpoint 1.32 probability from logic and partial beliefs Script 1.32

Jaynes (2003) builds the axioms of probability by extending classical logic to settings with uncertainty due to incomplete information. Leaning on basic principles of rational reasoning — consistency, and the ability to rank plausibility — he shows that any system for manipulating degrees of belief must obey the rules of probability. This motivates the axioms as the unique rational calculus for reasoning under uncertainty.

The script adds, disarmingly: "I would have loved to take this approach in the lecture as well, but due to a lack of time we just dumped the final framework onto you."

The script also recommends a podcast on the "Logic and Use of Probability" with the philosopher Branden Fitelson. It explains how probabilities can (and cannot) be used to judge how strongly evidence supports or undermines a scientific claim.

Remark 1.33 Bayes vs. frequentist Script 1.33

It is often said there are two approaches to probability.

• The frequentist approach interprets probabilities as long-run relative frequencies of events in repeated trials. The parameters of the underlying experiment are considered fixed but unknown.
• The Bayesian approach interprets probability as a degree of belief, or credence, allowing uncertainty about the parameters to be modelled directly with probability distributions. Bayesians have less trouble talking about events that by their nature can only happen once.

Bayesians rely on prior beliefs — one must start with some credences. Frequentists rely on repeatedly sampling from identical distributions, at least conceptually, and avoid prior beliefs. The script promises more detail later; it arrives in §3.5.