Probability
The Analysis of Data, volume 1
Integration over Product Spaces*
$
\def\P{\mathsf{\sf P}}
\def\E{\mathsf{\sf E}}
\def\Var{\mathsf{\sf Var}}
\def\Cov{\mathsf{\sf Cov}}
\def\std{\mathsf{\sf std}}
\def\Cor{\mathsf{\sf Cor}}
\def\R{\mathbb{R}}
\def\c{\,|\,}
\def\bb{\boldsymbol}
\def\diag{\mathsf{\sf diag}}
\def\defeq{\stackrel{\tiny\text{def}}{=}}
$
F.5. Integration over Product Spaces*
We consider in this section a measure space that is a product space of two measure spaces $(X,\mathcal{X}, \P_X)$ and $(Y,\mathcal{Y}, \P_Y)$. As stated in the previous section, the product space is the set $X\times Y$ (Cartesian product), endowed with the product $\sigma$-algebra $\mathcal{X}\otimes\mathcal{Y}$, and the product measure $ \P_{X\times Y}= \P_X\times \P_Y$. In this section we consider integration over product spaces and relate it to integrals over the component spaces. Although the section emphasizes products of two spaces, the results generalize to products of three or more spaces.
For notational convenience, we denote integration with respect to the measure $ \P_X$ on $X$ as $\int f \, \P_X(dx)$ and integration with respect to the measure $ \P_Y$ on $Y$ as $\int f \, \P_Y(dy)$.
Proposition F.5.1 (Fubini's Theorem).
Let $(X,\mathcal{X}, \P_X)$ and $(Y,\mathcal{Y}, \P_Y)$ be two probability measure spaces. Then for all integrable functions $f:X\times Y\to\R$
\begin{align*}
\int_Y \left(\int_X f \,d \P_X \right)\,d \P_Y = \int_{X\times Y} f\, d ( \P_X\times \P_Y) = \int_X \left(\int_Y f \,d \P_Y \right)\,d \P_X.
\end{align*}
We make the following comments.
- In the expression \[\int_X \left(\int_Y f \,d \P_Y\right)\,d \P_X,\] the inner integral is with respect to $ \P_Y$ over $y$, keeping $x$ fixed. It is thus a function of $x$, which is then integrated in the outer integral with respect to $ \P_X$. The expression \[\int_Y \left(\int_X f \,d \P_X \right)\,d \P_Y\] has a similar interpretation.
- Fubini's theorem may be extended for measures that are not probability functions.
- In the proof below, we omit a verification that the iterated integrals are finite. A more careful proof is available for example in (Billingsley, 1995).
Proof.
We prove the first equality. The proof of the second equality is similar. By Proposition F.4.4, the function $f(\cdot,x_0)$ is measurable, and therefore the integral $\int_X f\,d \P_X$ is a well defined function of $y$.
We assume first that $f$ is a non-negative function. If $f=I_E$, then Fubini's theorem follows from Proposition F.4.5 (in this case the inner integral $\int_X f\,d \P_X$ is $ \P_X(A)$, which is constant in $y$, implying that the outer integral is $ \P_X(A) \P_Y(B)$. If $f$ is a simple function, $\int_X f\,d \P_X$ is a linear function of measurable and integrable functions, which is a measurable and integrable function. Also, due to the linearity of the Lebesgue integral, the fact that Fubini's theorem holds for each of the indicator functions in the simple function linear combination implies that Fubini's theorem holds for the simple function $f$ itself. For a general non-negative function $f$, the integrals in Fubini's theorem are monotone limits of integrals of simple functions. Applying the monotone convergence theorem to the left hand side (twice) and to the central term (once) establishes that Fubini's theorem holds for a non-negative integrable $f$.
\begin{align*}
\int_Y \int_X f \,d \P_X \,d \P_Y
&= \int_Y \sup_{s:s\leq f} \int_X s\, d \P_X \,d \P_Y\\
&=\sup_{s:s\leq f} \int_Y \int_X s \,d \P_X \,d \P_Y \\
&= \sup_{s:s\leq f} \int_{X\times Y} s\,d( \P_X\times \P_Y) \\
&= \int_{X\times Y} f\,d( \P_X\times \P_Y).
\end{align*}
To prove Fubini's theorem for a general function $f$, we decompose $f$ into its positive and negative components $f=f^+-f^-$. The result follows from the case of non-negative $f$ and due to the linearity of the Lebesgue integral.
Fubini's theorem is particularly useful for decomposing product integrals of decomposable functions $f(x,y)=g(x)h(y)$ into a product of an integral of $\int_X g\,d \P_X$ and integral of $\int_Y h\,d \P_Y$.
Corollary F.5.1.
Let $(X,\mathcal{X}, \P_X)$ and $(Y,\mathcal{Y}, \P_Y)$ be two probability measure spaces. Then for all integrable functions $f:X\to \R$ and $g:Y\to \R$
\begin{align*}
\int_{X\times Y} (fg) \, d ( \P_X\times \P_Y) = \int_X f\,d \P_X \cdot \int_Y g\,d \P_Y.
\end{align*}
The Lebesgue Measure over $\R^d$*
Whenever we encounter in this book integrals over $\R^d$, we assume the integration is with respect to the product Lebesgue measure on $\R^d$. This measure is simply the product measure (see previous two sections) of $d$ copies of the Lebesgue measure spaces $(\R,\mathcal{B},\mu)$. Fubini's theorem implies that such integrals may be expressed using a sequence of iterated one dimensional integrals with respect to the Lebesgue measure. For example, in two dimensions we have
\[\int_{[a,b]\times[c,d]} f(x_1,x_2)\,d\bb x = \int_a^b \left(\int_c^d f(x_1,x_2)\,dx_2 \right) dx_1.\]
Fubini's theorem indicates that the product Lebesgue measure assigns to rectangles the integral value of their area (in two dimensions) or their volume (in three or higher dimensions). The product Lebesgue measure corresponds to the multivariate Riemann integral.
We denote integrals with respect to the product Lebesgue measure as $\int f(\bb x) \,d\bb x$ or as $\idotsint f(\bb x)\, d\bb x$, where the bold face emphasizes the vector nature of $\bb x$.