As a side note, the commentary presented below heavily resembles Elementary Analysis: The Theory of Calculus by Kenneth A. Ross. This is on purpose as Starling heavily based on the course notes based on this textbook.

The course is broken down into 3 main components:

1. Darboux Sums and Integrability
2. Integration Techniques
3. Power Series
4. Taylor Series

Darboux Sums and Integrability

From any regular calculus course, one would know that integration is the process of finding the area under its approximation of the area of a curve using extremely thin and equally-sized rectangles. But one can be more formal in how the area is calculated. From elementary school, one learned that to calculate the area of a rectangle by multiplying the length by its width. It turns out we can expand this concept to calculate the area of any curve by drawing a series of equally wide rectangles under the curve to approximate the area as seen below:

Approximation of the area of a curve using increasingly narrower rectangles. Taken from HyperPhysics

Eventually when the width of the rectangles becomes infinitesimally thin, the sum of these infinitesimally thin rectangles gives us the true area under the curve.

This technique of dividing an interval of a curve with increasingly smaller subintervals resulting in thinner and thinner rectangles to approximate area is not a mere theoretical concept but something that’s actually used in real life. I recall in one of my geography courses, the class was sent to a nearby bridge, tasked on measuring the depth of the river at different points, repeating the measurements in smaller partitions to get an approximate area of the river.

An image I found on google that looked familiar to what I learned in Geography to measure discharge. Source: Fondriest

One question that may arise is how to determine the height of these rectangles as it is clear that these rectangles don’t capture the curves nicely. There are a few ways to determine the height such as grabbing the function’s value at the midpoint between each subinterval $[t_i, t_{i+1}]$, the way the author first encountered in their time as a student in computer science. But this choice can feel arbitrary with no guarantees whether the height of each subinterval under-estimates or over-estimates the true area. One elegant approach introduced in this course is to consider both cases and see if the two cases can reconcile (i.e. be equal to each other) and this approach is called Darboux Sums or Darboux Integrals.

Formally, we define a partition for a function $f$ bounded on $[a,b]$ to be:

Partition of $[a,b]$ is a finite sequence $(t_m)$ with $a = t_0 \lt t_1 \lt t_2 \lt \dots \lt t_n = b$. Thus $P = \{t_0, t_1, t_2, \dots, t_n\}$

Put simply, it is a partition is a set of ordered numbers between $a$ and $b$.

The over-estimated area is called the UPPER darboux sum and the under-estimated rectangles are called the lower darboux sum making use of the definitions of supremum and infimum encountered in the previous course. As one can easily imagine, the two darboux sums take the “max” (supremum) and “min” (infimum) of the curve within each subinterval as the height of the rectangles.

For a subset $S \subseteq [a, b]$, we let:

$M(f,S) = sup\{ f(x) : x \in S\}$ i.e. “Max height of each rectangle”

$m(f,S) = \inf\{ f(x) : x \in S \}$ i.e. “Min height of each rectange”

Thus we can formally define the upper and lower darboux sums as the following:

UPPER Darboux Sum: $U(f,P) = \sum\limits^n_{k=1} \underbrace{M(f, [t_{k-1}, t_k])}_{\text{height}} \cdot \underbrace{(t_k-t_{k-1})}_{\text{base}}$

LOWER Darboux Sum: $L(f,P) = \sum\limits^n_{k=1} \underbrace{m(f,[t_{k-1}, t_k])}_{\text{height}}\underbrace{(t_k - t_{k-1})}_{\text{base}}$

Recall the objective is to approximate and eventually get the true area under the curve by taking the sum of the area of extremely thin rectangles. Also recall that the area of a rectangle is simply width multiplied by its height.

A sample of darboux sum where the lower sum is the area of the shaded rectangles and the upper sum is the area of the entire rectangles (shaded + unshaded regions). Taken from MIT OCW Real Analysis notes

One may see the upper and lower darboux sums reformulated as the following:

Upper Darboux Sum: $U(f,P) = \sum\limits^n_{k=1} M_i\Delta x_i$

Lower Darboux Sum: $L(f,P) = \sum\limits^n_{k=1} m_i \Delta x_i$

where:

$M_i = \sup\{f(x): x_{i-1} \leq x \leq x_i \}$

$m_i = \inf\{f(x): x_{i-1} \leq x \leq x_i \}$

Recall in our definition, $f$ is a bounded function on $[a,b]$ so it attains a global min and a global max value. In addition, every partition of $[a,b]$ will contain at least the endpoints $a$ and $b$. One key observation is that the local min and max are bounded by the global extremas:

• $M(f, [t_{k-1},t_k]) \leq M(f,[a,b]) = M$
• $m(f, [t_{k-1}, t_k]) \geq m(f, [a,b]) = m$

where $[t_{k-1},t_k] \subseteq [a,b]$. Intuitively, finer partitions capture the function more accurately, so the underestimate improves and the overestimate tightens (potentially decreases). Given this fact, we have the following:

$m(b-a) = m(f, [a,b])(b-a) \leq L(f,P) \leq U(f,P) \leq M(f,[a,b])(b-a) = M(b-a)$

Quick Breakdown:

Similar idea to show $m(f, [a,b])(b-a) \leq L(f,P) \leq U(f,P)$

Since $m(f,[t_{k-1},t_k]) \leq M(f, [t_{k-1}, t_k])$ (since $\inf \leq \sup$ by definition), we have $L(f,P) \leq U(f,P)$ for any single Partition P. ( Note: it is not immediately clear whether $L(f,P_1) \leq U(f.P_2)$ for different partitions $P_1,P_2$, a question we will address shortly.)

Given this proposition, we see these darboux sums are bounded and by consequence are guaranteed of the existence of an upper Darboux Integral $U(f)$ and lower Integral $L(f)$:

Upper Darboux Integral: $U(f) = \inf\{U(f,P) : P$ is a partition of $[a,b]$ $\}$. Often represented as $\overline{\int}_a^b f(x)dx$

Lower Darboux Integral: $L(f) = \sup\{L(f,P): P$ is a partition of $[a,b]$ $\}$. Often represented as $\underline{\int}_a^b f(x)dx$

Similar to how a limit exists if the LHS and RHS limits exist, we say $f$ is INTEGRABLE on $[a,b]$ if $U(f) = L(f)$ or $\overline{\int} a^b f(x)dx = \underline{\int}_a^b f(x)dx$

Darboux Integral: f is INTEGRABLE on $[a,b]$ if $U(f)=L(f)$. Often represented as $U(f)=L(f)=\int\limits_a^b f(x)dx$

where $f(x)$ can be seen as the height and $dx$ as the width of the rectangle approaching to 0.

Showing whether a function is integrable is a lot of work and thus will be omitted. Please read Ross Analysis or search online for an example. However, what I will show and find more interesting is to show a function $f$ this NOT integrable. In Engineering, we are always given ‘‘nice” functions that always have integrals but it turns out there are functions that don’t have an “area”. One potential example is comparing the area between two functions that appear to be the same but one has finite number of holes and the other has infinitely many holes in the graph. One is integrable and the other is not.

Example:

\(f(x) = \begin{cases} 1 & x \in \mathbb{Q} \\ 0 & x \notin \mathbb{Q} \end{cases}\) on $[a,b]$. Show that f is not integrable.

Note that every interval $[t_{k-1}, t_k]$ contains a rational and irrational number (i.e. by the density of $\mathbb{Q}$)

We have: \(\begin{align*} M(f, [t_{k-1},t_k]) = 1 \forall k &\implies U(f,P)=1(b-a) = (b-a) &&\implies U(f) = b-a \\ m(f, [t_{k-1},t_k]) = 0 \forall k &\implies L(f,P)=0(b-a) = 0 &&\implies L(f) = 0 \end{align*}\)

Thus $U(f) \ne L(f)$ and by definition, $f$ is not integrable on $[a,b]$

Any engineering student should be surprised of this result, especially on the idea that one can be asked to determine if a function is integrable or not. This is one of many examples that differentiates between Math and Engineering students. Math students are expected to not take anything for granted unless told otherwise (reality is that there are some materials that are just too complex at the moment to learn).

Previously, we saw that for any partition $P, U(f, P) \leq U(f,[a,b])$. We previously reasoned that a finer partition does a much more accurate job in capturing the function’s true behavior. Geometrically, we are summing more thinner rectangles whose lower and upper values $M_i$ and $m_i$ are tighter to the true value. Thus we can generalise the following:

Lemma 1: Let $f$ be bounded function on $[a,b]$. If $P,Q$ are partitions of $[a,b]$ with $P\subseteq Q$ (i.e. $Q$ is a “finer” partition within $[a,b]$ / has more points in the partition). Then:

$L(f,P) \leq L(f,Q) \leq U(f,Q) \leq U(f,P)$

Essentially the lemma is stating that adding points to the partition decreases the upper sum and increases the lower darboux sum. Geometrically, this would correspond to having more thinner rectangles to approximate the area more accurately.

Similarly, we can now make the claim that for any partition, the lower darboux sum will always be less than (or equal to) any upper darboux sum regardless of the partition.

Lemma 2: Let $f$ be bounded on $[a,b]$ and let $P,Q$ be partitions of $[a,b]$. Then $L(f,P) \leq U(f,Q)$

Proof: Consider the partition $P \cup Q$. Note that we know nothing about the partitions $P$ and $Q$ but we do know we can construct a larger partition $P\cup Q$ such that:

• $P\subseteq P \cup Q$
• $Q\subseteq P \cup Q$

So by the previous lemma (lemma 1),

• $L(f,P) \leq L(f, P \cup Q)$ (i.e. adding points to the partition increases the lower darboux sum)
• $U(f,P\cup Q) \leq U(f,Q)$ (i.e. adding points to the partition decreases upper darboux sum)

Chaining these two conclusion, we have: $L(f,P) \leq L(f, P \cup Q) \leq U(f,P\cup Q) \leq U(f,Q)$

Thus lemma 2 tells us tht any any lower sum is below any upper sum. But the supremum and infimum are the tightest bounds we can get. What does lemma 2 tells us about these bounds?

Lemma 3: Let $f$ be bounded on [a,b]. Then $L(f) \leq U(f)$

Previously in lemma 2, we proved that regardless of the partition, the upper darboux sum is always greater (or equal to) the lower darboux sum. This is an extension of the previous lemma but even stronger by tighting the bound such that even the GREATEST lower darboux sum is less than the LEAST upper darboux sum.

By the definition of Darboux Integrals, we know that a function is integrable if its lower and upper sum are equal (i.e. $L(f) = U(f)$). However, this definition requires us to verify this equality directly which at times can be difficult due to $U(f)$ and $L(f)$ being defined as the infima and suprema of the upper and darboux sums of an infinitely many bounded finite-partitions respectively. One can use the following characterisation to determine if a function is integrable:

Theorem 1 (Cauchy Criterion): a bounded function $f$ on $[a,b]$ is integrable $\iff \forall \epsilon > 0 \exists$ a partition $P$ of $[a,b]$ such that $U(f,P) - L(f,P) \lt \epsilon$

As one may recall from the previous course: For any $\epsilon > 0$, if the difference between two values is less than every positive $\epsilon$ then the difference must be 0.

A common example is the value $1.\overline{9} = 1.9999\dots$ (i.e. the 9 repeats infinitely many times).

No matter what $\epsilon \gt 0$ you choose, we have $2 - 1.\overline{9} \lt \epsilon$

For instance:

• $|2 - 1.9| < 0.1$
• $|2 - 1.99| < 0.01$
• $|2 - 1.999| < 0.001$

As we add more 9s, the difference gets smaller. In this limit, the difference converges to 0. Thus $2 - 1.\overline{9} = 0$ or $1.\overline{9} = 2$.

Many integration problems found in Engineering have one thing in common, they tend to be continuous? Is there a particular reason for this? It turns out that it is easy to create integration problems as long as the functions are continuous:

Theorem 2: Every continuous function $f$ on $[a,b]$ is integrable

However, sometimes a student can be blessed or cursed (depending on perspective) with a piecewise function. But in particular, these discontinuous functions tend to have one thing in common, they are montonic functions. Recall that a monotone function is a function that is either increasing or decreasing (not strictly) and do not have to be continuous. For instance, take a piecewise montonic function, the step function that jumps upward at finitely many points but never decreases. Professors don’t pose students integration problems out of nowhere, they rely on the next theorem to pose piecewise functions to students:

Theorem 3: Every monotone functions on $[a,b]$ is integrable

The next set of facts will be given without any commentary:

let $f,g$ be integrable on $[a,b]$ and let $c\in\mathbb{R}$. Then:

1. $cf$ is integrable with $\int^b_a cf = c\int^b_a f$ (scalar multiplication)
2. f+g is integrable with $\int_a^b (f + g) = \int_a^b f + \int_a^b g$ (additivity)
3. if $f(x) \leq g(x) \forall x\in [a,b]$ then $\int_a^b f(x)dx \leq \int_a^b g(x)dx$ (monotonicity)
4. if $g$ is a continuous non-negative function and $\int_a^b g(x)dx = 0$ then $g(x) = 0 \forall x\in [a,b]$
5. let $a \lt c \lt b$ and if $f$ is integrable on $[a,c]$ and on $[c,b]$ then $\int_a^b f = \int_a^c + \int_c^b$ (additivity of intervals)
6. $|f|$ is integrable on $[a,b]$ and $|\int_a^b f| \leq \int_a^b |f|$ (triangle inequality)

Illustration of $\|\int_a^b f\|$ on the left and $\int_a^b \|f\|$ on the right

Previously, we stated that if $f$ is either montone or continuous, then it is integrable. But not all functions are monotone nor continuous which could severely restrict what types of curves we can compute the area under. But what if we broke the functions into separate individual pieces? That is what piecewise functions attempt to do and we shall be exploring further the integrability of such functions.

Piecewise Monotone: $f$ is called piecewise monotone on $[a,b]$ if there is a partition $P$ of $[a,b]$ such that $f$ is monotone on each open subinterval $(t_{k-1},t_k)$.

A piecewise monotone function

Piecewise Continuous: $f$ is called piecewise continuous on $[a,b]$ if there is a partition $P$ of $[a,b]$ such that $f$ is uniformly continuous on each open subinterval $(t_{k-1},t_k)$

A piecewise continuous function

Theorem 4: If $f$ is piecewise continuous or (bounded) piecewise monotone on $[a,b]$, then it is integrable on $[a,b]$

With theorem 4, we have now covered the integrability of nearly all functions encountered in engineering and applied mathematics (at least at a freshman level).

Now we have covered enough to discuss the one of the biggest theorem in calculus: the fundamental theorem of calculus (FTC). Specifically we will cover the first part of FTC but essentially the fundamental theorem of calculus connects differentiation covered in the previous course with integration to show they are roughly inverses of each other. In addition, it also teaches us how to compute the integral of a function much more efficiently (at least by hand).

Theorem 5 (Fundamental Theorem of Calculus 1): if $f$ is continuous on $[a,b]$ and differentiable on $(a,b)$ and if $f’$ is integrable on $[a,b]$, then $\underbrace{\int_a^b f’(x)dx}_{\text{sum of infinitesimal changes}} = \underbrace{f(b)-f(a)}_{\text{total change}}$

The first part of the theorem tells us that integrating a derivative gives us the original function’s change. Recall that the derivative $f’(x)$ tells us the rate of change of the function $f$ at each point. Summing (i.e. integrating) all these infinitesimal changes from $a$ to $b$ should give the total change $f(b) - f(a)$.

Thus, by the fundamental theorem of calculus part 1, we have a practical method to compute definite integrals. Instead of constructing darboux sums, we can use antiderivatives.

You may see the term definite and indefinite integrals thrown around, essentially one specifies its bound and the other does not. But here’s a more complete definition:

• Definite Integral: the integral has bounds $[a,b]$ and evaluates to a single real number
• Indefinite Integral: the integral has no bounds and represents a family of antiderivatives $F(x) + C$ where $C\in\mathbb{R}$

Suppose $a \lt b$, then what would it mean if we switch the bounds around? (i.e. $\int_b^a f(x)dx$?). By the fundamental theorem of calculus part 1, we have:

So if we were to swap the bounds, we would have: $\int_b^a f(x)dx = F(a) - F(b) = - (F(b) - F(a)) = - \int_a^b f(x)dx$

If $a \lt b$ then $\int_b^a f(x)dx = - \int_a^b f(x)dx$

If $F$ is an antiderivative of $f$ (i.e. $F’ = f$), then:

As a consequence, we also have the following (the second part of the FTC):

While FTC 1 lets us compute definite integrals using antiderivatives, FTC 2 guarantees that the antiderivatives exist.

Theorem 6 (Fundamental Theorem of Calculus 2): let $f$ be integrable on $[a,b]$ and let $F(x) = \int_a^x f(t)dt$. Then $F$ is continuous on $[a,b]$. Moreover, if $f$ is continuous at $x_o\in(a,b)$ then $F$ is differentiable at $x_o$, thus $F’(x_o) = f(x_o)$.

This is a complicated way of saying the derivative undoes the integration: $\frac{d}{dx}\int f(x)dx = f(x)$

Connection to Theorem 2: Recall from Theorem 2 that every continuous function on [a,b] is integrable. By FTC 2, we not have an explicit statement that every continuous function has an antiderivative.

On the same subject, it is important to note that even if an antiderivative exists, it does NOT mean it is integrable. There is a strong reason for why the formal definition of the FTC 2 requires $f$ to be continuous (or at least integrable). Having an antiderivative is not enough.

Example:

$F(x)$ is differentiable everywhere but DIFFERENTIABILITY $\bcancel{\implies}$ Integrability

For $x \neq 0, f(x) = F’(x) = 2x\sin(\frac{1}{x}) + x^2(\cos(\frac{1}{x}))(\frac{-1}{x^2}) = 2x\sin(\frac{1}{x}) - \cos(\frac{1}{x})$

As $x\to 0$, it widly oscillates between 1 and -1 according to Wikipedia and thus is discontinuous around the neighborhood $x = 0$. With some math I won’t get into, this function apparently broke the accepted knowledge of what functions were differentiable everywhere and its derivative was bounded, the derivative was always integrable. As this is a first-year course in Calculus, we are strictly referring Integrability as Riemann/Darboux integrable. Using the definitions presented in the blog, this function is not Darboux Integrable.

Integration Techniques

Thus far we have covered what it means to take an integral of a function and what sorts of functions are integrable. Let us now use the fundamental theorem of calculus part 1 and various integration techniques to compute integrals.

Here are some basic integration rules to know:

• constant: $\int a dx = ax + C, a\in\mathbb{R}$
• power rule: $\int x^n dx = \frac{x^{n+1}}{n+1} + C, n\neq -1$
• reciprocal: $\int \frac{1}{x}dx = \ln |x| + C$
• sum/difference: $\int [f(x) \pm g(x)]dx = \int f(x)dx \pm \int g(x)dx$

Exponential and Logarithmic Functions:

• $\int e^x dx = e^x + C$
• $\int a^x dx = \frac{a^x}{\ln(a)} + C, a \gt 0, a\neq 1$
• $\int \ln(x) dx = x\ln(x) - x + C$
• $\int \frac{1}{x\ln (x)} dx = \ln |\ln(x)| + C$

Trigonometric Functions:

• $\int \cos(x) dx = \sin(x) + C$
• $\int \sin(x) dx = -\cos(x) + C$
• $\int \sec^2(x)dx = \tan(x)+C$
• $\int \sec(x)\tan(x)dx = \sec(x) + C$
• And many more …

The issue with computing integrals is that it tends to be an exercise of memorisation and practice to know what techniques to use and what the anti-derivatives are. Hence why I don’t find this section particular interesting. But nonethless, it is important to go through different common integration techniques one needs to know to survive their calculus course.

Most functions have no anti-derivatives, and thus Mathematicians have discovered or invented techniques to tackle each problem. Before the release of LLMs, integration calculators were notorious for giving complicated solutions to integral problems as they often employed their own common generic integration techniques to solve the problem which weren’t student friendly solutions.

The most basic and fundamental integration technique is substitution which is essential to understand. It is the most fundamental integration technique that gets employed in many other integration techniques so it is primordial to master it.

Substitution Rule: $\int_a^b f(g(x))g’(x)dx = \int_{u(a)}^{u(b)} f(u)du, $ where $u = g(x)$

The substitution rule works by recognising composite functions. When you see an integrand with “chain rule” structure, you can often times simplify by substituting the problem to an easier integral. This looks more daunting than it should be so let’s look at a simple example:

Example: $\int_0^1 e^{2x} dx$

The integrand $e^{2x}$ is composite: $f(x) = e^x$ and $g(x)$ = 2x, so $f(g(x)) = e^{2x}$.

Our basic rule $\int e^x dx = e^x + C$ does not directly apply because of the coefficient 2 on $x$. This is where substitution helps:

Let $u = 2x$. Then:

Thus we can rewrite the integral as:

The integrand now is in the correct form and thus is solvable.

However, as the question posed a definite integral, we must account for the bounds? There are two approaches:

1. Change of Bounds Determine what the lower and upper bounds are by computing $u(a)$ and $u(b)$ are

   \[x = 0: u(0) = 2(0) = 0 \\ \nonumber x = 1: u(1) = 2(1) = 2 \nonumber\]

   So now we can integrate directly:

   \[\begin{align*} \frac{1}{2}\int_0^2 e^u du &= \frac{1}{2} e^u|_0^2 \\ &= \frac{1}{2}[e^2 - e^0] \\ &= \frac{e^2 - 1}{2} \end{align*}\]

   The benefit with this approach is that we can work entirely in terms of $u$, there is no need to substitute back

2. Substitute Back: Solve the indefinite integral first, then substitute back $u = 2x$ before evaluating bounds:

   \[\begin{align*} \int_0^1 e^{2x}dx &= \frac{1}{2}\int_{x=0}^{x=1} e^u du\\ &= \frac{1}{2} e^u|_{x=0}^{x=1} \\ \end{align*}\]

   As the bounds are respect to $x$, let’s substitute back our original function $u = 2x$

   \[\begin{align*} \frac{1}{2} e^u|_{x=0}^{x=1} &= \frac{1}{2}e^{2x}|_0^1 \\ &= \frac{1}{2}[e^{2(1)} - e^{2(0)}] \\ &= \frac{e^2 - e^0}{2} \\ &= \frac{e^2 - 1}{2} \end{align*}\]

I always stress to my students to verify their work if feasible (i.e. if time permits and isn’t overly complex to do so) by taking the derivative of their indefinite integral and see if it matches the original integrand:

This matches our original integrand $e^{2x}$, so we can be confident our integration is correct.

This verification step catches algebraic errors, wrong coefficients, and misapplied rules. If your derivative doesn’t match the integrand, you have made a mistake somewhere.

The Substitution rule has various names including u-subitution rule or change of variables but the most important thing to take from this rule is its ability to simplify the integrand and it can potentially achieve this by cancelling a term due to the derivative of $u$.

Example:

For instance, consider the following problem: $\int \frac{x}{x^2+1}dx$

This problem looks very complicated but if we let $u = x^2+1$ then:

Resulting in the following:

Check:

The next integration technique is called Integration By Parts, useful when the integrand is a product of two separate functions where one becomes simpler when differentiated:

Integration By Parts: $\int uv’dx = uv - \int vdu$

Example: $\int x\sin(x) dx$

The integrand is a product of two functions $x$ and $\sin(x)$.

Choosing which function is to be $u$ or $v’$ takes practice but in general, I like to always default choosing $u$ to be a polynomial function like the linear function $x$ if applicable, else consider whose integral (antiderivative) is the least difficult. According to Wikipedia, there is the LIATE rule, a rule I never heard of till now. This rule is a general guide to choosing the $u$:

1. L Logarithmic functions
2. I Inverse Trigonometric Functions: $\arctan(x), \text{arcsec}(x)$, etc.
3. A algebraic functions such as polynomials
4. T: trigonmetric functions: $\sin(x), \tan(x)$, etc.
5. E: Exponential functions

Based on my personal experience and this rule, choose $u = x$ and $v’ = \sin(x)$. Thus we have:

Apply the integration by parts rule, we have:

Check: \(\begin{align*} (-x\cos(x) + \sin(x) + C)' &= \bcancel{-\cos(x)} + (\bcancel{-}x)(\bcancel{-}\sin(x)) + \bcancel{\cos(x)} + 0 \\ &= \boxed{x\sin(x)} \end{align*}\)

One trick to recall the rule is to start with the product rule and get the left side to be in the form $uv’$:

From my time as a teaching assistant in Calculus for Engineers, I learned there is an integration by parts table that one could adopt to organise the various substitution required to the problem. This is particular useful when the question requires to apply the rule multiple times. Perhaps I’ll cover this in the future.

The integration by parts can be an effective tool to solve problems that doesn’t seem to be a product of two functions such as $\int \ln(x)dx$. Recall that a product between 1 and any function is the function itself. This itself is a product:

Example:

Based on LIATE rule, logarithmic takes precedent over algebraic functions thus take $u = \ln(x)$

Thus,

Check:

Trigonometric Substitution:

Consider the following integration: $\int \frac{1}{\sqrt{4-x^2}}dx$. One’s first instinct may be to utilise substitution or integration by parts. However this will likely end poorly. It turns out this class of problems requires trigonometric substitution. Whenever one sees the following expressions in the following form, it is best to consider trigonmetric substitution first:

• $\sqrt{a^2-x^2}$
• $\sqrt{a^2+x^2}$
• $\sqrt{x^2-a^2}$

The key to knowing what trig to substitute x for lies in the identities which are convienently layed out below:

Trigonometric Substitution:

form	substitution	identity used
$\sqrt{a^2-x^2}$	$x=a\sin\theta$	$1-\sin^2\theta = \cos^2\theta$
$\sqrt{a^2+x^2}$	$x=a\tan\theta$	$1+\tan^2\theta = \sec^2\theta$
$\sqrt{x^2-a^2}$	$x=a\sec\theta$	$\sec^2\theta-1=\tan^2\theta$

Let’s see how this works in practice by looking at the motivating example: $\int\frac{1}{\sqrt{4-x^2}}dx$

Step 1: Identify the form and choose a substitution

The expression $\sqrt{4-x^2}$ fits the form $\sqrt{a^2-x^2}$ where $a = 2$.

By the table above, use $x = 2\sin\theta$

Step 2: Compute the derivative and simplify the square root

From $x = 2\sin\theta$: \(\begin{align*} x &= 2\sin\theta \\ \frac{dx}{d\theta} &= 2\cos\theta \\ dx &= \boxed{2\cos\theta d\theta} \end{align*}\)

Simplify $\sqrt{4-x^2}$:

Let’s restrict $\theta \in [\frac{-\pi}{2},\frac{\pi}{2}]$ so $\cos\theta \geq 0$. Thus $|\cos\theta| = \cos\theta$.

Step 3: Substitute into the integral

Step 4: Substitute $x$ back

Recall that $x = 2\sin\theta$,

Thus, $\int\frac{1}{\sqrt{1-x^2}}dx = \theta = \arcsin(\frac{x}{2})$

Check:

Hopefully, this motivating example gave you a glimpse of the power of trig substitution. There are many more interesting questions that one could solve that requires one to recall special triangles and the SOH CAH TOA rule to convert trig functions in respect to $\theta$ back to $x$ but will be a story for another time. But as an exercise, derive the area of a circle of radius r, you should get $\pi r^2$.

Partial Fractions:

Another class of rational functions that can be integrated utilises a technique called partial fractions but this only works (from my memory) with rational functions whose numerator and denominators are polynomials. The reason is quite simple, this technique relies on decomposing rational functions into smaller pieces.

Decomposition: for a rational function $\frac{p(x)}{q(x)}, it can always be decomposed into smaller pieces, each of which can be integrated

Example: $\frac{1}{x^2-1} = \frac{\frac{1}{2}}{x-1} - \frac{\frac{1}{2}}{x+1}$

A condition for partial fractions in general is that deg$(p(x)) \lt $deg$q(x)$. If deg$(p(x)) \geq $deg$q(x)$, use polynomimal long divisionto separate the polynomial part from the remainer:

Then apply partial fractions to $\frac{r(x)}{q(x)}$, where now deg$(r) \lt $deg$(q)$.

The form of the partial fraction decomposition depends on the type of factors in the denominator:

• Case 1: Distinct Linear Factors

  After factoring, if the denominators have distinct roots then we have each term in the form: $\frac{A}{ax+b}$

• Case 2: Repeated Linear Roots

  After factoring, if the denominator has repeated roots (i.e. $(ax+b)^n$), then we need to create a term for each and every power up to $n$:

  \[\begin{align*} \frac{A_1}{ax+b} + \frac{A_2}{(ax+b)^2} + \frac{A_3}{(ax+b)^3} + \cdots + \frac{A_n}{(ax+b)^n} \end{align*}\]

• Case 3: Distinct Irreducible Quadratic Factors

  Not all quadratics are reducible using real numbers and thus remain in the form: $ax^2+bx+c$. Then the term will have a corresponding partial fraction term: $\frac{Ax+B}{ax^2+bx+c}$

• Case 4: Repeated Quadratic Factors

  Similar to case 2, if you notice irredicuble quadratic terms in the denominator of our rational functions, we need to build up the terms repeatedly with higher powers:

  \[\begin{align*} \frac{A_1x+B_1}{ax^2+bx+c} + \frac{A_2x+B_2}{(ax^2+bx+c)^2} + \frac{A_3x+B_3}{(ax^2+bx+c)^3} + \cdots + \frac{A_nx+B_n}{(ax^2+bx+c)^n} \end{align*}\]

The constant terms $A_i, B_i$ are determine using linear algebra. Let’s go through a simple example:

$\int \frac{1}{4x^2-1}dx$

Immediately you should notice that this is in the form $a^2-b^2$ thus we can utilise what we saw in Highschool (differences in squares): $a^2-b^2 = (a+b)(a-b)$: $4x^2-1 = 4x^2 - 1^2 = (2x+1)(2x-1)$

Based on our general rule, we have two distinct roots and thus each term corresponds to a linear root: $\frac{A}{ax+b}$. Thus we reduced the rational function into the following parts:

From here, we are left with two unknowns: $A, B$:

The idea is to group each order as its own term such that we can solve the coefficients since the functions ${1, x, x^2, x^3, \dots}$ are linearly independent in the vectorspace for $x\in\mathbb{R}$

Thus we have:

• 1: $1 = B-A \implies B = 1 + A$

• $x$: $0 = A+B$

  \[\begin{align*} 0 &= A+B\\ A &= -B \\ &= -(1+A) \\ A &= -1 -A \\ 2A &= -1 \\ A &= \boxed{\frac{-1}{2}} \end{align*}\]

Plugging $\boxed{A = \frac{-1}{2}}$ into $B = 1 + A$, we have $\boxed{B = \frac{1}{2}}$

Or alternatively, we could try to utilise nice numbers such as 0 or the roots to the equation to cancel out some terms to retrieve $A$ and $B$ more quickly:

Consider: $1 = A(2x-1) + B(2x+1)$

• if $x = \frac{1}{2}$: \(\begin{align*} 1 &= A(0) + B(2) B &= \frac{1}{2} \end{align*}\)
• if $x = \frac{-1}{2}$: \(\begin{align*} 1 &= A(-2) + B(0) A &= \frac{-1}{2} \end{align*}\)

By choosing the roots, we remove one term at a time, isolating each constant. This is faster than expanding and comparing coefficients. Plugging in nice numbers such as $x = 0$ at times is sufficient as well.

Therefore, the integral is now:

A much more manageable smaller pieces of integrals that can now be solved:

Check:

Improper Integrals:

Thus far, we have studied integral of functions over closed, bounded intervals $[a,b]$. Even our integrability conditions required bounded domains. But there is another class of integrals called the IMPROPER INTEGRALS, where either:

• the domain extends to infinity OR
• the function has a discontinuity at an end point

The area under the curve of $\frac{1}{x^2}$ from $x\geq 1$

The remarkable result is that even though the domain is unbounded, the area is finite (equals to 1).

Improper Integrals: Without loss of generality, consider $[a, b)$, where $b$ is finite or $\infty$. Let $f$ be defined on $[a,b)$ and integrable on each $[a,d]$ for $a \lt d \lt b$.

We define $\int_a^b f(x)dx = \lim\limits_{d\to b^{-}}\int_a^d f(x)dx$, provided the limit exists

Consider the example: $\int_1^\infty \frac{1}{x^2}dx$:

Let’s consider the region between $(0, 1]$. Intuitively, since the interval length is only 1 compared to $[1, \infty)$ which has infinite length, surely the area should be finite, right?

Wrong! It turns out that $\int_0^1 \frac{1}{x^2}dx$ diverges (or converges to $\infty$), despite the bounded domain.

As $0\notin dom(f)$, the integral is improper:

We can generalise this behavior by solving $\int_1^\infty \frac{1}{x^p}$ and $\int_0^1 \frac{1}{x^p}$.

and

Integral	p < 1	p = 1	p > 1
$\int_0^1 \frac{1}{x^p} dx$	Converges	Diverges	Diverges
$\int_1^\infty \frac{1}{x^p}dx$	Diverges	Diverges	Converges

What we can conclude from the past few examples is that the size of the domain does not matter, but rather its the behaviour of the function within the interval that matters. An obvious result but sometimes taken forgranted.

Recall in the previous course, we explore the $p$-test which stated:

P-Series: $\sum \frac{1}{n^p}$ converges if $p \gt 1$ and diverges if $p\leq 1$

$\int_1^\infty \frac{1}{x^p}dx$ and $\sum\limits_{n=1}^\infty \frac{1}{n^p}$ both diverge when $p \leq 1$. Is there a pattern between the two? Turns out yes and it’s called the integral test:

Integral Test for Infinite Series: let $N\in\mathbb{Z}$ and suppose $f(x)$ is continuous, decreasing and non-negative on $[N, \infty]$ then $\sum\limits_{n=N}^\infty f(n)$ converges/diverges $\iff$ $\int_N^\infty f(x)dx$ converges/diverges respectively

It can be difficult to determine whether an infinite series converge or diverge, so the Integral test will be a handy tool.

If an integral is improper at both endpoints, one must split it up into 2 separate improper integrals and evaluate each integral separately.

Improper Integrals (Improper at Both Ends): If $\int_a^b f(x)dx$ is improper at both ends, then by definition $\int_a^b f(x)dx = \int_a^c f(x)dx + \int_c^b f(x) dx$, where $c\in(a,b)$ and $f$ is defined at $c$.

If one of the two improper integrals evaluates to $\infty$ and the other is $-\infty$, then the integral is undefined

For instance, consider the following example:

Area of $\frac{1}{1+x^2}$

The two endpoints are improper and thus are broken into two pieces:

1. $\int_{-\infty}^0 \frac{1}{x^2+1}dx$
2. $\int_0^\infty \frac{1}{x^2+1}dx$

Important: Both pieces must converge for the original integral to converge. If even one diverges, the entire integral diverges.

The problem will be left as an exercise to the readers.

A common mistake is to compute an improper integral that is improper at both endpoints as one:

By symmetry, this seems fine. But it is invalid because the integral is improper at both endpoints and thus must be split. We will soon see that the integral is in fact undefined:

Left Piece:

Right Piece:

As one diverges to $-\infty$ while the other to $\infty$, we cannot treat the two as equal opposites and thus they do not cancel each other.

Power Series

Have you ever wondered how a calculator computes transcendentals such as $\sin x$ or $e^x$? The secret lies in power series, infinite sums of terms that represent these functions accurately. Think of a power series as a polynomial with infinitely many terms. For instance, consider $e^x$, it can be represented as the following:

Evidentally, calculators must give an answer and thus cannot sum an infinite amount of terms. Therefore, it is important for us to know how many terms are enough to sufficiently give us an answer that is precise to our needs. This approximation is a subject that is also explored in this course.

Power Series: let $(a_n)_{n=0}^\infty$ be a sequence. The series $\sum\limits_{n=0}^\infty a_n x^n$ is called a power series (centered at 0) with coefficients $(a_n)^\infty_{n=0}$

In the previous course, we studied series convergence:

And utilised tests (p-test, ratio test, comparison test, etc.) to answer whether the series converge or diverge. With power series, we face a new question:

For which values of $x$ does this series converge? The questions and answers we seek is no longer simply whether a series converge but also to determine the interval of x-values where converge occurs (called the radius of convergence). In power series, convergence is dependent on $x$, for instance:

By ratio test:

This power series converges only when $x = 0$. This is consistent to a fact about power series, they always converge when $x = 0$ unsurprisingly.

Fact 1: every power series coverges at $x = 0$ as $0^0 = 1$

Consider another example: $\sum\limits_{n=0}^\infty x^n$. By geometric series where $x = r$, we have $\sum\limits_{n=0}^\infty x^n = \frac{1}{1-x}$ which converges when $|x| \lt 1$.

Thus, in this example, we have the power series converge when $x \lt 1$, a larger interval compare to the previous example which converged only at one point.

Now consider the last example: $e^x = \sum\limits_{n=0}^\infty \frac{x^n}{n!}$. This is a power series with $a_n = \frac{1}{n!}$. By fixing $x$ and applying the ratio test, we have the following:

So by ratio test, $\sum\limits_{n=0}^\infty \frac{x^n}{n!}$ converges and since $x$ was arbitrary, it converges $\forall x\in\mathbb{R}$

The last 3 examples illustrates 3 possibilities for a power series $\sum a_nx^n$, it can either:

1. converge only at $x = 0 \implies R = 0$
2. converges for all $x$ in a bounded interval certered at 0
3. converges $\forall x\in\mathbb{R} \implies R = \infty$

Theorem 7 (Radius of Convergence): For the power series $\sum\limits_{n=0}^\infty a_n x^n$, let $R = \lim\limits_{n\to\infty} \bigg| \frac{a_n}{a_{n+1}} \bigg|$

Then $\sum\limits_{n=0}^\infty a_n x^n$ converges absoutely for $|x| \lt R$ and diverges for $|x| \gt R$

where $R$ is called the radius of convergence of $\sum\limits_{n=0}^\infty a_n x^n$

Note: Endpoints to be checked separately

Example: Find radius of convergence of $\sum\limits_{n=1}^\infty \frac{x^n}{n}$

$\sum\limits_{n=1}^\infty \frac{x^n}{n} = \sum\limits_{n=1}^\infty a_nx^n$, where $a_n=\frac{1}{n}$

Since this limit exists, $R = 1$. Hence, $\sum\limits_{n=1}^\infty \frac{x^n}{n}$ converges $\forall x$ with $|x| \lt 1$ i.e. $\forall x\in(-1,1)$

We need to check the endpoints separately since it may converge over there as well.

For $x = -1$:

This converges by the alternating series test.

For $x = 1$:

Recall this is a harmonic series and thus diverges

Thus, $\sum\limits_{n=1}^\infty \frac{x^n}{n}$ converges for $x\in[1, 1)$.

Note that the radius of convergence is still 1, but we call $[-1,1)$ the interval of convergence which could be larger than your radius of convergence itself.

Center of Power Series: Power series can be centered at values other than 0. For a power series centered at $x = x_o$, it is represented as $\sum\limits_{n=0}^\infty a_n(x-x_o)^n$

The power series converges on $|x-x_o|\lt R$ and diverges on $|x-x_o| \gt R$. Again, the endpoints needs to be checked separately.

Example: Find the interval of convergence of $\sum\limits_{n=0}^\infty \frac{5^n}{n}(x-2)^n$

1. Find the radius of convergence: $a_n = \frac{5^n}{n}$ so:

   \[\begin{align*} \lim\limits_{n\to\infty} \bigg|\frac{a_n}{a_{n+1}}\bigg| &= \lim\limits_{n\to\infty}\bigg|\frac{\frac{5^n}{n}}{\frac{5^{n+1}}{n+1}} \\ &= \lim\limits_{n\to\infty} \left( \frac{5^n}{n} \right)\left(\frac{n+1}{5^{n+1}}\right) \\ &= \frac{1}{5}\lim\limits_{n\to\infty} \frac{n+1}{n} \\ &= \frac{1}{5} \end{align*}\]

   Thus $\boxed{R = \frac{1}{5}}$ and $x\in(x_o - R, x_o + R) = (2 - \frac{1}{5}, 2 + \frac{1}{5}) = (\frac{9}{5}, \frac{11}{5})$

2. Check endpoints: For $x = \frac{9}{5}$:

   \[\begin{align*} \sum\limits_{n=1}^\infty\frac{5^n}{n}\left(\frac{9}{5} - 2\right)^n &= \sum\limits_{n=1}^\infty\frac{5^n}{n}\left(\frac{-1}{5}\right)^n \\ &= \sum\limits_{n=1}^\infty\frac{5^n}{n}\left(\frac{(-1)^n}{5^n}\right) \\ &= \sum\limits_{n=1}^\infty\frac{\bcancel{5^n}}{n}\left(\frac{(-1)^n}{\bcancel{5^n}}\right) \\ &= \sum\limits_{n=1}^\infty \frac{(-1)^n)}{n} \end{align*}\]

   By the alternating test, the series converges

   For $x = \frac{11}{5}$:

   \[\begin{align*} \sum\limits_{n=1}^\infty\frac{5^n}{n}\left(\frac{11}{5} - 2\right)^n &= \sum\limits_{n=1}^\infty\frac{5^n}{n}\left(\frac{1}{5} - 2\right)^n \\ &= \sum\limits_{n=1}^\infty \frac{1}{n} \end{align*}\]

   As this is a harmonic series, the series converges at $x= \frac{11}{5}$

Therefore, the interval of convergence for this power series centered at $x_o = 2$ is $[\frac{9}{5}, \frac{11}{5})$

As power series are a function of $x$ on its interval of convergence, a natural question in calculus is:

• is it continuous
• is it differentiable
• is it integrable

Power series is an accumulation over a sequence of functions, more specifically an accumulation of polynomials which we know are continuous. However, just because each function in a sequence is continuous and converges to some function $f$, it does not mean the convergent function $f$ is continuous itself. Therefore, to guard this continuity property in convergent functions, we will need a different notion or type of convergence, the uniform convergence. First we shall describe a type of convergence that does not guarantee continuity to the convergent function:

Pointwise Convergence: Let $(f_n)$ be a sequence of funtions. We say that $(f_n)$ CONVERGES POINTWISE to $f$ on a set $S$ if $\forall x\in S$, the sequence $f_n(x)$ converges to $f(x)$

Note: Does not guarantee $f$ is continuous even if each $f_n$ is continuous

That is to say:

Pointwise Convergence: $f_n\to f$ pointwise on $S$ if $\forall x\in S, \lim\limits_{n\to\infty} f_n(x)=f(x)$

i.e. $\forall x\in S, \forall \epsilon \gt 0 \exists N $ such that $n \gt N \implies |f_n(x)-f(x)|\lt \epsilon$

A counterexample to the preservation of continuity to the convergent function presented in the textbook is the following sequence $f_n(x) = x^n$ on $[0,1]$. $f_n \to f$ pointwise on [0,1] but is not continuous as:

Similarly to how uniform continuity differs from regular continuity by choosing a $\delta$ not dependent on $x$, uniform convergence also tries to pick a $N$ such that it only depends on $\epsilon$ and not $x$. The idea is to have the values $f_n(x)$ be ``uniformly” close to the values of $f(x)$ for all $x$:

Uniform Convergence: The sequence of function $(f_n)$ on $S\subseteq \mathbb{R}$ CONVERGES UNIFORMLY to a function $f$ on $S$ if $\forall \epsilon \gt 0, \exists N $ such that $n \gt N\implies |f_n(x) - f(x)|\lt \epsilon \forall x\in S$

Uniform convergence is a stronger type of convergence as it applies to all $x$ and thus we have the following relation:

$(f_n)$ converges to $f$ uniformly $\implies (f_n)$ converges to $f$ pointwise

This should not come to no surprise since if $(f_n)$ converges uniformly, it should converge to $f$ in a subset of $\mathbb{R}$.

The contraire does not hold unexpectedly:

$(f_n)$ converges to $f$ pointwise $\bcancel{\implies} (f_n)$ converges to $f$ uniformly

Example: $f_n(x) = x^n$ on $[0,1]$. Recall that $f_n$ converges pointwise to:

But does it converge uniformly as well? Spoiler it does not, the reason we motivated uniform convergence is to show the preservation of continuity which will come next.

Suppose by contraction that it does.

Then take $\epsilon = \frac{1}{2}$, I can find $N$ such that

$n \gt N \implies |f_n(x) - f(x)| \lt \frac{1}{2} \forall x \in [0,1]$.

In particular, for $n = N+ 1 \gt N$ and $\forall x\in [0,1)$, we have:

$f(x) = 0 \implies |f_{N+1}(x) - f(x)| = |f_{N+1}(x)| = x^{N+1} \lt \frac{1}{2} \forall x\in [0,1)$.

Rearranging,

From here, I can think of two approaches:

1. Limit Argument: Since $x^{N+1}$ is continuous and $x^{N+1} \lt \frac{1}{2} \forall x\in [0,1)$:

   We have $1 = \lim\limits_{x\to 1^-} x^{N+1} \lt \frac{1}{2}$ which is a contradiction.

2. Explicit Construction: We begin with the obvious fact that $1 \lt 2$ and build up to get $1 \lt 2^\frac{1}{N+1}$ and then choose an $L$ such that a contradiction arrives: \(\begin{align*} 2 \gt 1 &\implies 2^\frac{1}{N+1} \gt 1^\frac{1}{N+1} \\ &\implies 2^\frac{1}{N+1} \gt 1 \end{align*}\)

   By the density of $\mathbb{Q}, \exists L\in\mathbb{Q}\subset \mathbb{R}$ such that $\frac{1}{2^\frac{1}{N+1}} \lt L \lt 1$.

   By choosing such an $L$ that meets this criteria, we have $L\in[0,1)$ such that $L \gt \frac{1}{2^\frac{1}{N+1}}$ but by (*) we also have $L \lt \frac{1}{2^\frac{1}{N+1}}$ which is a contradiction.

Let us now show an example in how to prove something is uniform convergent:

Example: Let $f_n(x) = \frac{1}{n}\sin(nx)$ for $x\in\mathbb{R}$. Prove that $f_n\to 0$ uniformly.

Rough Work: Let $\epsilon \gt 0$ be given. Then,

Choose $N = \frac{1}{\epsilon}$.

Heurestic to show Uniform Convergence: In your rough work, you want to bound the function such that it is not dependent on $x$ such as $\frac{1}{n}$

e.g. take $N = \frac{1}{\epsilon}$ for instance where it does not depend on $x$ in a uniform convergence proof

An illustration how $f_n$ is bounded to the $\epsilon$ window and as $\epsilon$ gets smaller, the $f_n$ gets squeezed but remains in the window. i.e. Eventually as you go further down the sequence (a larger $N$), you will eventually find a function flat enough to squeeze into the window. Extracted from MIT OCW Real Analysis Notes

Now we have seen what it means for a function to converge pointwise and uniformly, let us now revisit the purpose of introducing a strongegr notion of convergence (uniform convergence):

$f_n$ is continuous $\forall n$ and $f_n \to f$ pointwise $\bcancel{\implies} f$ is continuous

But

$f_n$ is continuous $\forall n$ and $f_\to f$ uniformly $\implies f $ is continuous

Theorem (Uniform Convergence Preserving Continuity): let $(f_n)$ be a sequence of functions on $S$ such that $f_n \to f$ converges uniformly. If each $f_n$ is continuous, so is $f$

If we take the contrapositive of this theorem above, we obtain another tool to disprove uniform convergence:

Disproving Uniform Convergence via Continuity: If $f_n \to f$ pointwise and each $f_n$ is continuous but $f(x)$ is not continuous, then this convergence is not uniform

Before proceeding to the consequences of uniform convergence and the preservation of continuity, let’s revisit our favorite function in this chapter: $f_n(x) = x^n$ and recall that $f_n \to f$ pointwise on $[0,1]$ where,

This does not uniformly converge to $f$ as it was discontinuous at $x = 1$. But what about for $x\in[0,1)$?

For $f_n = x^n$ on $[0,1), \lim\limits_{n\to\infty} f_n(x) = 0 \forall x$

The constant function $0$ is continuous $\forall x$ but is it uniformly convergent on $[0,1)$? NO, we showed previously that it did not

What about from a fixed positive number $b \lt 1$. We will make the claim that on $[0,b], f_n \to 0$ uniformly:

Rough Work: Let $\epsilon \gt 0$. We want $N$ such that $n \gt N$:

Note: $b^n \lt \epsilon$ is only possible because $b \lt 1$ thus $b^n \lt 1$ (i.e. for $b\lt 1 \implies b^{N+1} \lt b^{N}$)

Thus we have,

Thus choose $N = \frac{\ln\epsilon}{\ln b}$

In conclusion, for the examples, we have seen for $f_n(x) = x^n$,

\(f(x) = \begin{cases} 0 & 0\leq x \lt 1 \\ 1 & x = 1 \end{cases}\)

• on $[0,1], f_n \to f$ pointwise but not uniformly
• on $[0,1), f_n \to 0$ pointwise but not uniformly
• on $[0,b]$ for $b \lt 1, f_n \to 0$ uniformly

I recall being a bit surprised by this result how fixing a point allows $f_n$ converge uniformly to $f$. But now that I think of it, if we were to start plugging in numbers as $x \to 1$, we’ll notice the following:

• $x = 0.5: 0.5^n \to 0$ fast ($N_1 > N$)
• $x = 0.9: 0.9^n \to 0$ slowly (i.e. $N_2 » N_1 > N$)
• $x = 0.99: 0.99^n \to 0$ even slower (i.e. $N_3 »> N_2 » N_1 > N$)
• $x \to 1: x^n \to 0$ extremely and arbitrarily slowly (i.e. $N_4 »»»»»»» N_3 »> N_2 » N_1 > N$)

Thus as $x\to 1$, we observe the growth to converge $f_n(x)$ to 0 get slower and slower

We can now say the following:

Theorem: let $(t_n)$ be a sequence of continuous function on $[a,b]$ which converges uniformly to $f$.

Then $\lim\limits_{n\to\infty} \int_a^b f_n(x)dx = \int_a^b \lim\limits_{n\to\infty} f_n(x)dx = \int_a^b f(x)dx$

Previously, we saw that $f_n = x^n$ does not converge uniformly from [0,1) as the terms converges to $f$ so slowly. In MATH1052, we introduced cauchy sequences, a sequence that converges and exhibits a behavior whereby after some $N$, the distance of any two functions in the sequence are arbitrarily so close that convergence is guaranteed.

Cauchy Sequences: a sequence $(s_n)$ is cauchy if $\forall \epsilon \gt 0, \exists N $ such that $m,n \gt N \implies |a_n-a_m| \lt \epsilon$

$(s_n)$ is cauchy $\iff (s_n)$ is convergent

We want a similar result for functions because if you recalled, this entire section is dedicated to power series which are infinite series of functions (even though it may not seem like it with our discussion in sequences and convergence).

Uniformly Cauchy: a sequence $(f_n)$ of functions on $S\subseteq \mathbb{R}$ is called UNIFORMLY CAUCHY if $\forall \epsilon \gt 0, \exists N$ such that $m,n \gt N \implies |f_n(x) - f_m(x)| \lt \epsilon \quad\forall x\in S$

To tie this together with uniform convergent, we have the following result:

Lemma: if $f_n \to f$ uniformly then it is uniformly cauchy

and

Theorem: Let $(f_n)$ be a uniformly cauchy sequence of functions on $S\subseteq \mathbb{R}$ then $\exists$ a function $f$ on $S$ such that $f_n\to f$ uniformly

Thus far, we have talked about functions in a sequence but never tied it with power series. Let us now be equipped with what it means for a series of functions to a power series:

$\sum\limits_{k = 0}^\infty g_k(x)$ is a power series of functions if $g_k(x) = a_kx^k$

For instance, the following function is a series of function but is not a power series of functions: $\sum\limits_{k=0}^\infty \frac{x^k}{1+x^k}$ because it is not in the right form.

Uniform Convergence of Function Power Series: let $\sum\limits_{k=0}^\infty g_k(x)$ be a series of functions on $S$ such that each $g_k$ is continuous. If $\sum\limits_{k=0}^\infty g_k(x)$ is uniformly convergent to $g(x)$, then $g$ is continuous

This result should not be of any surprise as we previously saw that

• sums of continuous functions are continuous
• uniform convergence preserves continuity among convergent sequences of functions

Let us now introduce another equivalence relation to uniformly cauchy in respect to function power series:

Uniform Cauchy Definition for $\sum\limits_{k=0}^\infty g_k(x)$: $\sum\limits_{k=0}^\infty g_k(x)$ is uniformly cauchy on $S$

$\iff \forall \epsilon \gt 0, \exists N $ such that $m,n \gt N \implies |\sum\limits_{k=0}^n g_k(x) - \sum\limits_{k=0}^m g_k(x)|\lt \epsilon$

$\iff \forall \epsilon \gt 0, \exists N $ such that $n\geq m \gt N \implies |\sum\limits_{k=m}^n g_k(x)| \lt \epsilon$

Now that we have seen what it means for a power series to uniformly converge, we can now introduce a nice test to determine whether it converges uniformly:

Weierstrass M-Test: let $(M_K)$ be a sequence of non-negative numbers with $\sum\limits_{k=0}^\infty M_k \lt \infty$. If $|g_k(x)| \leq M_k \forall x\in S$, then $\sum\limits_{k=0}^\infty g_k(x)$ converges uniformly on $S$

Let us see an example of how to utilise this test to determine whether an infinite function power series converges ot a continuous function:

Example: Show that $\sum\limits_{k=0}^\infty \frac{1}{n^2}\sin(nx)$ converges to a continuous function on $\mathbb{R}$

First note that: $|\frac{1}{n^2}\sin(nx)| = \frac{1}{n^2}|\sin(nx)|\leq \frac{1}{n^2}$

Let $M_n = \frac{1}{n^2}$. Then $\sum\limits_{k=0}^\infty M_n = \sum\limits_{k=0}^\infty \frac{1}{n^2} \lt \infty$ by P-series

So by M-test, $\sum\limits_{k=0}^\infty \frac{1}{n^2}\sin(nx)$ converges uniformly on $\mathbb{R}$ Since $g_n(x) = \frac{1}{n^2}\sin(nx)$ is continuous on $\mathbb{R} \forall n$ and converges uniformly, the limit must be continuous.

Recall: if $\sum\limits_{k=0}^\infty a_k$ is a series of numbers then $\sum\limits_{k=0}^\infty a_k$ converges $\implies \lim\limits_{n\to\infty} a_n = 0$

Though the opposite direction does not hold (think of $\sum \frac{1}{n}$ diverges by P-series but the limit is 0. We have an analogous version of this fact for series of functions:

if $\sum\limits_{k=0}^\infty g_k(x)$ converges uniformly on $S$ then $\lim\limits_{n\to\infty} \sup\{|g_n(x)| : x\in S\} = 0$

One consequence of this lemma above is that we can use the contrapositive to conclude whether a function power series does not converge uniformly, which is probably a much more useful fact:

if one can show $\lim\limits_{n\to\infty} \sup\{|g_n(x)| : x\in S\} \neq 0$, then one can conclude that $\sum g_k(x)$ does not converge uniformly on $S$

Example: Consider the power series $\sum \frac{x^n}{3^n}$

Before utilising our new tool to determine whether this power series converges uniformly, let’s establish the convergence interval:

Note that the radius of convergence is 3: $R = \lim\limits_{n\to\infty}\left|\frac{a_n}{a_{n+1}}\right| = \left|\frac{\frac{1}{3^n}}{\frac{1}{3^{n+1}}}\right| = 3$

At $x = 3: \sum 1 = \infty$ and at $x = -3: \sum(-1)^n$ diverges. Thus our interval of convergence is: $I = (-3,3)$

Using the new tool, we have $\sup\{|\frac{x^n}{3^n}| : x\in (-3,3)\} = 1 \forall n$ so $\lim_{x\to\infty} \sup\{|\frac{x^n}{3^n}| : x\in (-3,3)\} = 1 \neq 0$

So by the lemma, we know $\sum \frac{x^n}{3^n}$ does not converge uniformly on $(-3,3)$

But what if we restricted the interval to a concrete number similar to how we found $f_n(x) = x^n$ converges uniformly if we restricted $S$ from $[0,1)$ to $[0, b]$. It turns out if we restrict the endpoints of our interval of convergence to be smaller and concrete, our once non-uniformly convergent sequence/power series does converge.

Theorem: let $\sum a_n x^n$ be a power series with a radius of convergence $R \gt 0$. If $0 \lt b \lt R$, then the power series converges uniformly on $[-b,b]$ and converges to a continuous function on $(-R,R)$

Let’s revisit the previous example and restrict the interval to $b \lt R = 3$. For this exercise, we will avoid invoking the theorem above and show you the lengthy process to obtain the same result:

Let $b\in(0,3)$ so if $x = b\implies \sum \frac{b^n}{3^n} = \sum\left(\frac{b}{3}\right)^n$ converges by gemoetric series (as $0 \lt b \lt 3$)

1. To show $\sum \frac{x^n}{3^n}$ converges uniformly on $[-b,b]$, we can utilise the Weierstrass M-test: $\forall x\in [-b,b], \left|\frac{x^n}{3^n} \right | \leq \left|\frac{b^n}{3^n}\right| = \left(\frac{b}{3}\right)^n = M_n$. Since $\sum M_n$ converges, by the Weierstrass M-test, $\sum\frac{x^n}{3^n}$ converges uniformly on $[-b,b]$.

2. Now all that is left is to show $\sum \frac{x^n}{3^n}$ converges to a continuous function on $[-b,b]$:

   Note that $\frac{x^n}{3^n}$ is continuous on $\forall n$. Thus, $\sum\frac{x^n}{3^n}$ is continuous on $[-b,b] \forall b\in[0,3)$. As $b$ is arbitrary, $\sum \frac{x^n}{3^n}$ converges to a continuous function on $(-3, 3)$.

This is definitely much longer than simply invoking the theorem which gives us both uniformly convergence and convergence to a continuous function for free: Since $R = 3 \gt 0$ and $b \in (0,R)$, by the theorem above, we have $\sum \frac{x^n}{3^n}$ converges uniformly on $[-b,b]$ and converges to a continuous function on $(-3,3)$.

Recall that one of the motivations of utilising power series is to represent transcendental functions such as $e^x$ as an infinite series of polynomials. What would be desirable is if we could differentiate and integrate power series term by term. For instance, it would be nice if we could do the following

But for these properties to work, we must ensure that the radius of convergence remains the same in these new forms:

Lemma: If $\sum\limits_{n=0}^\infty a_n x^n$ has a radius of convergence $R$, then so does $\sum\limits_{n=1}^\infty na_nx^{n-1}$ and $\sum\limits_{n=0}^\infty \frac{a_n}{n+1}x^{n+1}$

As the radius of convergence remains the same even after taking the form of what seems like to be its differentiatiable and integrable equivalences, let us now introduce the theorem that ties this together to permit us to integrate and differentiate the series term by term:

Theorem: Suppose $f(x) = \sum_{n=0}^\infty a_nx^n$ has a radius of convergence $R \gt 0$. Then, $\int_0^x f(t)dt = \sum\limits_{n=0}^\infty \frac{a_n}{n+1}x^{n+1}$ for $|x| \lt R$

Theorem: Let $f(x) = \sum_{n=1}^\infty a_nx^n$ have radius of convergence $R \gt 0$. Then $f$ is differentiable on $(-R,R)$ and

$f’(x) = \sum_\limits{n=1}^\infty na_nx^{n-1}$ for $|x| \lt R$

Note: Notice that the index changes from 0 to 1 in the differentiable form. The index does not change for the integral

Thus far. the previous theorem only discusses differentiablity at the open intervals $(-R, R)$ but what about at its endpoint? That is what the next theorem, Abel’s Theorem, attempts to tackle. We know from last year that differentiability implies continuity (i.e. to be differentiable at $x = a$, then by definition, it must also be continuous at $a$)

Abel’s Theorem: let $f(x) = \sum a_nx^n$ have a radius of convergence $R \gt 0$, $R \lt \infty$. If $\sum a_nx^n$ converges at $x = R$, then it is continuous there. The same is true at $x = -R$.

As a consequence, we have that every power series converge to a continuous function on its entire interval of convergence including at its endpoint provided it is defined and does indeed converge there.

Every power series converges to a continuous function on its entire interval of convergence (including its endpoints if defined and converges there)

i.e. $\sum \frac{x^n}{n^2}$ has an interval of convergence (I.O.C) on $[-1,1]$ and thus is continuous on $[-1,1]$

Taylor Series

Recall that one of the motivations of power series is to represent transcendental functions as an infinite series of polynomials. This is the math that allows calculators be able to compute $e^x$ for instance within an acceptable error or to allow oneself to integrate a complicated expression into simple series of polynomials. However, how does one translate a function such as $e^x$ or $\sin(x)$ into a series of polynomials? That is what will be explored in this section, working with taylor series which gives us a methodical way to transform a function into a power series.

Taylor Series: let $f$ be defined on an open interval containing $x_o$. Suppose that all order of derivatives of $f$ exist at $x_o$. Then the series

$\sum\limits_{n=0}^\infty \frac{f^{(n)}(x_o)}{n!}\left(x-x_o\right)^n$ is called the taylor series for $f$ about $x_o$

For $n\geq 1$, the remainder $R_n(x)$ is defined by:

$R_n(x) = f(x) - \sum\limits_{k=0}^{n-1}\frac{f^{(k)}(x_o)}{k!}(x-x_o)^k$

When we approximate a function $f(x)$ using a finite Taylor polynomial of degree $n$ which we call $P_n(x)$, the approximation is rarely perfect. This difference must be taken into account and thus the remainder

On a note about the remainder, we also have this result:

$f(x) = \sum\limits_{k=0}^\infty \frac{f^{(k)}(x_o)}{k!}(x-x_o)^k \iff \lim\limits_{n\to\infty}R_n(x) = 0$

Suppose we have a power series in the form $f(x) = \sum a_n(x-x_o)^n$. We have never discussed how to determine what $a_n$ is. In our definition of taylor series, all order of derivatives of $f$ exists at $x_o$. This,

Let $x = x_o$, then:

Then by induction, we have:

Taylor Coefficient: $a_n = \frac{f^{(n)}(x_o)}{n!}$

Let us now go through an example to determine the taylor series of $\sin(x)$ centered at 0 (Note: This is also called a Maclaurin Series, a Taylor series centered at 0).

First let’s observe the orders of the derivatives of $\sin(x)$:

So we have: \(f^{(n)}(0) = \begin{cases} 0 & \text{if n is even} \\ 1 & \text{if n = 4k+1} \\ -1 & \text{if n = 4k+3} \end{cases}\)

and $a_n = \frac{f^{(n)}(0)}{n!}$ so the taylor series of $\sin(x)$ around 0 is

Let’s now explore the one applications of taylor series I hinted at in the beginning to better understand the power of formulating a function as a taylor series:

For Approximations:

How many terms do we need to approximate $\sqrt{e} = e^\frac{1}{2} = \sum\limits_{n=0}^\infty \frac{\left(\frac{1}{2}\right)^n}{n!}$ with an error less than $0.0001$?

Recall that $e^x = \sum\limits_{k=0}^\infty \frac{x^k}{k!} = \sum\limits_{k=0}^{n-1} \frac{x^k}{k!} + R(x)$

where $R_n(x) = \frac{f^{(n)}(y)}{n!}x^n = \frac{e^y}{n!}x^n$ for some $y\in(0,x)$

Since we are evaluate for $e^\frac{1}{2} = e^x \implies x = \frac{1}{2}$:

So a crude estimate would be that $e \lt 3$ (recall $e = 2.71828\cdots$)

We can bound $e^\frac{1}{2}$:

So we can bound the remainder:

If we were to iterate, the first time this condition holds true is when $n = 6$ and thus the error is less than $\frac{1}{6!2^5} \approx 0.000043$

While this does not give as the true error, it is nonethless cool that we can bound the error using what we know about Taylor series.

There are plenty more applications of taylor series such as solving tough integrals, solving the Basel problem, and solving differential equations. But that is all for this topic and my commentary on the course materials.

Resources

• MATH2052 Notes from Charles Starling
• Elementary Analysis: The Theory of Calculus by Kenneth A. Ross
• MIT OCW Real Analysis Notes

Note: I refer to this page as a commentary to the course or an accompanying to the course, these are not class notes. I the author of this page will omit many details and only present the materials and proofs to things I find personally noteworthy and the proofs presented may differ from those listed in the textbook or from the course notes itself.

To prevent tampering of permission flags of the process’s pages by sealing it, both the Linux Kernel and glibc introduced mseal() in the past year or two.

In an ELF program, the read-only section (.rodata) should not be writtable and we can confirm this via objdump that was indeed the intention with the existence of READONLY label:

$ objdump -h ./rodata_write | grep -A1 .rodata
$ objdump -h ./rodata_write | grep -A1 .rodata
./rodata_write:     format de fichier elf64-x86-64

--
13 .rodata       00000088  0000000000401358  0000000000401358  00001358  2**3
                 CONTENTS, ALLOC, LOAD, READONLY, DATA

Thus, when the program is executed, we should be expecting something like this in their mappings (see /proc/<pid>/maps)

00401000-00402000 r--p 00001000 00:2d 26970                              /tmp/rodata_write

where this region of memory is marked as read-only as expected (i.e. notice how it r--p and not rw-p for instance in this region).

However, if I invoke mprotect() which allows us to change protection of the calling process’s memory map, we can in fact modify the rodata. If I define a constant string “Hello”, we can see that it is stored in rodata section:

$ objdump -s -j .rodata ./rodata_write | grep -B2 Hello
Contenu de la section .rodata :
 401358 01000200 00000000 00000000 00000000  ................
 401368 48656c6c 6f000000 55736167 653a2025  Hello...Usage: %

Normally I would not be able to overwrite this section of memory as it does not have write permission but thanks to mprotect(), I could:

mprotect((void *)page_start, len, PROT_READ | PROT_WRITE)

As seen below:

$ ./rodata_write Bye
Before: Hello
After:  Bye

And if we were to check the mapping again, we’ll now see the same region of memory is now marked as write (w):

00401000-00402000 rw-p 00001000 00:2d 26970                              /tmp/rodata_write

But if I comment out the line containing mprotect(), we eventually will hit a segfault when trying to overwrite the const string residing in the .rodata section with “Bye”:

$ ./rodata_write Bye
Before: Hello
Erreur de segmentation     (core dumped)./rodata_write Bye

Hopefully this was a small enough example to illustrate what mseal() can be used for.

Resources:

• mseal() gets closer
• Introduction of mseal
• A deep dive into Linux’s new mseal syscall

Out of laziness, the code below is mainly generated by Claude but one can easily replicate this by following along trailofbits example of using mprotect() to obtain rootshell.

#define _GNU_SOURCE
#include <stdio.h>
#include <stdint.h>
#include <string.h>
#include <sys/mman.h>
#include <unistd.h>

static const char message[] = "Hello";

int main(int argc, char *argv[]) {
    if (argc < 2) {
        fprintf(stderr, "Usage: %s <replacement string>\n", argv[0]);
        return 1;
    }
    const char *replacement = argv[1];
    size_t repl_len = strlen(replacement);
    size_t msg_len = sizeof(message) - 1;  // exclude null terminator

    if (repl_len > msg_len) {
        fprintf(stderr, "Replacement too long: max %zu chars (got %zu)\n",
                msg_len, repl_len);
        return 1;
    }

    long page_size = sysconf(_SC_PAGESIZE);
    uintptr_t addr = (uintptr_t)message;
    uintptr_t page_start = addr & ~(page_size - 1);
    size_t len = (addr - page_start) + sizeof(message);
    printf("Before: %s\n", message);
    printf("PID: %d\n", getpid());
    getchar();
    if (mprotect((void *)page_start, len, PROT_READ | PROT_WRITE) != 0) {
        perror("mprotect");
        return 1;
    }

    // Zero out the old string first, then write replacement
    memset((void *)message, 0, msg_len);
    memcpy((void *)message, replacement, repl_len);

    printf("After:  %s\n", message);
    getchar();
    mprotect((void *)page_start, len, PROT_READ);

    return 0;
}

Thus far, we have covered the first 17 chapters covering topics ranging from how the OS ‘virtualises’ the CPU (aka give the illusion the CPU has the entire CPU to itself), how OS schedules processes up to address spaces which eventually builds up to Page tables (not yet covered but is the very next topic).

Here are some samples of what I shared with the group (I will be anonymising any names for privacy reasons). But to begin, here are some quotes I liked:

You should be using Emacs. If you are using vi, there is probably something wrong with you. If you are using something that is not a real code editor, that is even worse

and

Student: So we have to think? Well, I’m up for that. I mean, what else do I have to do anyhow? It’s not like I have much of a life outside of this book. Professor: Me neither, sadly. So let’s get to work!

Note: I use VIM and not emacs but nonetheless a funny comment

Chapters 1 and 2: Introduction

The first two chapters was an introduction to OS and the key concepts it plans to cover. Giving us a sneak preview of what is to come and does a good job explaining the role of the OS and how each piece is connected to each other. It mentions how OS can be seen as a resource manager where each CPU, memory, and disk is a resource and it is the OS’s job to manage these resources efficiently or “fairly” with others. Furthermore, it mentions how the OS gives the “illusion” that the system has a large of number of virtual CPUs to allow many programs to seemingly run at once by “virtualising” the CPU. It also mentions how the OS needs to provide primitives to address correctness in concurrent programs where a program can have many concurrent executing threads. There was a lot of topics covered so I cannot mention them all but it mentioned about the difference between kernel and user mode, what traps are briefly and the need for hardware support.

In response to someone’s comment on how the book uses the term “virtualisation” in a more theoretical sense rather than the commonly adopted meaning of virtualisation technology such as KVM and QEMU:

Virtualisation in a general sense is to create a “virtual” version of a something such as a resource. Two main virtualisation discussed briefly in this week’s reading is the CPU and memory. The OS gives the illusion to the process that it has the entire CPU and memory to itself. In reality, there are only a finite number of CPU cores and memory in the system. In other words, the process thinks it has all the memory and CPU to itself exclusively but in reality, it is sharing CPU and memory among many different processes. The OS is doing the work behind the scenes to schedule, allocate, and etc. Virtualisation removes the need for the process to be aware of other processes in the system to ensure it doesn’t touch other process’s memory and hog hardware resources. The OS has a way to ensure resources are shared among many processes and make it seem like multiple processes can run simulatenously (concurrency) on a single core CPU.

In sprit, virtualisation solutions such as virtual machines is similar to what the author is talking about. Maybe it’s a bit of a stretch but a VM is a virtual instance of a machine. It gives the guest OS the illusion that it has all the CPU and physical memory even though in reality it does not. The guest OS does not need to think about other the existence of the “OS” and processes in the host machine, it just focuses on itself. This provides both simplicity and isolation.

In response to their interpretation of the following text: “Don’t forget that each instruction of the program is in memory too; thus memory is accessed on each instruction fetch.”, I didn’t exactly agreed entirely with their take of the sentence but it was not wrong either.

The way I read this sentence differs from your perspective. When the OS is trying to execute a program, it creates a virtual address space to the process. We typically load into memory the executable instructions which should be located in the read-only code section of the address space. Important to distinguish executuable instructions and processes. Processes are “running” instances and executable is simply a file containing metadata and machine instructions. i.e. With one executable, you can have many instances (processes) of it running on the machine. For a process to advance to the next instruction, the CPU needs to fetch the next instruction from the process’s address space. Thus “Memory is accessed all the time when a program is running”.

In regards to caches, you are correct about instructions being cached. To expand a bit on this topic, on modern CPUs, for each level in the cache hierarchy (maybe not on L3 cache), there is a separation between data (d) and instruction cache (i) so you may see when profiling your code L1i, L2i, L1d, L2d. If instruction fetch works similarly to how data fetching works, the CPU will receive more than what it asks for from the main memory due to something called cacheline. There’s more to just instruction and data caching, on the CPU, there is a specific cache on the MMU (memory management unit) for caching virtual to physical address mapping (TLB - translation lookaside buffer) which we will see in later chapters. Exciting topics.

I no longer recalled what the conversation was about but the conversation included software in flight control systems and here was my comment out of context:

For flight control systems, what is important is reliability and predictability I believe and not raw performance. In such scenario, it would be more ideal to use a RTOS (realtime OS) where it’s runtime behavior is much more predictable.QNX and seL4, freeRTOS are such examples.

In regards to bound checking and code correctness, you might be interested in eBPF and seL4 microkernel. eBPF has gotten a lot of attraction in the Linux community as it provides a sandbox that runs on kernel space. ebpf programs provided in userspace are verified in the kernel for “safety” by the ebpf verifier before execution can happen which guarantees (at least in theory) no infinite loops, no dead code (i.e. all instructions must be reachable), and with some memory restriction guarantees.

seL4 is a microkernel RTOS that has been mathematically verified for correctness which is extremely rare and it’s written in C. I definitely need to read more on this OS but it’s been on my to read list: https://sel4.systems/About/whitepaper.html

I’ve mentioned the term microkernel a few times. Microkernel as the name implies removes a lot of the components that a typical (monolithic) OS is composed of such as the networking stack, file system, and drivers. OSTEP containing sections on filesystem is obviously in the perspective of a monolithic OS but most of the key concepts are the same anyways. Microkernel approach is interesting and it will be interesting to take a look into how different microkernels overcome or at least try to mitigate overhead costs due to message passing.

Someone did comment how flight control systems are concerned about context switches as it can lead to non-determinism of latency bounds which I do recall being a concern raised in the texts I read up on for RTOS.

Chapter 3: Processes

The chapter talked what a process is (i.e. a running program) and how a program is itself just a lifeless piece of data containing static instructions and static data. How time sharing of CPU allows us to have programs run concurrently by giving each program a tiny slice of the CPU and this is how it gives us the illusion that the CPU is runnign multiple programs parallely (concurrency and parallelism are two concepts that seem the same but are actually different). It also briefly mentions how programs are created and executed, what a stack pointer and program counter are, the various states a process can be (i.e. running, ready, blocked), what a zombie state is and etc.

In response to a question how one can turn to the OS to obtain better performance, I briefly mentioned the following that one could consider (I am no expert, I am just a Mathematics student):

As you alluded to, you can tune the OS to have system performance gains for your workloads. Let’s ignore RTOS and focus on what system performances you can do on Linux and why.

1. Schedulers: there exists different schedulers you can use to better address your workloads. For instance, Latency-criticality Aware Virtual Deadline (LAVD) scheduler was created from what I recall for gaming workloads on Linux (i.e. for the Steam Deck) to reduce stuttering during gameplay. While the deadline is meant for gaming, Meta seem to also adopted this scheduler for some of their servers as it suited their workloads well [^1].
2. CPU: Context switching is great to juggle between many different threads but it comes at the cost, mainly due to non-effective use of caches. You can pin threads (called thread affinity) to certain CPU cores. If you are aware of the hardware you are executing on, you can go even further and ensure threads sharing the same global data reside in the same die to share the same L3 cache (e.g. for AMD Zen 5 architecture, this would be cores in the same CCD or CCX in older architectures). This helps make better use of caches and reduce cache coherency contentions. Thread migration can be costly.
3. Tickless Kernel: You can configure the Linux kernel to be “tickless” to avoid the regular timer interupts which can improve CPU intensive workloads by reducing jitters. Here’s an excerpt where I first heard this from [^2]:

   The periodic interrupt has further negative effects. For instance, it pollutes the cache and TLB. As a fine- grained effect of jitter, it degrades performance of codes that rely on barriers between threads, such as frequently happens in OpenMP – The Art of HPC Volume 1 (1.7.2)

4. Large Page: as name implies you can change the size of the page which databases may employ. Great for memory-intensive applications if data tends to be clustered (spatial locality) from my understanding by reducing TLB misses

[^1] https://lwn.net/Articles/1051430/

[^2] https://github.com/VictorEijkhout/TheArtofHPC_pdfs/tree/main/vol1

Chapter 5: Process API

The chapter discussed various APIs when working with processes such as exec(), fork(), wait(), kill(), etc. It also talked about UNIX pipes and how a process is created.

What stood out for me was how the separation of the calls between fork() and exec() allows the shell to run some code in between. I have no clue why this piece of fact stood out to me as I have in the past took advantage of this feature a fair bit in the past. For instance, the prerequisite course to operating system (a 2nd year course) at my former university required us to implement a shell so we needed to implement redirection as described in the readings and I recall pipes worked in a similar way. I also made use of this to do some setup for a Linux Pluggable Authentication (PAM) I worked on years back.

I recall when I took OS, we made a lot of memes and jokes relating to kill such as a parent killing their child. kill is probably not the best name to encapsulate its abilities (as signals does more than terminating a process) and can inevitably make young students to spawn jokes about the name but it is hard to change names once it becomes adopted widely in the industry.

Chapter 9 & 10:: Multi-Processor Scheduling

After going through different scheduling policies that exist and their effects on turnaround time and response, this chapter presented difficulties getting our traditional schedulers originally designed for single core CPUs to work effectively on multi-processor systems that are now common in our everyday lives. Previously covered scheduling policies include FIFO, Shortest job first, shortest time to completion first, round robin, multi-level feedback queue, lottery scheduling, and Linux Completely Fair scheduler.

One of the major issues with multi-processor scheduling is that we now need to care about how to share data among multiple processors, cache coherency problems and correctness. This is my response to what others have discussed on the difficulties and complexity multi-processor systems introduced to us programmers:

As others stated, when working on multiple processors one does need to care about cache coherency and cache locality for performance. Pinning threads to CPU cores is one of the ways to mitigate thread migration (i.e. executing a thread in another core and thereby not making use of cache locality and could trigger cache coherency issues if the cache has been invalidated). This is called thread affinity which I mentioned in week 2. However, this could cause a load unbalance but one looser approach is to set the thread to be pinned in the same die that shares the same L3 cache. On AMD Zen 5 CPUs, cores residing in the same CCD share the same L3 cache. There are some CPUs that can have two CCD meaning you could have 6-8 cores packed together in a die that shares the same L3 cache so there will be performance more noticeable penalty if a thread were to migrate from one CCD to another. While pinning a thread to a set of CPU cores that share the same L3 cache will not take full advantage of L1 and L2 cache, it nonetheless helps alleviate the workload imbalance a bit and reduce cache coherency contentions. Though in enterprise or HPC world, I would imagine they would have to consider distributed topological layout on top of NUMA topology. So it likely becomes even more important to ensure data resides within some locality at the node level and then at the NUMA level (like the CPU L3 cache I mentioned earlier). I’m not exactly sure how well aware the Linux schedulers are of this type of this inter/intra toplology by default or if they rely on some scheduling software like SLURM (heard of it but never taken a close look at it yet). I know traditionally we scaled vertically but in the past decade with the rise of cloud computing, companies have been scaling horizontally as well. Since MPI exists, HPC probably did scale horizontally out of necessity and probably have specialised schedulers to handle this.

Chapter 12 & 13: Address Virtualisaiton

The chapter discussed on the importance and reason why address virtualisation is needed and the address space. Someone made a comment on the arbitrary convention of the direction of how the stack grows and this was my response:

While I am not sure about the literature convention of the direction of the stack and heap growth, in practice there is an architectural dependent aspect to the growth of the stack and heap if my memory serves me correctly. There are two properties to consider in regards to how the stack behaves:

• Direction: Ascending or descending
• Item the Stack Pointer is pointing to: full or empty

It’s been a while as I’ve read the chapters back in January, but I don’t recall them mentioning about full vs. empty stack pointers. If the stack is full, the stack pointer (SP) points to the topmost item in the stack (i.e. latest item in the stack). In the empty stack, the stack pointer will point to the next free location in the stack. Knowing how your architecture behaves can be important as one can imagine when writing an assembler or tracing through the code disassembly (or writing assembly). Combining the two properties, one can classify a stack into 4 types. Something interesting that I only became aware of sometime last year around this time of year when I was going through an assembly book.

Chapter 14: Memory API

While I do not recall the specifics of the chapter on what it was about, I do recall it talked about various memory apis (hence the title) such as what is mmap(), and how brk and sbrk relates to malloc() and free() to increase and decrease the program’s “cieling” of the heap (i.e. the location in which the heap ends). Someone asked if any of us has tried different memory allocators such as the jemalloc and mimalloc and hence why you will see the allocator mentioned in my response:

Somehow I missed the existence and mention of brk() and sbrk() in the textbook and in my various years of programming till I decided to take OS for fun again when I went back to school to study Mathematics. Fun fact, malloc() could instead use mmap instead of brk if the allocations surpasses a certain threshold (i.e. MMAP_THRESHOLD - take a look at the manpage for malloc or mallopt for more details).

jemalloc has been in my radar but I have never personally used it myself. It is pleasant to hear Meta’s renewed interest in picking up jemalloc again after initially dropping development last year.

While valgrind is a great tool to detect memory leaks, I find address sanitiser (ASAN) more convenient as I could quickly enable this on debug builds to catch memory-related issues with acceptable performance hit. The notes on how the OS will reclaim the memory on userspace reminds me how I tried to squash all the memory leaks for internal utilties we had at one of the places I interned at to only be told there was no point as the programs were short-lived userspace programs (like a few seconds at most). This was a good lesson because freeing more complicated data structures if done wrong could be more damaging and it also overcomplicates the program + unnecessarily increases the binary size in our limited storage. Of course it is good practice to always free memory but it may not be worth the risk + engineeirng hours to eliminate every single memory leak if its just for a user-space utility.

Rust’s design to be both memory safe and performant makes it an interesting contender to C/C++, something I definitely should take a closer look at (I’ve only briefly looked over the language to get a taste of it but nothing too deep). Seeing the growing interest to adopt Rust in the kernel, at least for drivers makes me think I should take another look at the language in the future. C supposedly has a memory-safe implementation via the compiler called Fil-C but its performance was lacking at the time I’ve read about it, something along the lines of being up to 4x slower than regular C.

Chapter 16: Segmentation

The chapter talks about segmentation being a solution to resolve internal fragmentation that is caused by having a SINGLE pair of base and bound pair in the MMU for then entire addres space which can cause a lot of wasted space since most of it may not be used. Thus, the idea is that you give each segment of the address space such as the stack, code, and heap their own base and bound pair in the MMU. This way you can ensure each segment takes the necessary amount of data. To keep track of the segments, it mentions how the hardware now needs to keep track of not only the base and size of each segment but also the direction of its growth and permissions (i.e. protection bit). This protection bit is what I focused on my conversation where I tried to manipulate a page that was considered read only using mprotect() and how mseal() fixes the issue (likely going to be reposted as a separate microblog and blog):

When reading about the protection bit, it reminded me that Linux introduced mseal() sometime in 2024 with glibc introducing the API in 2025. To give context, the protection bit is supposed to make different sections readable, writable or executable (in the page table). In theory, this would imply that the readonly section of the binary for instance (.rodata) should not be writable. If we were to run objdump, we see that is indeed the intention with the READONLY label

$ objdump -h ./rodata_write | grep -A1 .rodata
$ objdump -h ./rodata_write | grep -A1 .rodata
./rodata_write:     format de fichier elf64-x86-64

--
13 .rodata       00000088  0000000000401358  0000000000401358  00001358  2**3
                 CONTENTS, ALLOC, LOAD, READONLY, DATA

When I execute the program, I should expecting something like this in their mappings (/proc//maps)

00401000-00402000 r--p 00001000 00:2d 26970                              /tmp/rodata_write

where this region of memory is marked as read-only.

However, if I invoke mprotect() which allows us to change protection of the calling process’s memory map, we can in fact modify the rodata. If I define a constant string: “Hello”, we can see that it is stored in rodata section:

$ objdump -s -j .rodata ./rodata_write | grep -B2 Hello
Contenu de la section .rodata :
 401358 01000200 00000000 00000000 00000000  ................
 401368 48656c6c 6f000000 55736167 653a2025  Hello...Usage: %

Normally I would not be able to overwrite this section of memory as it does not have write permission but thanks to mprotect(), I could:

mprotect((void *)page_start, len, PROT_READ | PROT_WRITE)

As seen below:

$ ./rodata_write Bye
Before: Hello
After:  Bye

And if we were to check the mapping again, we’ll now see the same region of memory is now marked as write (w):

00401000-00402000 rw-p 00001000 00:2d 26970                              /tmp/rodata_write

But if I comment out the line containing mprotect(), we eventually will hit a segfault when trying to overwrite the const string residing in the .rodata section with “Bye”:

$ ./rodata_write Bye
Before: Hello
Erreur de segmentation     (core dumped)./rodata_write Bye

Something intteresting to see as prior to reading proposals to introduce mseal(), I did not know protection bits were modifiable by the user.

If you are interested in the reading group, you may be able to still sign for the next session soon, though I am not exactly clear on the details. But it definitely has been enjoyable to go through the textbook with a group, covering my favorite subject: OS.

Here are the following mirrors as of the time of writing (April 29 2026):

• Neocities: https://randombits.neocities.org/
• Codeberg: zakuarbor.codeberg.page also accessible via randombits.ca
• Github: https://zakuarbor.github.io/

The long-term goal is to rent a VPS to “self-host” the blog (at https://randombits.ca/) while maintaining the others as mirrors for readers who have their own platform preferences. Though neocities will remain my platform of choice for site discovery and community networking. Thus for the neocities community, nothing will change in the short and long term outlook of the site.

Watching Jeff Geerling’s video for the ARM mainboard made me think it would be a perfect project for hardware interns + newhires of IBM and Oracle respectively to task them to make a working prototype for the Framework 13. I definitely was not thinking of this as I passed IBM on my way to work. Though that campus has no relation with the IBM POWER and Telium design team if I recalled correctly. From what I recalled, IBM Canada was purely software work particularly in databases. data analytics, compilers, and other software services.

On another note, I recently found out that s390x and Telium chips run on Mainframes, not POWER, at least not any modern mainframes. POWER are used in servers. All these years I thought POWER was also used in mainframes … I told many folks the wrong information … oh well at least I now know.

Though to be realistic, I have no use for an ARM, RISCV, nor a POWER system so it would be more of an expensive collection rather than something I would use. Thus its probably better for my wallet if I abstain from my temptation to collect any decently powered alternatives to AMD64 (x86-64) systems. I never used my collection of Raspberry Pi for its architecture anyways. Also if I was to buy an alternative to AMD64, I would prefer it to be big endian by default. Many architectures have shifted away from big endian including POWER due to software compatbility reasons but I think that would be the reason I would consider buying one. Not that I have much money as a student …

On a random note, Framework released two new products that interests me, the first being the Framework OCuLink Dev Kit which allows one to connect a desktop GPU to the laptop, and the other being a Wireless Touchpad Keyboard. I definitely had a need both at home and at work to debug issues and having 2 in one solution would have been useful for debugging purposes. Though these are products I would not purchase with my student income obviously.

As a side note, the commentary presented below heavily resembles Elementary Analysis: The Theory of Calculus by Kenneth A. Ross. This is on purpose as Starling heavily based on the course notes based on this textbook.

edit: The author was not expecting the amount of time and motivation required to finish this billet (blog post in french). Thus they have decided to write only the follow up course for completion sakes but will likely not write anymore course commentaries. Sorry to anyone looking forward to reading the pedagogy of linear and abstract algebra as it also differs a lot from your regular linear algebra course. I encourage anyone interested to read Linear Algebra Done Right by Axler as he does an excellent job approaching linear algebra in a rigourous manner.

The approach to Calculus differs depending on one’s program of study. However, one thing remains true throughout them all, that calculus is the study of changes. However, one may be shocked in the differences of the content between the various discipline. There is calculus for engineers, calculus for science, calculus for business, and most important of all, Calculus for future Mathematicians. The author has a more holistic view of what is covered in an introductory course to calculus as they were a former student of Computer Science in its past life, a teaching assistant to Calculus for Engineers and also a student of Mathematics. Yes this does imply that the author has taken calculus twice, once at another university and another at the university it is attending of which the content will be discussed below.

Calculus for engineers at Carleton University is very rushed but does not skimp on the knowledge and techniques required for their discipline. In this sense, the author is amazed in the speed of which the freshman engineering student have to learn. While they may skimp on some minor details and omit certain topics such as taylor series, they cover two separate calculus course in just under 4 months, or more accurately within 3 months since the last month is exam season. In this, I applaud those who have succeeded in the course and truly drilled the various integration and differentiation techniques covered in the course. MATH1052 on the other hand will seem very foreign to even those who have taken freshman calculus as we will see why soon.

MATH1052 starts with defining all the axioms that exists in the field over the real numbers. A field essentially is a set of properties that the additive (+) and multiplicative (x) must respect over some set such as the reals denoted as $\mathbb{R}$. Your average joe will take all of the axioms for granted but as aspiring Mathematicians, one must never take things for granted or at least add it to its long list of things to look at in the (cough cough) near future.

Field: Any system of numbers satisfying all 9 axioms below is considered a field: $(\forall a,b,c\in\mathbb{R})$

1. a+(b+c) = (a+b)+c (ADDITIVE ASSOCIATIVITY LAW)
2. a+b = b+a (ADDITIVE COMMUTATIVITY)
3. $\exists 0\in\mathbb{R}$ such that $a + 0 = a$
4. $\forall a\in \mathbb{R}, \exists (-a)\in\mathbb{R}$ such that $a+(-a) = 0$
5. a(bc)=(ab)c
6. $a\cdot b = b \cdot a$
7. $\exists 1\in\mathbb{R}$ such that $a\cdot 1= a$
8. $\forall a\neq 0, \exists a^{-1}\in\mathbb{R}$ such that $a\cdot a^{-1}=1$
9. $a(b+c) = a\cdot b + a\cdot c$ (Distributivity Law)

Obviously the reals (the numbers us average joe are used to) is a field. The rationals ($\mathbb{Q}$) it turns out is also a field which may not come to a surprise but for some reason I had the inkling idea back in my time as a freshman computer science student over a decade ago (yes the author is old) that it wasn’t due to a tiny condition I have forgotten.

Firstly one needs to define what the natural numbers $\mathbb{N}$ are since this will be used in constructing the definition of the rationals $\mathbb{Q}$ later. Depending on your professor, one may have previously learned that $0\in\mathbb{N}$ but in our definition, we will omit 0 and hence we will informally define the natural numbers as:

$\mathbb{N} = {1, 2, 3, \cdots}$

As one can observe $\mathbb{N}$ is a subset of the integers $\mathbb{Z}$ denoted as $\mathbb{N} \subset \mathbb{Z}$ except all of its numbers are greater than 0. So it is the set of all positive whole numbers not including 0. With this established, let’s define the rationals:

$\mathbb{Q} = \{\frac{m}{n} | m\in\mathbb{Z}, n\in\mathbb{N}\}$

Now let’s see what little freshman CS idiot the author was a decade ago (and still is but let’s ignore that). It conjectured that the rationals cannot be a field as $0\in\mathbb{Q}$ but 0 has no multiplicative inverse. This is where knowing the little details matter and will be a reoccuring theme through one’s study in Mathematics. What the author has forgotten was that the axiom states that for all a not equal to 0, there exists an inverse in the set. If a set is a field, then we can say that the set is closed under addition and multiplication. Some math jargon for you. Though being closed under the two operations is not suffice to say it is a field.

One may question why there is even a need to codify whether a set is a field or not. It turns out that without one of these properties, the usual Mathematics we have come to learn and perhaps love will shatter. For instance, a vector space requires to be operating on scalars that are part of a field. Omitting a single property will render linear algebra as we know it to no longer apply as its theorems were built and studied under the assumption that the scalars were a field. Imagine if we designed computers not using boolean algebra but under some non-field set such as a set of 4 numbers under modular arithmetic ($\mathbb{Z}/4\mathbb{Z}$ - don’t worry if you don’t understand this). Chaos will ensue, at least from an engineering perspective as we would require a ton more circuitry and logic to ensure the math we operate on works. Imagine a calculator where dividing by certain numbers (not zero) becomes undefined, it will be a useless calculator without those costly patches in the hardware.

Aside: While the rationals are a field and hence we get both additive associativity and additive commutativity, you will be surprised that this is not the case in hardware. Floating point numbers are weird and special attention should be taken to minimize the error. Something I learned during my 2 year hiatus from my studies which you can read over here.

Based on the 9 axioms that make up a field, the following statements are true:

$\forall a,b,c\in\mathbb{R}$

1. $a+c = b+c \implies a = b$
2. $a\cdot 0 = 0$
3. (-a)b = -(ab)
4. (-a)(-b) = ab
5. ac=bc & $c\ne 0 \implies a = b$
6. $ab = 0 \implies a = 0 \lor b = 0$

Proving these properties are left as excercise to the readers. It is quite funny experience trying to prove Mathematical properties that you took for granted and often leads to banging your head on the wall questioning your decision to study Mathematics. While others may question their career choice, I quit my job to play around with numbers and symbols so I can’t say I was not expecting this.

My attempt to prove 1 * 0 = 0, hopefully it is correct ...

The next set of axioms pertain to ordering on the reals which will be helpful in proving some theorems that will follow next:

ORDER AXIOMS on $\mathbb{R}$:

1. Given $a,b\in\mathbb{R}$, either $a\leq b$ or $b \leq a$
2. If $a\leq b$ & $b\leq a$, then $a=b$
3. If $a \leq b$ & $b \leq c$, then $a \leq c$ (i.e. $\leq$ is transitive)
4. If $a \leq b$ & $c\in \mathbb{R}$, then $a+c \leq b+c$
5. If $a\leq b$ & $0\leq c$, then $ac \leq bc$

All the axioms from before are required to build up the set of properties I am about to present to you, 7 ordering properties that will be essential in your mathematical career. I cannot stress this enough, knowing these properties will help you solve proofs relating to inequalities in this course and in all future courses:

Theorem: properties of an ordered field

1. if $a \leq b$, then $-b \leq -a$
2. if $a \leq b$ & $c\leq 0$, then $bc \leq ac$
3. if $0 \leq a$ & $0 \leq b$, then $0\leq ab$
4. $0 \leq a^2$
5. $0 \lt 1$
6. if $0 < a$, then $0 < a^{-1}$
7. if $0 < a < b$ then $0 < b^{-1} < a^{-1}$ for $a,b,c\in\mathbb{R}$

Math is a subject that builds on top of itself so every concept you learn is built on top of other concepts (except for the axioms). How I view Mathematics (Mathematicians, please don’t kill me) is that each theorem and corollary is another tool to solve more difficult problems. But it is also fun to ponder about to see why a particular theorem is true and thus one should try proving one of these properties listed above at least once. I find the first and 7th property particularly useful when working on inequalities. Notice the 7th property explicitly adds a restriction that this property only holds in the reals, there’s a reason for this.

Suppose we have a field with the elements 0,1,2 under modular arithemtric ($F_3$):

Multiplication table for $F_3$= {0, 1, 2} under arithmetic mod 3

1 < 2 but in $F_3$, their inverse is itself so if proposition 7 was true then $0 < 1 < 2 \implies 0 < 2^{-1} < 1^{-1} \equiv 0 < 2 < 1$ which is a CONTRADICTION. Funny how I never realised that proposition 7 only holds true for the reals (and perhaps on any ordered field) when I took the course but now reviewing the course content years later made me realise this. Another example of how the little details matter in Mathematics.

The next theorem introduced relates to the absolute value operator, but first we must define what an absolute value is:

Absolute Value: If $a\in\mathbb{R}$ then the absolute value of $a$ is: \(|a| = \begin{cases} a & \text{, if } a\geq 0 \\ -a & \text{, if } a \lt 0 \\ \end{cases}\)

I have seen many students not registering in their brain this definition which ends up biting them later in the course whether it’s a proof question or solving some straightforward math question. Even I am guilty of this myself, forgetting that the following question has two solutions for instance:

This means $x$ can either be -2 or 2. The important thing to take away is that $\sqrt{x^2} = (x^2)^\frac{1}{2} \ne x$ but rather $|x|$. The issue with the exponent laws is that it deletes crucial metadata/information and assumes $x>0$ (i.e. that x is positive).

Theorem: $\forall a, b \in\mathbb{R}$,

1. $|a| \ge 0$
2. $|ab| = |a||b|$
3. $|a+b| \le |a| + |b|$ (Triangle Inequality)

All 3 properties of the theorem are important to know but the triangle inequality stands out the most. It is one of the three distinguishing axioms that defines whether or not a function $d(x,y)$ is a metric, something taught in MATH3001 (or in MATH2000 if you have Jaworski). The most important hint I can give about the proof of the triangle inequality is the following: $-|x| \le x \le |x| \forall x\in\mathbb{R}$.

Bounds

The content covered thus far has been constructing basic properties of mathematical operations involving addition, multiplication, ordering and absolute values. Well the next set of content will be on bounds, a central topic if you are interested in analysis. But first we need to go over some notations and definitions:

Let $S\subseteq \mathbb{R}$ be a nonempty set:

Max: a number $s_o$ is called a MAXIMUM of $S$ if it satsifies the following:

• $s_o\in S$
• $s\le s_o \forall s\in S$

Thus $s_o = \max S$

Min: a number $s_o$ is called a MINIMUM of $S$ if it satisfies the following:

• $s_o \in S$
• $s_o \le s \forall s\in S$

Thus $s_o = \min S$

One important note is that some sets may not have a min or a max such as $(a, \inf) = \{x\in\mathbb{R} | a \lt x \}$ does not have a max nor a min. However, every finite set in $\mathbb{R}$ has a max and a min. To prove that $(a, \inf)$ has no min, it is important to use the definitions above to show otherwise. For instance, I would begin by assuming a min $s_o$ exists and then proceed to specially craft a number such as the midpoint between $a$ and $s_o$ (i.e. $\frac{s_o+a}{2}$ and show that this is first in the set first before showing that it is in fact smaller than our supposed min $s_o$. The same idea can be applied to show a max does not exist in the set. Though I haven’t done any max/min proofs in a while so I could be wrong in the details but the main takeaway is that you need to use the definitions to disprove the existence of a max or min.

UPPER BOUND: If $M\in\mathbb{R}$ satisifies $s\le M \forall s\in S$, then $M$ is called an UPPER BOUND for S and we say $S$ is bounded above

LOWER BOUND: If $M\in\mathbb{R}$ satisifies $m\le s \forall s\in S$, then $M$ is called an LOWER BOUND for S and we say $S$ is bounded below

BOUNDED: If a set $S$ is bounded below and above, we say the set $S$ is BOUNDED

Example: While $(a, \infty)$ does not have minimum, $a$ will be the lower bound since all elements in the set has to be greater than $a$. The set has no upper bound and thus is not a bounded set.

However, the problem with our definitions of upper and lower bound is that there could be many upper and lower bounds for the set. For instance, the finite set with two elements $S = \{0, 1\}$ has infinitely many number of upper and lower bounds in $\mathbb{R}$. I could say 100, or even a 1000 is an upper bound to the set $S$ and -1000 is a lowerbound to S. There’s no utility to this statement. It’s as vague as telling someone “I’m thinking of a number greater than 0” and expecting them to identify it intelligently in one guess. There are infinitely many numbers one could select and serves no purpose aside being vague to ensure the player never had a decent chance to win the game from the start.

(though if you were to play this game, it would be devilish to choose a large interval whereby their chances of getting the answer is so small that there was no point of playing the game to begin with)

This is where our next set of definitions becomes useful in describing upper and lower bounds:

Let $S\subseteq \mathbb{R}$ be nonempty:

• If $S$ is bounded above and has a least upperbound, the least upperbound is called the supremum of $S$ denoted as $\sup S$
• If $S$ is bounded below and has a greatest lowerbound, the greatest lowerbound is called the infimum of $S$ denoted as $\inf S$

More formally,

SUPREMUM: $M = \sup S$ if the following two conditions holds true:

• $\forall x\in S, M\ge x$ (i.e. $M$ is an upperbound of $S$)
• If $N$ is some other upperbound of $S$, then $M\le N$

INFIMUM: $m = \inf S$ if the following two conditions holds true:

• $\forall x\in S, m\le x$ (i.e. $m$ is a lowerbound of $S$)
• If $n$ is some other lowerbound of $S$, then $n \le m$

Though sometimes it is good to replace the 2nd condition of both supremum and infimum with the contrapositive to show that if a potential candidate is smaller/greater than the supremum and infimum, it is actually not a upper/lower bound respectively. For instance, for supremum, we could rewrite the second statement as:

If $N < M$, then $\exists s\in S$ such that $N < s$ (i.e. N is actually not an upperbound)

The reason why this definition is more valuable is that the original statement was indicating that the supremum is the smallest upperbound out of all the upperbounds to the set. This can be extremely difficult or impossible to prove and hence we take the contrapositive. This reduces the problem to just proving that any specific candidate $N < M$ would in fact not be an upperbound which is a much more manageable problem.

Question: Does the supremum need to exist in the set $S$?

Answer: No it does not. Unlike the maximum of a set which must lie in the set itself, there is no restriction in our definition that requires the supremum (nor infimum) to be itself a member of the set.

Example: $S = \{ x\in\mathbb{R} | x < 2\}$ has a supremum $\sup S = 2$ where $2\notin S$

But is this always the case? What property do we need such that any set $S$ has a supremum? And where where would the supremum lie in? For instance, what common number sets/space would be sufficient to encapsulate this.

The naturals and integers obvious bad choices to gaurantee a supremum due to its inability to represent fractions. But the rationals, our set of fractions $\mathbb{Q}$ can infinitely get close to any numbers in $\mathbb{R}$. In fact, between any two rationals, there’s always another rational. Furthermore, there are infinitely numhber of rationals between two rationals (this is refered to as the Density of $\mathbb{Q}$. Surely this would be a good candidate. However, note the wording: $\mathbb{Q}$ can infinitely get close to any numbers, this does not meant it can represent the number itself despite how dense $\mathbb{Q}$.

Like how there is an opposite force for everything, there exists irrational which are equally as dense i.e. between two real numbers, there exists an irrational number. These “gaps” in $\mathbb{Q}$ explain why $\mathbb{Q}$ is not sufficient for our question.

Here’s a simple classical counterexample to this:

$ S = \{ x\in\mathbb{R} | x \le \sqrt{2} \} $

The $\sup S = \sqrt{2}$ but the irrational number $\sqrt{2}\notin \mathbb{Q}$ and thus the set has no supremum over $\mathbb{Q}$ which Mathematicians term to be not complete.

The COMPLETENESS AXIOM: Every non-empty set of real numbers which is bounded above has a least upperbound (supremum) which is an element of $\mathbb{R}$

i.e. If $S\subseteq \mathbb{R}$ is bounded above, $\sup S$ exists and is a real number

The $\mathbb{R}$ does not have such gaps as $\mathbb{Q}\subset\mathbb{R}$ and similarly with the irrationals $\mathbb{I}\subset\mathbb{R}$. In fact $\mathbb{R} = \mathbb{Q} \cup \mathbb{I}$.

As a corollary, the same statement can be made about the infimum:

Corollary of the Completeness Axiom: If $S\subseteq \mathbb{R}$ is bounded below, then $\inf S$ exists and $\inf S\in\mathbb{R}$

As a consequence of the completeness axiom (actually a lot of things derive or require the complete axiom to be true), we can claim no matter how big a number $b$ is, one can choose any positive number $a$ (even an incredibly small one) such that it can be added to itself enough times to dwarf b. This is what Mathematicians call the Archimedean Property:

Archimedean Property: If $a > 0$ and $b > 0$ then $\exists n\in\mathbb{N}$ such that $na > b$

The proof is left as an exercise to the readers (or attend class) but one can use proof by contradiction by supposing that the property itself fails implying $b$ itself is an upperbound of $S = \{ na | n\in \mathbb{N} \}$ and thus by completeness, $M = \sup S$ exists. THen you could reasonably say $M-a < M$ then proceed to show that there must exist some $n\in\mathbb{N}$ such that $na > M-a$ and “magically” show that in fact that M is less than an element in the set S so M cannot be an upperbound of S. That’s the sketch of the proof, you can fill in the blanks yourself.

As a consequences of the archimedean property, we have:

1. If $a > 0, \exists n\in\mathbb{N}$ such that $\frac{1}{n} < a$ (i.e. there will always be someone smaller than yourself or in mathematical terms, there is always a fraction smaller than your number)
2. If $b > 0, \exists n\in\mathbb{N}$ such that $b < n$ (i.e. equally there will always be someone greater than yourself)

The archimedean property and its consequences are beautiful concepts that we can take to heart. Firstly that no matter how bad you are, there’s always someone will do worse than you and inveresly, no matter how much better you think you are, there’s always someone better than you. Share one of the two consequences to a friend depending on the context. If a classmate or collegue is being obnoxious about how much better they are than you, give them the second consequence. If someone is depressed about their outlook in life, share them the first consequence, though I don’t know how effective that would. Thank goodness I am not a counsellor.

Sequences

A sequence as you already know is an infinite list of numbers but more formally:

Sequences: a sequence is a function from the set of integers larger than some $m\in\mathbb{Z}$ into $\mathbb{R}$:

$S:\{n\in\mathbb{Z} | n \ge m \} \to \mathbb{R}$

$S(n) \to S_n$

Example: Take the sequence that alternates between -1 and 1: $(-1, 1, -1, 1, -1, 1, \ldots )$

We can define an element in the sequence as $a_n = (-1)^n \forall n\in\mathbb{N}$. To represent the sequence: $(a_n)^\infty_{n=1}$.

As this is a course in Calculus, limits are at the heart of everything we do in this course.

Limit of a Sequence: A sequence $(s_n)$ of real numbers converges to a real number $L $ if $ \forall \epsilon > 0, \exists N\in\mathbb{N}$ such that $n > N \implies |s_n-L| < \epsilon$

• if $(s_n)$ converges to L, $\lim_{n\to\infty}s_n = L$ or $s_n \to L$ and we call the number $L$ the limit of $(s_n)$
• if a sequence does not converge then we say the sequence $s(n)$ diverges
• Note: a sequence cannot have more than one limit. It either has a limit or does not.

In a more condensed form:

$\lim\limits_{n\to\infty}s_n = L \iff \forall \epsilon > 0, \exists N$ such that $n > N \implies |s_n - L| < \epsilon$

The formal definition of limits of a sequence is a mouthful but we can informally translate this as:

for any positive number $\epsilon$, no matter how small it is, we can find a number $N$ large enough such that whenever $n > N, s_n$ is a distance less than $\epsilon$ from our limit $L$

Note: $|s_n - L|$ represents the distance between $s_n$ and $L$

It is important that you understand this definition conceptually because this is one of many types of proofs you will see in this course. At the same time, do not panic if you struggle to use this formal definitions to prove a limit because the definition of limits for sequences is quite similar with the limits of a function and what it means for a function to be continuous and even uniformly continuous. While there are nuances between sequences, functions, and uniform continuity, the core idea is the same: for any $\epsilon > 0$, everything eventually stays within an $\epsilon$-window of the limit. Once you master the proofs to one of the 3 or more limit definitions covered in the course, you’ll master all of them.

A visual view of the epsilon window where the sequence converges to L after $n>N$

In these types of proof, the positive number $\epsilon$ must be arbitrary (i.e. not fixed). Thus your task as the prover is to find an $N$ which makes any term of the sequence after $N$ to be within this epsilon window. A fixed $N$ that works for one $\epsilon$ will generally fail for a smaller one — your $N$ must be a function of $\epsilon$.

Here’s a simple rough example to give you an idea: Prove that $\lim\limits_{n\to\infty}\frac{1}{n}=0$

It is often not evident what $N$ to choose, so we start with a rough work and later rewrite our proof more formally. In rough work, we assume the conclusion and work backwards to discover what $N$ should be. Then in the formal proof, we revevrse the direction by defining what $N$ is first and then show the implication holds forward.

Rough Work: Let $\epsilon > 0$ be given. We want to find $N$ such that $n > N\implies |\frac{1}{n}-0|<\epsilon$. Assuming the conclusion hold true, we have:

For the above statement to occur:

Thus we will choose $N=\frac{1}{\epsilon}$. I will leave the formal proof as an exercise to you the readers, most of the work has already been done.

As previously, mentioned, a sequence cannot have more than one limit. It either has a limit or does not.

Theorem: Let $s(n)$ be a sequence of real numbers. Then $\lim\limits_{n\to\infty}s_n=$ and $\lim\limits_{n\to\infty}s_n=t \implies s = t$

Originally, I wanted to leave the proof as an exercise but the proof involves a technique extremely important in Mathematics that I could not pass it.

Proof: Suppose $\lim\limits_{n\to\infty}s_n = s$ and $\lim\limits_{n\to\infty}s_n=t$ Let $\epsilon > 0$ then $\frac{\epsilon}{2}$ is also greater than 0 as well. We apply the definition of convergence to $\frac{\epsilon}{2}$.

We can have our $\epsilon$ window to be $\frac{\epsilon}{2}$ because $\epsilon$ is arbitrary and therefore it can be as arbitrarily small as we want it to be. In simpler terms, we just renamed $\epsilon$ as $\frac{\epsilon}{2}$

Then $\exists N_1$ such that $|s_n-s|<\frac{\epsilon}{2}, \forall n > N_1$ (*)

Then $\exists N_2$ such that $|s_n-t|<\frac{\epsilon}{2}, \forall n > N_2$ (**)

Let $N=\max{N_1, N_2}$, this is a technique often employed when there are multiple numbers to consider in these limit types proof.

If we want to prove that s = t, that is equivalent as saying its difference, or rather distance is 0: $|s - t| = 0$

However, we have no direct way to relate s and t to each other. The only information we have thus far is that they are both limits to the same sequence $(s_n)$. So the question is how can we relate s and t in a way that involves the sequence. This bridge it turns out is our magical number 0. This is a powerful technique whereby adding or subtract 0 has zero impact in the equation yet has profound effect to our consciousness.

The effect of introducing an intermediate term that cancels out transforms a once stagnant problem into something that gives hope to us that we are one step closer to our objective. What is great about rigorous introductory courses is how well-structured the course is, building up concepts from the ground up. We have started with fields, ordering and eventually to the triangle inequality. Currently, we can see some semblance of our limits from (*) and (**) but not quite exactly in the form we desire. This is where a student should stop and reflect what has been covered in the course thus far and try to pull out tools and tricks to manipulate the problem to our advantage and the triangle inequality is one of such tools.

Recall the triangle inequality tells us that $|a + b| \le |a| + |b|$, so if we take $a = s-s_n$ and $b = s_n - t$, then we can invoke the triangle inequality as follows (I like to imagine if magic was real, it would be very scientific and mathematical in nature):

If we are trying to prove that the limit $s, t$ of the same sequence are equal to each other, we must have this inequality be less than some $\epsilon$-window. What may not be obvious at first glance is that $|s - s_n|$ and $|s_n - s|$ are the same thing. Taking the absolute difference of two numbers measures distance and distance are symmetric (one of the three properties of a metric) but for those who have a hard time seeing this:

Therefore, we now have the following:

If $n>N$, both (*) and (**) are true, then for $n > N$, we now have:

Since $|s - t| < \epsilon$ for all positive number $\epsilon$, we must have $|s - t| = 0 \implies s = t$ as needed.

If a sequence does not have a limit, how do we show the sequence does not converge? It turns out we must find at least one value of $\epsilon$ for which the definition fails.

Example: Show that the sequence $(s_n)$ does not converge where $s_n = (-1)^n$.

This an alternating sequence between -1 and 1. Let’s pick $\epsilon=1$ and if that doesn’t work we’ll revise it later. Notice that the distance between $|1 - (-1)| = 2$ thus we need to choose an epsilon window $2\epsilon \le 2$ to guarantee that 1 and -1 cannot both fit inside this window simultaneously. That is the intuition behind choosing $\epsilon=1$.

The goal is to for the $\epsilon$ I pick, I cannot find an $N$ such that $|s_n - L| < \epsilon=1 \forall n > N$.

Suppose by contradiction that $\lim\limits_{n\to\infty}s_n = L, L\in\mathbb{R}$. Since $\epsilon = 1 > 0, \exists N$ such that $n > N \implies |s_n - L| < 1$. Thus we have:

We can break the sequence into two cases:

• n is even: $|1 - L| < 1$
• n is odd: $|-1 - L| = |-1(1+L)| = |1 + L| < 1$

We know that the distance of two neighboring terms in the sequence has a distance of 2 and thus:

Notice a pattern? Earlier we showed how introducing an intermediate term that cancels each other out had a profound impact to manipulate our equation to something closer to our desired outcome. Then we had invoke the triangle inequality, the same can be repeated again in this question:

Now we have the two even-odd cases of our sequence that can be substituted into the equation: \(|(1-L) + (1+L)| \le |1-L| + |1 + L| < 1 + 1 = 2 \nonumber\)

Thus far we have shown $2 \le |1 - L| + |1 + L| < 2$ which is a contradiction. Therefore, no such $L\in\mathbb{R}$ exists and $(-1)^n$ does not converge (i.e. it diverges)

Similarly how we showed a set can be bounded, we can equivalently show a sequence can be bounded as well.

Note The difference between a set and a sequence are the following:

• sets containly only unique elements meanwhile a sequence can repeat the same number many times
• sets are not ordered but the position of each element in the sequence matters
• sets are finite meanwhile sequences have infinitely many elements

Bounded Sequence: A sequence $(s_n)_{n=m}^\infty$ is called BOUNDED if the set $\{s_n | n \ge m \}$ is bounded

Equivalently, $(s_n)^\infty_{n=m}$ is bounded $\iff \exists M > 0$ such that $|s_n| \le M \forall n$

Here are some noteworthy theorems of sequences that I think are interesting to remember, albeit perhaps a bit informally. You have seen these be called the limit laws elsewhere. The proofs are left as an exercise (or attend the class or read the textbook).

Theorem: $\lim\limits_{n\to\infty} k s_n = k \lim\limits_{n\to\infty} s_n$ for some $k\in\mathbb{R}$

Theorem: $\lim\limits_{n\to\infty} (s_n + t_n) = \lim\limits_{n\to\infty}s_n + \lim\limits_{n\to\infty}t_n$ provided both converges

Theorem: $\lim\limits_{n\to\infty}s_nt_n = (\lim\limits_{n\to\infty}s_n)(\lim\limits_{n\to\infty}t_n)$ provided both converges

Theorem: $\lim\limits_{n\to\infty}\frac{s_n}{t_n} = \frac{\lim\limits_{n\to\infty}s_n}{\lim\limits_{n\to\infty}t_n}$ as long as $\lim\limits_{n\to\infty}t_n \ne 0$

Theorem: $\lim\limits_{n\to\infty}s_n^{-1} = \frac{1}{\lim\limits_{n\to\infty}s_n}$ as long as $\lim\limits_{n\to\infty}s_n \ne 0$ and converges

What the limit laws permit us is to determine the limit of a sequence very quickly. The $\epsilon-N$ method is essential for proving results from first principles and is often much difficult method whereas the limit laws let us compute limits quickly provided convergence is established (but its often assumed in non-rigourous courses). Here are some limits whose proofs will be left as an exercise:

• $\lim\limits_{n\to\infty}\frac{1}{n^p} = 0$ for $p > 0$
• $\lim\limits_{n\to\infty}a^n = 0$ if $|a| < 1$
• $\lim\limits_{n\to\infty}n^\frac{1}{n}=1$
• $\lim\limits_{n\to\infty}a^\frac{1}{n} = 1 \quad \forall a > 0$

To prove one of the limits listed above, an important concept called the squeeze theorem needs to be introduced.

Squeeze Theorem: if $s_n \le t_n \le r_n$ and $s_n \to L, r_n \to L$ then $t_n \to L$

The squeeze theorem (also called the sandwich rule) is literally what it sounds like, if a sequence is bounded below and above by two different sequences and the two different sequences converge to the same number then by effect, the sequence we are working on also converges to the same number.

A common theme you will find in calculus and in analysis in general is that things can have infinite limits i.e. is unbounded. For instance, the sequence $(n^2)$ is unbounded and thus does not converge (i.e. diverges to infinity). An infinite limit is another way of saying it diverges to infinity or negative infinity.

Infinite Limits: For a sequence $(s_n)$, we say $\lim\limits_{n\to\infty}s_n = \infty$ if for every $M > 0, \exists N $ such that $n > N\implies s_n > M$ (i.e. diverges to \infty$

In plain language, no matter what $M$ you pick, you can always find a bigger number in the sequence

Of course there is the negative infinity version of this definition and yes I know it sounds ridiculous at first to prove something does not converge. Previously, we have stated that to prove something does not diverge, it suffices to find one single $\epsilon$ such that the epsilon window cannot be true. It turns out that there are sequences, series and functions whose convergence is still a mystery such as the flint hill series:

Kurisu from Steins;Gate challenging the audience to solve a series whose convergence is not yet known by the Mathematical community. Source: reddit

Non-convergence does not imply a sequence has no limits. It simply means that the sequence does not converge to a single real number.

Existence of a Limit: A sequence $(s_n)$ has a limit (i.e. a limit exists) if either:

• $s_n$ converges
• $s_n$ diverges to $+\infty$
• $s_n$ diverges to $-\infty$

Let’s now look at an example of proving $(n^2)$ has a limit even if it cannot be bounded:

Example: Show $\lim\limits_{n\to \infty}n^2 = \infty$

Sketch/Rough Work: Given $M > 0$, we need to find $N$ such that for $n > N$, we have:

Proof: Let $M > 0$ and choose $N=\sqrt{M}$.

Then for $n > N \implies n > N = \sqrt{M}$ and as $n, M > 0$:

$n^2 > M$ as required.

Now that we established the existence of infinite limits, we now have the following limit law as a consequence:

Theorem: Let $(s_n)$ be a sequence of positive numbers. Then $\lim\limits_{n\to\infty}s_n = \infty \iff \lim\limits_{n\to\infty}\frac{1}{s_n} = 0$

Before we proceed to the next topic, it is important to note that $\pm\infty$ are not real numbers and thus the limit laws that we became familiar with do not apply. In fact, we will soon learn that we call these indeterminate forms (e.g. $\infty - \infty, \frac{\infty}{\infty}, and 0\cdot\infty$). If you are fortunate enough, you will learn that not all infinities are treated equal. There are different classes of infinities and some are larger than the others.

Let us now proceed to finding tools or other methods to show a given sequences converges without knowing the limit in advance. Often times even with the formal definition, we discover the limits upon trying to prove the statement. But surely, there is a better way and yes there is but it only occurs for a certain subclass of sequences.

Before we learn this magical spell, we first need to define what it means for a sequence to be increasing and decreasing.

A sequence $(s_n)$ is said to be:

• INCREASING: if $s_n \le s_{n+1} \forall n$
• DECREASING: if $s_n \geq s_{n+1} \forall n$

Notice that the definitions does not imply the sequence is strictly increase nor strictly decreasing via the usage of $\le$ and $ge$ in lieu of $\lt$ and $gt$ in their definition. This implies a straight constant line is both always increasing and always decreasing. Weird right?

If a sequence is always increasing or always decreasing, it is called monotonic (i.e. can only go in one direction). Restricting a sequence to be monotonic AND bounded leads us to the following theorem:

Theorem: Any bounded monotone sequence converges

Let’s consider why the theorem requires boundness and that the sequence is monotonic:

• without boundness, the sequence can diverge to $\pm\infty$ and thus does not converge
• without montonic property, we could have a bounded sequence that jumps/alternates between two numbers and thus never truly converges to a single real number (i.e. $((-1)^n)$ alternates between -1 and 1

Here’s a question that I myself got wrong when reviewing the material:

Question: Suppose we have a sequence where the distance between each subsequent neighbor gets closer and closer to each other. Does this sequence converge? Answer: No it does not, the two terms could be inching towards each other so slowly that convergence is not guaranteed

Thus, this brings us to the next type of sequences: cauchy sequences

Cauchy Sequences: a sequence $(s_n)$ is called a CAUCHY sequence if $\forall \epsilon > 0, \exists N$ such that for all $m,n > N \implies |s_n - s_m| < \epsilon$

What Cauchy sequence is telling us is that as $m, n$ get large, $s_n, s_m$ get closer and closer to each other. An important note is that $s_n$ and $s_m$ do not have to be neighbors. It’s as if there’s a clustering whereby the terms are getting closer and closer to each other, where a bunch of terms are being compressed more and more to each other.

This increasing clutering of points allows us to make the following statement:

Theorem: A sequence is convergent $\iff$ it is a cauchy sequence

We now have two tools (theorems) we can use to determine whether or not a sequence converges or not without finding its limit. This implies we can use our beloved limit laws without explicitly proving convergence.

Previously, we stated that every bounded monotone sequence converges. We also discovered that without monotonic property, convergence is no longer a guarantee. But what if we look at its subsequence?

For $s_n = (-1)^n$, the sequence $(s_n)$ does not converge as it alternates between -1 and 1. But if we were to filter every 1st or 2nd term, we obtain a new subsequence of a list of -1 or 1 respectively and those converge. But before we explore the implication of convergence i.e. does every subsequence converge? we first need to define what a subsequence is:

Subsequence: let $(s_n)$ be a sequence. Then a sequence $(t_k)_{k\in\mathbb{N}}$ is called a subsequence of $(s_n)$ if for each $k\in\mathbb{N}, \exists n_k$ such that $n_1\lt n_2\lt n_3\lt\cdots$ and $t_k=s_{n_k}$

Since we can chose any terms in order when creating a subsequence, it should come as to no surprise of the following:

Theorem: Every sequence has a monotone subsequence

Since any sequence has a monotone subsequence, what can we say about convergence? It turns out that:

Bolzano-Weierstrass Theorem: every bounded sequence has a convergent subsequence

Recall how the monotonic convergence theorem required the sequence to be both monotonic and bounded? Since every sequence has a montonic subsequence and if we create a monotonic subsequence from a bounded sequence, then it surely comes no surprise that the subsequence is also bounded. Since the subsequence is monotone and bounded, the monotone convergence theorem guarantees it converges.

Bolzano-Weierstrass theorem implies the following:

Theorem: Cauchy sequences are bounded

Though I think that was already obvious, if all cauchy sequences are convergent to a single value, there’s no way it is not bounded either. Of course, I will like to stress, being bounded does not imply convergence at all.

What may not be obvious is whether the subsequence of a convergent sequence converges to the same limit (at least it is not obvious till you are reminded what it means to converge to a limit $L$). The answer is yes (actually in hindsight this should have been obvious but my little brain didn’t think it was):

Theorem: If $(s_n)$ converges to $L$ then every subsequence converges to $L$ as well

As I alluded to previously, we will be also studying the convergence of series, specifically of infinite series but first we need to define what partial sums are:

Partial Sums: $s_n = \sum\limits_{k=m}^n a_k = a_m + a_{m+1} + \ldots + a_n$

Partial sums is simply a finite sum of numbers in a series.

An infinite series is said to converge if the sequence of partial sums $(s_n)$ converges to a real number: $\sum^\infty_{n=m} a_n = s \iff \lim_{n\to\infty}s_n = s \iff \lim\limits_{n\to\infty}(\sum\limits_{k=m}^n a_k) = s$

In simpler terms, if the sequences of partial sum converges then the infinite sum converges as well.

Some Common Series:

• Geometric Series: $\sum\limits_{n=0}^\infty r^n = \frac{1}{1-r}$ where $|r| \lt 1$
• Harmonic Series: $\sum\frac{1}{n}$ diverges
• P-Series: $\sum\frac{1}{n^p}$ converges if $p > 1$ and diverges if $p \le 1$

The harmonic series is an interesting case because intuitively, one would imagine it converges. Recall that any bounded monotone sequence converges. When we look at the sequence $s_n = \frac{1}{n}$ is indeed bounded between 0 and 1 and is monotonically decreasing. In fact it is intuitively obvious that $\lim\limits_{n\to\infty}\frac{1}{n} = 0$. While the sequence converges, the (harmonic) series does not. We will later show why the harmonic series diverge soon with the comparison test though there are many ways to show that the series diverge (e.g. the integral test taught in MATH2052).

Unlike with sequences where one could skip the first $N-1$ terms, series requires us to aggregate every single term in the series. Thus while one may be tempted to adopt intuition from sequences to series. This does not work. Perhaps it is just me, but my initial intuition was nudging me that since the terms were approaching to 0 at a relatively decent pace, the series could be bounded above. However, if one was to write the first few terms, they’ll quickly realise this cannot be the case as we can always find a grouping of terms that sum greater than 1/2:

The main issue is that $\frac{1}{n}$ grows much faster than desired for convergence and hence why the p-series requires that the power $p$ is greater than 1.

Comparison Test: Let $\sum a_n$ be a series with $a_n\ge0\forall n$

• if $\sum a_n$ converges and $|b_n| \le a_n \forall n$, then $\sum b_n$ converges (i.e. if the car ahead of you stops, you as well will stop)
• If $\sum a_n = \infty$ and $b_n \geq a_n \forall n$, then $\sum b_n = \infty$ (i.e. suppose you are in a one way road and the car behind you is always tailing you without slowing down, you also cannot slow down either)

I no longer recall who told me the car analogy, but it’s one simple way to remember the comparison test.

The issue with comparison test is that it requires us to already know good candidates to consider that converges or diverges already. Though we already know 3 types of series that can either converge or diverge making our lives much easier. Here’s a simple example in action:

Example: $\sum\limits_{n=1}^\infty \frac{1}{n^2+1}$. Does it converge or diverge

By p-series we know that $\sum \frac{1}{n^2}$ converges since $p > 1$. Thus we have:

Thus by comparison test $\frac{1}{n^2 + 1}$ also converges.

Absolute Convergence: a series $\sum a_n$ converges absolutely if $\sum |a_n|$ converges

• if $\sum |a_n|$ diveres but $\sum a_n$ converges, we say $\sum a_n$ converges conditionally

By implication, it is obvious that absolute convergence implies convergent but it does NOT apply in the reverse direction.

convergent $\nRightarrow$ (DOES NOT imply) absolute convergence

Proof: $\sum\frac{(-1)^n}{n}$ converges conditionally but $\sum |\frac{(-1)^n}{n}| = \sum\frac{1}{n}$ (a harmonic series) does not diverge by p-test

Ratio Test: let $\sum a_n$ be a series of non-zero terms for which $\lim\limits_{n\to\infty}|\frac{a_{n+1}}{a_n}| = L$ exists. Then:

1. $\sum a_n$ converges absolutely if $L < 1$
2. $\sum a_n$ diverges if $L \gt 1$
3. $\sum a_n$ may either converge or diverge if $L = 1$

As the name implies, the ratio test compares what I view as the eventual growth behaviours of consecutive terms in the series. If the growth is less than 1, it indicates a shrinking behavior where the next term in the series does not dominate the current term. However, when the ratio is greater than 1, then the subsequent term dominates its previous term and hence will not converge i.e. it exhibits a growth pattern. Both growth patterns behave similarly with the geometric series. Take caution that when the limit $L = 1$, it tells us nothing about the series.

Root Test: let $a_n$ be a series for which $L = \lim_{n\to\infty}|a_n|^\frac{1}{n}$ exists. Then:

1. if $L \lt 1$, $\sum a_n$ converges absolutely
2. if $L > 1$, $\sum a_n$ diverges
3. The test fails if $L=1$

One of the beauty of the root and ratio tests is that their conditions behave the same, eliminating the need to remember an extra set of cases.

Alternating Series Test: if $(a_n)$ is a DECREASING sequence of non-negative numbers and $\lim\limits_{n\to\infty} a_n = 0$ then the alternating series $\sum(-1)^{n+1}a_n$ converges

The alternating series test is a bit of a unique case whereby there are two conditions to satisfy before the test can be applied.

Here’s a list of tests in order that one should try when solving the problem of convergence of series:

General Rule of Series Test:

Growth Hierarchy: $n^n \gt n! \gt a^n \gt$ polynomials

$n^n$	root
$n!$	ratio
$2^n$	root
$\frac{p(n)}{q(n)}$	comparison
$r^n$	geometric
$\frac{1}{n^p}$	p-series
$\frac{a^n}{b^n}$	geometric

What does it mean for a function to be continuous? One simple answer is to say that one should be able to draw the curve without ever lifting the pencil. A more sophisticated answer is to say a function is continuous when the function does not contain any holes, jumps or vertical asymptotes. That is to say,

Continuity: A function $f(x)$ is said to be continuous at $x=a$ if the follow conditions are all true:

1. $f(a)$ is defined
2. $\lim_\limits{x\to a} f(x)$ exists (i.e. $\lim\limits_{x\to a^-} f(x) = \lim\limits_{x\to a^+} f(x)$)
3. $\lim\limits_{x\to a} f(x) = f(a)$

However, the definition of limits above is not rigorous enough for us Mathematicians (or wanna-be). Recall that the definition of convergence of a sequence using limits involved an epsilon window and the work involved to determine an index N at which point we guaranteed all subsequent terms are within the $\epsilon$-window? We will be introducing a similar rigorous definition to define what it means for a function $f$ to approach $f(a)$. For functions, instead of $n\to\infty$, we have $x\to a$ and instead of asking whether the distance between the limit $L$ and $s_n$ fits within our $\epsilon$-window, we now ask whether the distance between $f(x)$ and $f(a)$ is infinitely close such that it fits within the $\epsilon$-window. Unlike the sequence which is one dimensional, functions are multi-dimensionals (2-D in our case) requiring an extra variable $\delta$ to specify how close $x$ has to be with $a$ for the distance between function at $a$ and $x$ fit within the $\epsilon$-window.

$\epsilon-\delta$: let $f$ be a real-valued function with domain of f $\subseteq \mathbb{R}$ be the limit of some sequence in domain of f and let $L\in\mathbb{R}$

Then $\lim\limits_{x\to \color{tomato}{a}} f(x) = \color{cornflowerblue}{L}$ means $\forall \epsilon > 0, \exists \delta > 0$ such that $0 < | x - \color{tomato}{a} | < \delta$ and $x\in dom(f) \implies |f(x) - \color{cornflowerblue}{L}| < \epsilon$

The blue band represents the $\epsilon$-window around $f(a)$ on the y-axis. The red band represents the $\delta$-window around $a$ on the $x$-axis. Any $x$ within the $\delta$-window maps to $f(x)$ within the $\epsilon$-window.

Example: Let $f(x) = \frac{x^2-1}{x-1}$. Prove $\lim\limits_{x\to\color{tomato}{1}}f(x) = \color{cornflowerblue}{2}$. Note that $dom(f) = {x\in\mathbb{R} : x \ne 1}$

First observe that this function can be simplified:

Rough Work: Given $\epsilon \gt 0$, we want to find $\delta > 0$ such that

If $x \ne 1$, then $f(x) = x + 1$:

Recall that we are trying to find the $f(x)$ as $x\to \color{tomato}{1}$ so we can say the following:

So take $\delta = \epsilon$

Proof: Let $\epsilon \gt 0$. Choose $\delta = \epsilon$. Let $x \ne 1 \implies f(x) = x + 1$.

Then for $0 \lt | x - 1 | \lt \delta \implies |f(x) - 2| = |x + 1 - 2 | = |z - 1| \lt \delta = \epsilon$ as required

We’ve thus far discussed sequences in respect to series, so let’s also relate sequences to functions, more specifically, to continuity:

Theorem: let $f$ be a real-valued function with $dom(f)\subseteq\mathbb{R}$ and let $a\in dom(f)$. Then

$f$ is continuous at $a \iff $(for every sequence $(x_n)$ in dom($f$) converging to $a$, we have $\lim\limits_{n\to\infty} f(x_n) = f(a)$)

Though the way I would short-hand this theorem is the following:

Theorem: f is continuous at $a \iff$ for $(x_n) \to a$, we have $\lim\limits_{n\to\infty} f(x_n) = f(\lim\limits_{n\to\infty} x_n) = f(a)$

Let’s work on a problem to illustrate the power of this new theorem by comparing between the classical $\delta-\epsilon$ proof and the theorem:

Example: Prove that $f(x) = 3x^3 - 2x^2 + x + 1$ is continuous on $\mathbb{R}$.

Approach 1: use $\delta-\epsilon$ definition:

• Rough Work: Given $\epsilon > 0$, we must find $\delta > 0$ such that $0 \lt |x - a | \lt \delta \implies | f(x) - f(a) | \lt \epsilon$:

That’s a lot of work and we still have not found a suitable $\delta$ …

Suppose $\delta \lt 1$ such that $|x - a|\lt \delta \implies |x - a| \leq 1$

Using the trick we learnt at the beginning (i.e. the power of adding 0 by inserting terms that cancel each other):

$|x| = |x - a + a | \leq | x - a | + |a| \lt 1 + |a|$

This weird bounding of $|x| \lt 1 + |a|$ may seem weird but this is simply to bound $x$ in terms of a fixed value (constant) $a$ to simplify the long expression we retrieved above:

Therefore choose $\delta = \min\{\frac{\epsilon}{9|a|^2 + 13|a| + 6}, 1\}$

That was a lot of work just to finish the rough work and find the $\delta$. Let’s now look at using the theorem we learned recently:

Approach 2: Using the theorem (theorem 17.2 in Ross Analysis)

Suppose $a\in\mathbb{R}$ and suppose $\lim\limits_{n\to\infty} x_n = a$. We need to show $\lim\limits_{n\to\infty}f(x_n) = f(a)$:

As $f$ is continuous at $a$ and since $a\in\mathbb{R}$ was arbitrary, by the thoerem (17.2), $f$ is continuous on all of $\mathbb{R}$

It is obvious which method was preferrable … (unless you are a machochistic)

As one knows, the function $\cos(x^2+2x)$ is a composite function composed of one or more functions. Most functions we encounter are composed this way, so it is important to understand how composition affects the domain, since continuity at a point is only defined for points within the domain of the function.

Composite Functions: Let $f,g$ be real-valued functions. If $x\in dom(f)$ and $f(x)\in dom(g)$, we define $g \circ f(x) = g(f(x))$.

Then $g \circ f$ is a function defined on $\{x\in dom(f): f(x)\in dom(g)\}$ ($g$ is composed with $f$)

Example:

Let $f(x) = x+ 1$ and $g(x) = x^2$ then:

$(f\circ g)(x) = g(f(x)) = g(x+1) = (x+1)^2$

$g\circ f)(x) = f(g(x)) = f(x^2) = x^2 + 1$

As we can see $f \circ g \ne g \circ f$ in general

Now that we are familiar with composed function and their domain, let us now discuss continuity for composed functions:

Continuity for Composed Functions: If $f$ is continuous at $a$ and $g$ is continuous at $f(a)$ then $g \circ f$ is continuous at $a$

Here are some facts about continuous functions:

1. Polynomials are continuous on $\mathbb{R}$
2. Rational function ($\frac{p(x)}{q(x)}$ where $p,q$ are poilynomials) are continuous on $\{x\in\mathbb{R} | q(x) \ne 0\}$
3. $f(x) = \sqrt{x}$ are continuous
4. $\sin x, \cos x, \tan x$ are continuous within their domain
5. $e^x, 2^x, a^x \forall a \gt 0$ are continuous on $\mathbb{R}$
6. $\log (x), \log_a(x) (a \gt 1), \ln (x) = log_e(x)$ are continuous on $(0,\infty)$
7. $f(x) = x^p, \forall p\in\mathbb{R}$ is continuous on $\mathbb{R}$

Suppose we restrict the domain of some function to some closed interval $I$, then we can make the claim that the function is bounded on this interval.
Furthermore, we can also make the claim that the max and min values of the bounded continuous function can be obtained. Unsurprisingly, since the interval is closed, it is obvious that we can obtain the max and min.

Bounded Functions: let $f$ be a real-valued continuous function defined on the CLOSED interval $[a, b]$. Then $f$ is BOUNDED on $[a, b]$. Furthermore, $f$ assumes its max and min values on $[a, b]$.

This result is absolutely not surprising at all. If we remove the criteria that the function is closed then:

A graph $x^2$ where $x\in(-2,2)$.

In the above graph $x^2$ restricted in the open interval $(-2, 2)$ does not obtains its max since $x = 2 \notin (-2, 2)$.

Thus one can convince themself that for a continuous and bounded function $f$ to obtain its max and min values, it must be restricted on a closed interval. Note that if there was no bounds, the curve $x^2$ only has a minimum at $x = 0$.

Below is an example of the consequence of a curve $x^2$ with a jump at $x = 0$ not obtaining its min despite being closed between $[-2, 2]$ since the curve is not continuous:

A graph $x^2$ where $x\in[-2,2]$ with a jump at $x = 0$

Though if you are a die-hard theoretical Mathematician, feel free to attempt the proof yourself if you are not sufficiently satisfied with the conclusion.

The next theorem builds on top of the previous theorem on continuous bounded functions. Suppose we are mapping the temperature throughout the day. If in the morning it was $10^\circ C$ and $20^\circ C$ in the afternoon then we can say for certainty that there was some point between the morning and the afternoon where it reached $17^\circ C$. Temperatures don’t magically jump, it increases progressively (I dropped out of physics so I could be wrong).

For American readers, please switch to Celsius

Intermediate Value Theorem (IVT): let $f$ be a continuous and real-valued function on a closed interval $[a, b]$. Suppose that $y$ lies between $f(a)$ and $f(b)$. Then there exists at least one $x\in(a, b)$ such that $f(x) = y$.

A temperature curve showing how between the morning and the afternoon, we can guarantee that there is some time between the two where the temperature is at $17^\circ C$

Uniform Continuity

Previously, our definition of continuity of a function was local in scope, in other words, we could argue that a function was continuous at a particular point. In doing so, we discovered that our $\delta$ depended on both $\epsilon$ window we were trying to satisfy and the specific point we were arguing continuity at. A different point might require a completely different $\delta$ for the same $\epsilon$ window. Hence why when we were trying to show $f(x) = 3x^3 - 2x^2 + x + 1$ was continuous on $\mathbb{R}$, we choose our $\delta$ to be $\min{\frac{\epsilon}{9|a|^2 + 13|a| + 6}, 1}$ where it depended on both the $\epsilon$ and the point $a$. Thus we had infinitely many number of $\delta$ values to apply to show the polynomial function was continuous on $\mathbb{R}$. This is what uniform continuity tries to solve by having $\delta$ only depend on $\epsilon$ and hence have a global scope. Though not all functions are uniformly continuous so it is important some properties of uniform continuity. But first, let’s go through the formal definition of uniform continuity:

Uniformly Continuity on S: let $f$ be a real-valued function on a set $S$. We say $f$ is UNIFORM CONTINUOUS on $S$ if $\forall \epsilon > 0, \exists \delta > 0$ such that $(x, y\in S$ and $|x - y| \lt \delta)\implies |f(x) - f(y)| \lt \epsilon$

i.e. $\delta$ depends only on $\epsilon$ and $x, y$ are abitrary points that are extremely close to each other in the set

Since uniform continuity is not dependent on any single point in the graph but rather requires a single $\delta$ to work for all pairs of points $x, y$ close to each other simultaneously, it is a global property. Any two points that are close in the domain are guaranteed to map to points that are close in the range, regardless of where they sit on the curve.

It should come to no surprise that if $f$ is uniformly continuous on a set $S$ then $f$ is continuous on $S$ since being uniformly continuous is a much stricter definition than continuity. However, the inverse is not true.

$f$ is uniformly continuous on a set $S \implies$ $f$ is continuous on $S$

i.e. uniform continuous $\implies $ continuous

Note: continuity $\bcancel{\implies}$ uniform continuity

We previously saw that if a function $f$ is continuous on a closed interval, it is bounded and attains it max and min. Furthermore, we also discovered the ability to infer the existence of an event occuring if it lies between two different other events via the intermediate value theorem. What could we say about uniform continuity?

Theorem (Bounded Continuous Functions implication on Uniform Continuity): If $f$ is continuous on a closed interval $[a, b]$, then $f$ is uniformly continuous on $[a, b]$

Thus we can say the following:

$f$ is continuous on a closed interval $[a, b] \iff f$ is uniform continuous on $[a, b]$

What additional conclusion could we make on uniform continuity? Well we discussed how if $f$ is uniformly continuous on an interval $S$, then $f$ is continuous as well on the same interval. So if we take a subset $U\subseteq S$, we can also make the same claim that $f$ is also uniformly continuous on $U$, and thus $f$ is continuous on the subinterval.

Let’s look at the function $\frac{1}{x}$ and see at what intervals is it uniformly continuous and where it is not to understand better what uniformly continuous is and is not.

Example: Prove that $\frac{1}{x}$ is uniformly continuous on $[b, \infty)$ where $b > 0$

Rough work: Let $\epsilon > 0$ and we want to find $\delta \gt 0$ such that $\forall x,y,\in [b, \infty)$ with $|x - y| \lt \delta$, we have $|\frac{1}{x} - \frac{1}{y}| \lt \epsilon$

Recall that for $f$ to be uniformly continuous on $[b, \infty)$, $\delta$ cannot depend on any points $x, y$. But we could utilise what we know about $x, y$ to achieve this:

Recall that $x,y\in[b, \infty) \iff x \ge b, y \ge b \iff \frac{1}{x} \le \frac{1}{b}, \frac{1}{y} \le \frac{1}{b}$

Thus we now have the following:

So take $\delta = \epsilon b^2$.

Notice that we’ve only shown that $f$ is continuous for provided that the domain is greater than 0 in a closed interval. Before we analyse what would happen if we changed the interval from closed to open for the function $\frac{1}{x}$, we’ll first need to revisit our favorite sequence, the cauchy sequence:

Theorem (Uniform Continuity Impact on Cauchy Sequence): Let $f$ be a uniformly continuous function on S. If $(S_n)$ is a cauchy sequence in $S$, then $(f(s_n))$ is also a cauchy sequence

Practical Use: The contrapositive is often evoked to disprove a given function is uniformly continuous by finding a cauchy sequence $(s_n)$ such that $(f(s_n))$ is not cauchy (and hence diverges)

Example: Show that $f(x) = \frac{1}{x}$ is not uniformly continuous on $(0, \infty)$

The goal is to use the contrapositive of the theorem presented above by crafting a cauchy sequence such that $f(s_n)$ is not cauchy.

Let $(s_n) = (\frac{1}{n})_{n\in\mathbb{N}}$. $(s_n)$ is cauchy since it converges to $0\notin (0, \infty)$. This will be important to note. To disprove uniformly continuitity, we deliberately chose a cauchy sequence that converges to the problematic region near 0 wher ethe slope of $\frac{1}{x}$ becomes arbitrarily large.

Then $(s_n)$ is a cauchy sequence in $(0, \infty)$ but $f(s_n) = \frac{1}{s_n} = \frac{1}{\frac{1}{n}} = n$ which is not cauchy. Hence by the theorem presented above, $f$ is not uniformly continuous on $(0, \infty)$

Heuristic on Disproving Uniform Continuity: if a function’s “slope” gets arbitrarily large on $S$ like $\frac{1}{x}$ near 0, then it won’t be uniformly continuous there i as there does not exist a single $\delta$ that can accommodte the increasingly steep behavior across the entire domain

Limits of a Function

Thus far we have talked a lot about limits in this course from sequences, partials sums, and with functions to define continuity. We will now talk about limits of a function in more details, particularly as it approaches $\pm\infty$.

Recall that the standard definition of limits $\lim\limits_{x\to a} f(x) = L$ means $\forall \epsilon > 0, \exists \delta \gt 0$ such that $0 \le |x - a| \lt \delta \implies |f(x) - L| \lt \epsilon$ One may recall that taking the limit of a function to a point $a$ is equivalent as approaching the limit from the left and the right of the point $a$:

$\lim\limits_{x\to a} f(x) = L \implies \lim\limits_{x\to a^-} f(x) = \lim\limits_{x\to a^+} f(x) = L$

Formally,

One Sided Limit Definition: let $L\in\mathbb{R}$, let $f$ be a function and let $a$ be a limit of some sequence in $dom(f)$ consisting of terms larger than the point $a$.

Then $\lim_{x\to a+} f(x) = L$ means $\forall \epsilon \gt 0, \exists \delta \gt 0$ such that $(x\in dom(f)$ and $a \lt x \lt a + \delta) \implies |f(x) - L| \lt \epsilon$

i.e. the limit of a function of $f$ at a point $a$ approaching from the right side is $L$

The definition for approaching from the left is left as an exercise to the reader.

The curve of $\frac{1}{x}$

Let’s revisit the function $\frac{1}{x}$, it has a vertical asymptote at $x = 0$ whereby the curve blows up to $\pm \infty$ from the left and the right side of the vertical asymptote:

The formal one-sided limit definition presented earlier is insufficient to express limits that converges to $\pm \infty$ (i.e. diverge) since $\infty\notin\mathbb{R}$ (i.e. $\mathbb{R}$ contains an infinitely many finite numbers but $\pm\infty$ is not a concrete finite number). Thus we now need a new definition for these limits converging to $\pm\infty$:

$\lim\limits_{x\to a^+} f(x) = \infty$ means ($\forall M \gt 0, \exists \delta \gt 0$ such that $a \lt x \lt a + \delta \implies f(x) \gt M$)

$\lim\limits_{x\to a^-} f(x) = \infty$ means ($\forall M \gt 0, \exists \delta \gt 0$ such that $a - \delta \lt x \lt a \implies f(x) \gt M$)

$\lim\limits_{x\to a} f(x) = \infty$ means ($\forall M \gt 0, \exists \delta \gt 0$ such that $0 \lt |x - a| \lt \delta \implies f(x) \gt M$)

The precise definition for approaching to $-\infty$ is left as exercice (though it’ll be shown in an example shortly). Essentially what the definition states is that regardless of how arbitrary large $M$, we can find a value $x$ within the neighborhood of $a$ (the $\delta$ window) such that $f(x)$ is larger than $M$. In other words, regardless how large $M$ is, there’s always going to be another value larger than it within the neighborhood. Sort of reminds me of the two archmidean properties. Let’s use these precise definition to prove the behavior of $\frac{1}{x}$:

Example: Show (1) $\lim\limits_{x\to 0^+} f(x) = \infty$ and (2) $\lim\limits_{x\to 0^-}f(x) = -\infty$

Rough Work for (1): Given $M \gt 0$, we want to define a $\delta \gt 0$ such that $0 \lt x \lt 0 + \delta \implies f(x) = \frac{1}{x} \gt M$. Thus we want the following:

So take $\delta = \frac{1}{M}$

Rough Work for (2): Given $M \lt 0$, we want to define a $\delta \gt 0$ such that $0 - \delta \lt x \lt 0 \implies f(x) = \frac{1}{x} \lt M$. Let’s recall the following:

• $x \lt 0$ so let’s define $x’ \gt 0$ such that $x = -x’$
• $M \lt 0$ so let’s define $M’ \gt 0$ such that $M = -M’$
• $-\delta \lt -x’ \lt 0 \implies 0 \lt \boxed{x’ \lt \delta}$

When approaching $+\infty$, we want $f(x)$ to exceed $M$ so we need $x$ to be small. But when we approach $-\infty$, we want $f(x)$ to go below $M$, so the inequalities are reversed. This could explain any unease you may had when working on the proof for (2). Thus we shall take $0 \lt \delta = \frac{1}{M’} = \frac{-1}{M}$ where $M < 0$ (i.e. negative). While I try not to give a complete proof as this is an accompanying material to the course and not a replacement, I think the rough work will make more sense when seeing it in proven formally:

Let $M \lt 0$ and choose $0 \lt \delta = \frac{-1}{M}$. Then $0 - \delta \lt x \lt 0 \implies -\delta \lt x \implies -(-\frac{1}{M}) \lt x \implies \frac{1}{x} \lt M$ as required. (i.e. we need to show that for any $M \lt 0$, for all $x$ in the neighborhood of 0 (from the left), $f(x)$ will be less than $M$). Therefore $\lim\limits_{x\to 0^-}f(x) = -\infty$.

Observe how our favorite function thus far, $\frac{1}{x}$, exhibits another convergence, but this time to a finite value 0 in what appears to be another asymptote called the horizontal asymptote. Thus far we have discussed the precise definition of what it means to converge to a point from the left and the right as the curve approaches to a particular point in the cartesian plane to either a finite value or to $\pm \infty$. Yet, we still lack the language to describe limits as $x$ approaches to $\pm \infty$. Thus here are the last missing pieces:

Limit Definitions @ $\pm\infty$: $\lim_{x\to\infty} f(x) = L$ where $L$ could be $\pm \infty$ or $L\in\mathbb{R}$

1. If $L\in\mathbb{R}$: $\forall\epsilon \gt 0$, $\exists \alpha\in\mathbb{R}$ such that $(x \gt \alpha)\implies |f(x)-L|\lt\epsilon$
2. If $L = \infty$: $\forall M \gt 0$, $\exists \alpha\in\mathbb{R}$ such that $(x \gt \alpha)\implies f(x)\gt M$
3. If $L = -\infty$: $\forall M \lt 0$, $\exists \alpha\in\mathbb{R}$ such that $(x \gt \alpha)\implies f(x)\lt M$

As usual, the precise definitions for $\lim\limits_{x\to -\infty} f(x) = L$ is left as an exercise to the readers. Suppose we have a sequence $(x_n) \to a$, would the limit $\lim\limits_{x\to a} f(x) = \lim\limits_{n\to\infty} f(x_n)$? That is what the next theorem is about:

Theorem: Let $f$ be defined on a set $S$ and let $a$ be the limit of some sequence in $S$ (including the possibility that $a = \pm\infty$). Let $L\in\mathbb{R}$. Then:

$\lim\limits_{x\to a}f(x) = L \iff $ for every sequence $(x_n)$ in $S$ with limit $a$ but $x_n \ne a \forall n$, we have $\lim\limits_{n\to\infty}f(x_n) = L$

Let’s revisit the limit laws again but for functions instead of sequences:

LIMIT LAWS: let $f, g$ be functions defined on a set $S$ for which $\lim\limits_{x\to a}f(x) = L, \lim\limits_{x\to a}g(x) = M$ for $L, M$:

1. $\lim\limits_{x\to a}(f(x) + g(x)) = L + M$
2. $\lim\limits_{x\to a} (fg)(x) = LM$
3. $\lim\limits_{x\to a} (\frac{f}{g})(x) = \frac{L}{M}, \qquad M \ne 0, g\ne 0$ around $a$

Note: the limit laws above apply for one-sided limits as well as for approaching the function to $\pm\infty$ (i.e. $a$ does not need to be a real number).

The limit laws handle sums, products, and quotients. But what about more complex functions? Composite functions are necessary to construct any complex functions yet we will soon discover the limit laws do not apply.

WARNING: Composite functions does not respect the limit laws

The graph of $1+x\sin(\frac{\pi}{x})$

Let $x_n = \frac{2}{n}$ for $n\in\mathbb{N}$ and $\lim\limits_{n\to\infty}x_n = 0$.

When n is even, $f(x_n) = 1 + x_n \sin(\frac{\pi}{x_n}) = 1 + \frac{2}{n}\sin(\frac{\pi}{\frac{2}{n}}) = 1 + \boxed{\frac{2}{n}\sin(\frac{n\pi}{2})} = 1 + \boxed{0} = 1$

When n is odd, $f(x_n) = 1 + x_n \sin(\frac{\pi}{x_n}) = 1 + \frac{2}{n}\sin(\frac{\pi}{\frac{2}{n}}) = 1 + \boxed{\frac{2}{n}\sin(\frac{n\pi}{2})} = 1 + \boxed{\pm \frac{2}{n}} \ne 1$

Hence the composition is: \(\begin{align*} $g\circ f$(x_n) = g(f(x_n)) = \begin{cases} g(1) & \text{$n$ is even} \\ g(1\pm\frac{2}{n}) & \text{$n$ is odd} \end{cases} = \begin{cases} -4& \text{$n$ is even} \\ 4 & \text{$n$ is odd} \end{cases} \end{align*}\)

As $g\circ f(x_n)$ alternates between -4 and 4, $\lim\limits_{n\to\infty}g\circ f(x_n)$ does not exist and thus neither does $\lim\limits_{n\to\infty}g\circ f(x)$

So what can we conclude about the limits for a composite function then? Well there are now two conditions that need to be satisfied instead of one. For the composite function $g\circ f$:

1. the inner function $f(x)$ converges to a value that exist in the outer function’s (i.e. $g(x)$) domain (a requirement already for the composite function to exist)
2. $g$, the outer function, is continuous on $f(x) = L$ (NEW CONDITION)

Theorem on the Limits of Composite Functions: let $f$ be a function defined on $S$ for which $\lim\limits_{x\to a}f(x) = L$ exists within $L\in\mathbb{R}$. Let $g$ be a function defined on $\{f(x) | x\in S\} \cup \{L\}$ which is continuous at $L$. Then:

$\lim\limits_{x\to a}(g\circ f)(x) = g(L)$ (i.e. $\lim\limits_{x\to a}(g\circ f)(x) = g(\lim\limits_{x\to a}f(x)$)

Here are some facts about the following functions:

• $\lim\limits_{x\to 0} \frac{\sin(x)}{x} = 1$
• $\lim\limits_{x\to 0} \frac{\cos(x) - 1}{x} = 0$

But one may ask how did one came to this conclusion? If we were to approach this problem using what we have seen thus far, it would seem impossible. But you may recall seeing how the squeeze theorem could be utilised to determine whether a difficult sequence converges by bounding it between two simpler sequences. We’ll employ the same idea to determine the limits presented above by bounding these difficult functions between other simpler functions and see what happens.

Squeeze Theorem (for functions): suppose $f(x)\leq g(x)\leq h(x) \forall x$ and $\lim\limits_{x\to a}f(x) = L = \lim\limits_{x \to a} h(x)$ then $\lim\limits_{x \to a} g(x) = L\in\mathbb{R}$

Example: Show $\lim\limits_{x\to 0} \frac{\sin(x)}{x} = 1$

For $\frac{-\pi}{2}\lt x \lt \frac{\pi}{2}$ and $x\ne 0$, we have:

where:

• $\lim\limits_{x\to 0} g(x) = \lim\limits_{x\to 0} \cos x = \cos(0) = 1$
• $\lim\limits_{x\to 0} h(x) = \lim\limits_{x\to 0} 1 = 1$

So by squeeze theorem, $\lim\limits_{x\to 0} \frac{\sin(x)}{x} = 1$

Note Showing $\lim\limits_{x\to 0} \frac{\cos(x) - 1}{x} = 0$ is more involved but as a hint:

DIFFERENTIATION

After many weeks taking a calculus course, we finally have reached the point of learning what most think of calculus: differentiation (and integration). I am not going to delve much into this subject despite being a central component of Calculus simply because it is both a review from Highschool calculus and is not distinct from a regular calculus course itself.

The central theme in the course thus far has been on limits considering how many weeks have been dedicated to the study of limits and its convergence both using the precise definition and the simple definition of what it means to take the limit of a function as it approaches to a finite number or to $\pm\infty$. It turns out unsurprisingly that the definition of a function being differentiable at a particular point is taking the limit of a function $f$ around the point $a$. This is commonly known as determining the rate of change by observing the slope of the tangent:

Differentiation: Let $f$ be a function defined on an open interval containing the point $a$. We say a function $f$ is differentiable at the point $a$ (or has a derivative at $a$) if:

$\lim\limits_{x\to a} \frac{f(x) - f(a)}{x - a}$ exists and is finite

In other words:

$f’(a) = \lim\limits_{x\to a} \frac{f(x) - f(a)}{x - a}$

Setting $x = a + h$, as $x\to a$, we have $h\to 0$, giving the equivalent formulation:

$f’(a) = \lim\limits_{h\to 0} \frac{f(a+h)-f(a)}{h}$

What makes calculus for Mathematic students vs. Engineering is that we do not rely on intuition of what it means to take the “instantaneous rate of change” but build this up using the precise definition using the $\delta-\epsilon$ proofs to have a clear understanding of what it truly means to take the limit.

Using the definition of differentiation, we can derive all sorts of derivative rules that we come to love and memorise:

Differentiation Rules: Let $f,g$ be differentiable at $x = a$ and let $c\in\mathbb{R}$ be a constant. Then $cf, f+g, fg,$ and $\frac{f}{g}, g(a)\ne 0$ are differentiable at $x = a$. Their derivatives are:

1. $(cf)’(a) = cf’(a)$
2. $(f+g)’(a) = f’(a) + g’(a)$
3. $(fg)’(a) = f’(a)g(a) + f(a)g’(a)$
4. $(\frac{f}{g})’(a) = \frac{f’(a)g(a) - f(a)g’(a)}{g^2(a)}, \qquad g(a)\ne 0$
5. $(g\circ f)’(a) = g’(f(a))f’(a)$

Chain Rule: if $f$ is differentiable at $x = a$ and $g$ is differentiable at $f(a)$, then $g\circ f$ is differentiable at $a$, with $(g\circ f)’(a) = g’(f(a))f’(a)$

One gripe I have with textbooks is that they often don’t present the entire story of the derivatives of transcendental functions such as $\sin x, \cos x, e^x$ causing many students to omit the chain rule.

For instance, here are the derivatives of common transcendental functions:

• $(\sin x)’ = \cos x$
• $(\cos x)’ = -\sin x$
• $(e^x)’ = (e^x)$

The derivatives above assume the argument is simply $x$. The moment you have anything more complex such as $\sin(x^2)$ or $e^{3x}$, the chain rule is mandatory. Thus when I teach students the derivatives of transcendental functions, I ensure I explicitly present the chain rule in their derivatives as follows:

• $(\sin x)’ = (\cos x)x’$
• $(\cos x)’ = (-\sin x)x’$
• $(e^x)’ = (e^x)x’$
• $(a^x)’ = (a^x \ln a)x’$
• $(\ln x)’ = \frac{1}{x}$
• $(tanx)’ = (sec^2x)x’$

Note: $(e^x)’ = (\bcancel{\ln e} e^x)x’ = (e^x)x’$

Example: Take the derivative of $h(x) = e^{\sin(2x)}$

$h(x) = g\circ f(x)$ where:

• $g(x) = e^x$ and $g’(x) = (e^x)x’ = e^x$
• $f(x) = \sin(2x)$ and $f’(x) = \cos(2x)(2x)’ = 2\cos(2x)$

By chain rule: $h’(x) = (g\circ f)’(x) = g’(f(x))f’(x) = e^{2\sin(2x)} \cdot 2\cos(2x) = 2\cos(2x)e^{\sin(2x)}$

An interesting question is whether if a function being continuous implies differentiability. The answer as it turns out is a resounding no.

Example: Is $f(x)$ differentiable at $x = 0$?

The graph of $\|x\|$

To show $f$ is differentiable at $x = 0$, then we want to show $\lim\limits_{x\to 0}\frac{f(x) - f(0)}{x - 0}$ exists. Recall for the limit to exist, it’s left handed and right handed limits must exist.

Right-Sided Limit:

Left-Sided Limit:

As $\lim\limits_{x\to 0^+} \frac{|x|}{x} \ne \lim\limits_{x\to 0^-} \frac{|x|}{x}$, $\lim\limits_{x\to 0}\frac{f(x) - f(0)}{x - 0}$ does not exist and thus is not differentiable despite being continuous. Thus we can only conclude from one side, that if $f$ is differentiable at $x = a$ then it is continuous but not the other way around.

Theorem: if $f$ is differentiable at $x = a$, then $f$ is continuous at $x = a$

i.e. differentiable at $x = a \implies f$ is continuous at $x = a$ (but not the other way around)

A good tip to determining whether a function is differentiable at a particular point is to see if there are any sharp edges like we see at $x = 0$ in the function $|x|$.

Mean Value Theorem

One common theme you will notice in analysis is a class of theorems that proves the existence of a phenomen but in a more vague manner. For instance, the Intermediate Value Theorem (IVT) tells us that if a value $y$ lies between $f(a)$ and $f(b)$ then there exists at least one value $x$ in between $a$ and $b$ such that $f(x) = y$ (given $f$ is a continuous function). IVT does not tell us precisely what the $x$ value is but rather bounds the value between two points. Another such theorem in the same spirit as IVT is the Rolle’s Theorem which tells us the existence of a local min or max within two points. But first let us review how to know if we obtain a min or max:

Theorem: Let $f$ be defined on an open interval containing $x_o$. Suppose that $f$ attains its max or minimum at $x_o$ and that $f$ is differentiable at $x = x_o$ then $f’(x_o) = 0$

For a continuous and differentiable function to hit its local min or max at $x = x_o$, we must observe absolutely 0 change at that point (i.e. $f’(x_o) = 0$. We will later explore how to graph functions based on its derivatives but it is important to understand that if the derivative at a particular point is not zero, then the graph is experiencing a change and thus has not hit a local min or max.

Recall that to hit a local min, we must have the derivatives $f’(x_o - \delta) \lt f’(x_o) = 0 \lt f’(x_o + \delta)$ or what I call a happy face. i.e. the signs of the derivatives changes from negative to positive.

A graph of $x^2$

If I was to draw a table to visualise this change of slope it would look something along the lines of:

or

Now here’s the Rolle’s theorem now that we established the fact that an extremum occurs when the slope is 0 (i.e. $f’(x_o) = 0$.

Rolle’s Theorem: Suppose $f$ is continuous on $[a,b]$, $f$ is differentiable on $(a,b)$ and that $f(a) = f(b)$. Then there exists $c\in(a,b)$ with $f’(c) = 0$

Note: Rolle’s Theorem is actually a special case of the Mean Value Theorem presented below where $f(a) = f(b)$ where by the average rate of change is $\frac{f(b)-f(a)}{b-a} = 0$, so the MVT guarantees there exists some $c$ where $f’(c) = 0$

The next theorem relates to how one can draw a tangent line parallel to the tangent slope which itself does not seem impressive from face value as it states the existence of the average rate between two points is equal to a instantaneous rate of change between the two points:

Mean Value Theorem (MVT): let $f$ be continuous on $[a, b]$ and differentiable on $(a,b)$ then there exists $x\in(a,b)$ with $f’(x) = \frac{f(b)-f(a)}{b-a}$

Illustration of MVT extracted from Paul's Math Notes

While seemingly inconsequential, the consequences of MVT are interesting. For instance, if $\forall x\in\mathbb{R}, f’(x) = 0$ then $f$ is a constant line. But more interestingly is when two functions seemingly have the same slope (i.e. derivative):

Collary of MVT 1: Let $f,g$ be differentiable function on $(a,b)$ and suppose $f’=g’$ on $(a,b)$. Then $\exists c \in\mathbb{R}$ (a constant) such that $f(x) = g(x) + c \forall x$

The strongest corollary of the MVT presented in the course is understanding the behavior of a function based on its slope (derivative):

Collary of MVT 2: Let $f$ be differentiable function on $(a,b)$:

1. $f’(x) \gt 0 \forall x\in(a,b) \implies f$ is STRICTLY INCREASING on $(a,b)$
2. $f’(x) \lt 0 \forall x\in(a,b) \implies f$ is STRICTLY DECREASING on $(a,b)$
3. $f’(x) \geq 0 \forall x\in(a,b) \implies f$ is INCREASING on $(a,b)$
4. $f’(x) \leq 0 \forall x\in(a,b) \implies f$ is DECREASING on $(a,b)$

L’Hôpital’s Rule

We have finally made it to the last major topic covered in the course, the L’Hôpital rule, the one rule whose name will reveal whether or not you failed French class (i.e. H is silent in French and as Canadians, we should know at least that much). Thus far in the course, we’ve been working with nice limits whose mathematical results won’t make us scratch our heads. But suppose we have the following limit:

where $a$ can be a one-sided limit, a number or even $\pm \infty$. There are forms where $\lim_{x\to a} f(x)$ and $\lim_{x\to a} g(x)$ could be equal to either $0$ or $\pm \infty$ and would cause our heads to scratch as the results are undefined. These forms are what we call indeterminate forms and there are many such as:

• $\frac{0}{0}$
• $\frac{\infty}{\infty}$
• $0 \cdot \infty$
• $\infty - \infty$
• $0^0$
• $\infty^0$
• $1^\infty$

For instance $\lim\limits_{x\to 0} \frac{\tan (x)}{x} = \frac{0}{0}$ which is undefined.

While some of these forms can be manipulated to be solvable such as in the case of $0\cdot \infty$ where one could rewrite the limit in the form of $\frac{f(x)}{\frac{1}{g(x)}}$ or $0^0$ as $e^{g(x)\ln f(x)}$, others requires another technique:

L’Hôpital’s Rule: let $s$ be one of $a, a^+, a^-, +\infty, -\infty$. Suppose $f,g$ are differentiable functions for which $\lim\limits_{x\to s}\frac{f’(x)}{g’(x)} = L$ exists.

If $\lim\limits_{x\to s} f(x) = \lim\limits_{x\to s} g(x) = 0$ or if $\lim\limits_{x\to s}|g(x)| = \infty$, then $\lim\limits_{x\to s} \frac{f(x)}{g(x)} = L$

To clarify, while there exists indeterminate forms not in rational forms such as $\infty - \infty$, to apply L’Hôpital’s Rule, one must rewrite it into rational form (for $\infty-\infty$ case, you would need to multiply by it’s conjugate and rationalise it).

i.e. indeterminate form $\xrightarrow{\text{rewrite}} \frac{0}{0}$ or $\frac{\infty}{\infty} \xrightarrow{\text{L’Hôpital}} L$

Example: Find $\lim_{x\to 0}\frac{\tan(x)}{x}$

Recall that $\lim\limits_{x\to 0} \tan(x) = \tan(0) = 0$ so we have $\lim\limits_{x\to 0}\frac{\tan(x)}{x} = \frac{0}{0}$ form

As both functions are differentiable around 0 (and $x’ = 1 \ne 0$), we can apply the L’Hôpital Rule: $\lim_{x\to 0}\frac{\tan(x)}{x} = \lim_{x\to 0}\frac{(\tan(x))’}{(x)’ = \lim_{x\to 0}\frac{\sec^2(x)}{1}} = 1$

Example: $\lim_{x\to 0^+} x\ln(x) = 0\cdot \infty$

Recall that $\ln(x)$ and $\frac{1}{x}$ are differentiable around $0^+$ and that $(\frac{1}{x})’ = (x^{-1})’ = -x^{-2} = \frac{-1}{x^2}$ is non-zero for $x\to 0^+$

So we can try L’Hôpital’s rule by rewriting it as a rational function:

As stated earlier, one needs to ensure to rewrite the indeterminate form into a rational form such that it is in the form: $\frac{\infty}{\infty}$ or $\frac{0}{0}$. We have seen an example of rewriting $0\cdot \infty$ but let’s now look at rewritting $0^0$ using the fact $e^{\ln(y)} = y, y\gt 0$:

Example: $\lim\limits_{x\to 0^+} x^x$

$x^x$ can be rewritten as $e^{\ln (x^x)} = e^{x\ln (x)}$

Since we know $e^x$ is a continuous function, and is differentiable everywhere. Thus we have:

Credits:

• MATH1052 Notes from Charles Starling
• Elementary Analysis: The Theory of Calculus by Kenneth A. Ross

Note: I refer to this page as a commentary to the course or an accompanying to the course, these are not class notes. I the author of this page will omit many details and only present the materials and proofs to things I find personally noteworthy and the proofs presented may differ from those listed in the textbook or from the course notes itself.

>>> 3.14*10
31.400000000000002

This behavior can have unintended consequences which I intend to explore a bit more in a future blog. But what I want to share with you is about the additive associativity property that we come to love when operating on a field. For simplicity’s sake, especially for those without Mathematical background, think of the field as the number system you use in your everyday life that has a set of additive and multiplicative properties that you are already familiar with. Additive Associativity is the property that allows you to regroup operations and thus changing the order of priority to perform the addition such as:

(a + b) + c = a + (b + c)

If something is associative, then this regrouping of priority of operations should not affect the final outcome. At work, I have been attending some classes to learn more about using GPUs to do Mathematical computations. In one of the classes, it came to my attention that floating point numbers are not associative. This fact shocked me. We were working on computing the Basel problem which is a famous mathematical problem taught to first years:

In Mathematics, infinite series are not associative but finite series are. For some fixed $N$, I was expecting the order in which we add these numbers would not matter but it apparently does.

To illustrate:

is what I would expect but this does not appear to be the case as seen in the last digit:

 forward:1.5397677311665408
backward:1.5397677311665405

Now one might question which direction is correct or at least more accurate. The most accurate way for this particular example, the basel problem, is to add the numbers in the opposite (backwards) direction due to how starting with a large number like 1 will dominate the subsequent additions and hence lose accuracy as the hardware will round off the smaller terms and eventually deviate from the true sum as the rounding errors accumulate. However, when working on many threads, this can complicate things as one needs to now consider how to reduce the independent terms in such a way that minimises inaccuracy while remain performant. But that’s a topic for another time (I am still researching this).

• Book Recommendations: random non-fiction books I liked and would recommend for anyone interested in computer science
• Blog: My long form content that I have been maintaining from my former blogsite. Hopefully I resolved all the issues when it comes with migration including photos and internal links
• Archives for µBlog as it was getting a bit too overwhelming to display all the micrblogs in one giant page
• UI Changes: With the help of AI (I am bad at colors), I increased the contrast and outlined links to be more obvious. Lately, I’ve been enabling limited contrast mode at home and at work as I find it easier to navigate so I did the same on this website.

As to why I made these changes, it’s not for a good reason … I am just procrastinating from preparing for my language exam … Ironic how I signed up for the exam in hopes I would study more seriously but alas I just procrastinate by doing other things. Anyhow, it’s a nice edition to the site.

Below are some examples of the visual UI changes on the site:

Before:

Now:

Before:

Now:

On another note, I have decided that maintaining two separate sites to be a burden and it also made my blog look dead. Therefore, for now on, both neocities and my blog will get both updates and are now effectively mirrors of each other. Actually in fact my current blog will be sunsetting soon as it now has a new link.

Over the years, I have changed where I hosted my blog and websites which can be a bit cumbersome for people to follow me. Thus I now have bought the domain name randombits.ca in hopes that visitors will no longer need to follow my new link every few years. But who knows if I’ll keep this domain name for years to come. Only time will tell. At the moment the url directs visitors to my codeberg page but it is a mystery to me as to why trying to visit the mirror via the usual codeberg page link directly (i.e. user.codeberg.pages) does not display the right content but that’s fine for now. It’ll probably fix itself in due time and in the event I migrate once again to another hosting provider, I can fix it then so I can have 3 mirrors as opposed to one. I love redundancy :) It does make monitoring traffic much more difficult but I try my best to avoid monitoring stats for my mental health anyways. On another note, I do know a few folks having trouble visiting neocities due to their browser telling them it’s unsafe. I am not sure as to why but it could be their ISP providers telling them this. I recall having a video call with someone in Indonesia a year or two ago whose browser was complaining about it. If I have multiple mirrors, perhaps this would not be an issue for them as they could simply try another server. Perhaps I’ll revive my Github account and have it act as another mirror.

There are two potential concerns I have with merging my microblog and my other blogsite:

1. People will now see any potential chaotic content I may be posting on my blogsite which was geared to be more professional. In the past I indicated that neocities was my place to write random things that may not be a good fit for my other website. Well that no longer applies anymore as the sites are merged.
2. Merging two websites means a lot more storage. I was in a pleasant surprise that neocities is quite generous with their storage limit of 1GiB, plenty enough for me. This amount may not be sufficient in the far future but that’s a problem for the future me. There’s also an issue with what file types I can upload in neocities. I would need to be a paid subscriber to upload certain file types and even then, it’s not open to all file types.

I have no qualms paying for the subscription for a year or two to return the service for what neocities has provided me. But I don’t think that would be a viable long-term solution considering I could rent a cloud instance for about the same price with more capabilities. I recognise that hosting is not cheap nor does it scale linearly so it is my intention to eventually buy a Neocities pro member in a year time from now once I graduate university and get a job. Sadly I cannot go straight to masters as I don’t simply have the funds to justify spending another tens of thousands of dollars. I already spent a lot doing a 2nd undergraduate degree for fun … a very bad financial decision.

Not totally related but this reminds me of a shared library exploit whereby the shared library intercepts the filesystem calls to readdir and getdents which is a neat trick. Though this is at a userspace-level exploit and requires one to preload the malicious library (LD_PRELOAD).

The scary thing about this rootkit and any decent rootkits in general is its ability to hide itself. It is designed to hide itself thanks to it’s kernel-level privileges. It can intercept calls to various filesystem and network syscalls and APIs to conceal itself. This is why I am against the idea of allowing kernel-level anti-cheat code to be on my system. The idea of introducing a new level/ring in between userspace and kernel space or to introduce some new capabilities in userspace with controlled but limited and secured access to the kernel has been floated for years (though we do have eBPF which sort of functions like this). The 2024 Crowdstrike Incident for instance has compelled Microsoft to roll out a new security level to hopefully prevent this from ever occurring. Though it is not the issue of ensuring availability of the system that I am worried about, its the fact that we are placing trust to a non-open source third-party to have access to the kernel. Who knows what craziness they could do even if not intentionally (i.e. supply chain attacks).

Thankfully it would appear that this rootkit can be detected if observed at a third-party computer (i.e. examining the hard drive on another computer) or capture network traffic on a non-infected system. Fun fact, you can snoop harddrives without knowing their login as long as the drive is not encrypted. This was an eye-opener experience and made me think of my family desktops that we threw out over a decade ago.

On a weird note, Singularity also happens to be the name of an experimental microkernel OS by Microsoft that relied on software construction and memory safe language to ensure memory isolation and failure containment.

Unclear what you mean “click” but here’s my three stage answer. Sorry for its length in advance. Also this is from an old student who started most of their programming journey way before LLMs like ChatGPT even existed. In my opinion, you will experience many clicks in your career.

Summary:

1. Think Like a Programmer: The first “click” occurs when one is able to break down a problem into smaller tasks and then have the ability to translate their ideas on how to translate each small task into code

2. Think Like a Computer Scientist: Another “click” occurs when one makes the realization that there are different ways of approaching a problem that are more space or time efficient. The ability to solve hard problems by making use the tools you acquired from your education and personal experience and thinking outside of the box.

3. Having a Personal Connection: Another “click” occurs when one has a personal or professional relationship with programming. When one writes a program that someone can actually use and benefit from. This is the experience of creating a program with more initimate relation with as school projects are just for grades and your tiny simple programs previously were just for practice and familiarization.

1. The Beginning - Think Like A Programmer

From my observation, there seems to be a learning curve to think like a programmer. This is the ability to breakdown an assignment into smaller tasks and then reason how to approach each task. This often takes the average student anywhere between 2-6 months typically. However, this wall only gives you access to think like a programmer. I am unsure how first year CS courses are like at this university but at some universities, they would give a student a 2-4 page document that describes the overall goal of the project and some minimal starter code and have students bang their head from going through the starter code and the document to figure out how to achieve the objectives and what is even wanted for the assignment.

This phase is crucial to overcome and you will gain an extremely important ability, the ability to think like a programmer. In addition, programmers often work on existing codebase so gaining the ability to read existing codebase and connecting how various components come together to solve your problems that will continue throughout your programming career.

Advice: Do not panic if you are lost in your first month or two of programming. It is normal not being able to translate high level ideas to code. This takes practice. I often tell students try writing something simple outside of class to review what they learned. When I was a kid self learning programming from books and from whatever resources there were on the internet, I would often write stupid little 10-20 line program in C such as asking a user their name and age and determine if they can vote or not, making a simple calculator that takes in two numbers and an operation to execute, printing alternative messages and etc. Small Text adventure or other simple games like Black Jack are great simple ideas. Text parsing are also great ideas.

Students often want to write complex programs which is great but you got to start from somewhere.

2. Think Like A Computer Scientists: Introduction to Data Structures

Now that you got a hang onto the basics of programming, you are ready to learn the foundations of Computer Science which may be after your first CS course or in 2nd year depending on your university. This is where you need to think more critically how you organize your data more critically and create different approaches to solve the same problem to be more efficient. Time complexity and various data structures are shown.

[A 2nd-year introductory Data structure and algorithm course] is a great course to witness the power of time complexity and making use of different data structures. You are in 2nd year and perhaps you have not taken this course yet but this course will equip you with the basic understanding and appreciation of different algorithms and data structures. If you can do all the assignments, this is where I think the “click” will occur. At least this will click to you that coding is more than copying and pasting. The course was a fun elective because it required you to understand the concepts from the course and think outside of the box sometimes. It gave me nostalgic memories of when I was young hitting my head to the wall till I can think of a potential solution.

Note: I am a Mathematics student who took a data structure course for fun at some university

I had the fortunate realization of the importance of time complexity and data structures when I was in Highschool trying to parse large sets of text data to create a relationship between different sets of data which required a lot of insertion and searches. That was when I theorized the basics of time complexity and data structures as I would spend hours and days writing several implementations on paper (I made sure to carry a pencil and paper everywhere). It was so fun to see that many of my theories were actually a known concept taught in university (all my chemistry theories were debunked unfortunately but it was fun discussing it with my teachers). I have other examples such as in first year (not [university]), we had to write a generic primitive puzzle solver “AI” that can solve different puzzles. The condition of the assignment was that we needed to have the AI solve a set of different puzzles like sudoku and Peg solitaire under 3 mins using Bread-first search and Depth-first search. My initial implementation took an hour and I kept iterating and then I had this genius idea of pruning the trees to reduce the search space which I later found out was a common technique in AI years later. Why am I babbling about this? Good question, I am bored but to answer more seriously, this is an example of “click”. The ability to think of a solution outside of what you learned or being able to identify and make use of whatever it is in your toolbox to solve your problem is more than copying and pasting.

3. Solving Real Life Problems: Finally writing something useful for once

Many students have said over the years to me of their frustration with their relationship with coding. They know the basics but are stuck on how to make use of their knowledge. Coding has become some abstract concept to them similar to Math. They know the basics but they don’t see personally how coding can help them do anything cool. This is a bigger problem in Mathematics but this can occur in CS. These students don’t have a personal nor professional realtionship with programming. They want to go outside of Hello World or whatever pet assignments they are given in class. Some students don’t understand the beauty of data structures and might say “what’s so cool about arranging numbers” or “counting the number of times a letter occurs in a string”. For them, programming will “click” when they can apply what they learn to solve more personal real world problems, something they have a connection with. Creating websites and games are amazing for this if they cannot secure a job. Being able to visualize and interact with your program is such an amazing experience or having your code being used by someone. I started my programming journey with web programming back in middle school and for some reason it was such a cool experience. Seeing how my text written in notepad could be rendered into an actual webpage was such a fascinating sight. It’s as if I was partipcating in those blogs and personal sites from Angelfire and Geocities era. Though for me, this is when it “clicked” to me that programming wasn’t for some genius wizards and motivated me to go to the library, writing notes from programming book in Indigo [a Canadian Bookstore] (which is like a library if we are honest), and browsing Youtube back when there wasn’t many videos on programming.

Scripts are such a powerful tool as well, I’ve written scripts to help automate my tedious data entry job in the past, process and analyze volumes of data for different disciplines such as in geography and in helping someone with their linguistic research. Programming is such a powerful tool and for some, programming “clicks” when it becomes a useful tool.