added a noise filtering notebook with a bunch of prose, no code yet

Author: pavelkomarov

notebooks/README.md

@@ -10,5 +10,6 @@
 | [DCT Types Extension Experiments](https://github.com/pavelkomarov/spectral-derivatives/blob/main/notebooks/dct_types_extension_experiments.ipynb) | This is where I worked out exactly how to take derivatives with the DCT-II/III and DST-III rather than the DCT-I and DST-I and poked around to find limitations. The correct path now lives in [the math](https://pavelkomarov.com/spectral-derivatives/math.pdf), but the intuition of the limitations best given by example here.|
 | [DFT, DCT, DST Relationship](https://github.com/pavelkomarov/spectral-derivatives/blob/main/notebooks/dft_dct_dst_relationship.ipynb) | Explicit formulae and code examples to show how the coefficients of DCT and DST variants are related to DFT coefficients. A mathematical derivation of a couple of these relationships lives in [the math](https://pavelkomarov.com/spectral-derivatives/math.pdf), but it's useful to see others too and have a quick reference.|
 | [DFT Play](https://github.com/pavelkomarov/spectral-derivatives/blob/main/notebooks/dft_play.ipynb) | Due to aliasing, it's possible to find the DFT coefficients with many different combinations of wavenumbers, so long as they cover $\{0, ...N\}$ modulo $N$. Here lies some code to demonstrate this fact.|
+| [Filtering Noise](https://github.com/pavelkomarov/spectral-derivatives/blob/main/notebooks/filtering_noise.ipynb) | |
 | [Fourier](https://github.com/pavelkomarov/spectral-derivatives/blob/main/notebooks/fourier.ipynb) | Usage examples of `fourier_deriv`, in one and multiple dimensions, with an emphasis of when this method will work and when it will break.|
 | [Fourier Periodic Extensions](https://github.com/pavelkomarov/spectral-derivatives/blob/main/notebooks/fourier_periodic_extensions.ipynb) | It's sometimes possible to take even and odd extensions of a function and produce something periodic or nearly periodic, which can then be spectrally differentiated with the Fourier basis. This notebook demonstrates how to do this.|

notebooks/filtering_noise.ipynb

@@ -0,0 +1,85 @@
+{
+ "cells": [
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "4b25a1a6-3a0f-476d-a936-fc3253114079",
+   "metadata": {},
+   "source": [
+    "# Dealing with Noise\n",
+    "\n",
+    "Taking derivatives is particularly challenging in the presence of noise, because the peaky, pointy nature of random variations can cause dramamtic local deviations of function slope.\n",
+    "\n",
+    "## Noise and Signal are Typically Frequency-Separable\n",
+    "\n",
+    "> \"Every spectrum of real noise falls off reasonably rapidly as you go to infinite frequencies, or else it would have infinite energy. But the sampling process aliases higher frequencies in lower ones, and the folding ... tends to produce a flat specturm. ... *white noise*. The signal, usually, is mainly in the lower frequencies.\"  \n",
+    "--Richard Hamming, *The Art of Doing Science and Engineering*, Digital Filters III\n",
+    "\n",
+    "We expect our sampling rate to be fast enough to capture fluctuations of interest in the data, e.g. the Nyquist rate (2x the highest frequency of interest in the data), so most signal energy is expressed by relatively low frequencies. Noise causes fluctuations in each sample, tying it to the sampling rate: The more we sample, the more we can smear out the noise's spectrum across frequencies, hopefully putting most of its mass at *higher frequency* than the data. Thus we can separate data from noise in the frequency domain!\n",
+    "\n",
+    "*All noise reduction methods are at bottom low-pass filters*: averaging, Kalman filtering, etc.\n",
+    "\n",
+    "## Filtering with the Fourier Basis\n",
+    "\n",
+    "A classic FIR or IIR low-pass filter from Signal Processing, like a Butterworth Filter, works from one end of the signal toward the other \"causally\", dampening higher-frequencies with only *local*, past and present samples, literally by taking some weighted combination of a few local input values and adding a weighted combination of a few past output values.\n",
+    "\n",
+    "But if we have the whole history of a signal, then there is no need to constrain ourselves locally; we can transform the entire thing to a Fourier basis representation, where modes correspond to frequencies. Hence a common noise removal strategy is:\n",
+    "\n",
+    "1. FFT the signal\n",
+    "2. Zero out higher Fourier modes/coefficients\n",
+    "3. IFFT to recover the filtered signal\n",
+    "\n",
+    "This achieves an *idealized* lowpass filter, where we get perfect cutoff, as opposed to a causal filter where we instead get power rolloff of, e.g., 20dB/decade.\n",
+    "\n",
+    "## Connection to Error Correcting Codes\n",
+    "\n",
+    "Because a spectral representation builds a function out of basis functions that span the entire domain, every point takes the entire domain under consideration. This makes the reconstruction much more *robust* to perturbations than one that uses only a few neighboring points. I.e. it's much harder to *corrupt* the signal so thoroughly that it can't be successfully recovered.\n",
+    "\n",
+    "This has an analog to [error-correcting codes](https://www.youtube.com/watch?v=X8jsijhllIA), except that in the context of continuous signals, \"corruption\" means (discrete representations of) continous numbers have slightly inaccurate values, whereas in error correcting codes each datum is a *single discrete bit* which can simply be *flipped*. But notice the bits corresponding to each subsequent parity check of a Hamming Code correspond to \"higher-frequency\" selections:\n",
+    "\n",
+    "<img src=\"hamming.png\" width=500 />\n",
+    "\n",
+    "From Richard Hamming's book, *The Art of Doing Science and Engineering*, with my drawings in the margins.\n",
+    "\n",
+    "In a spectral method, the coefficients say \"You need to add in this much of the $k^{th}$ basis function, but in error correcting codes the analogous parity checks say \"There is/isn't a parity error among my bits.\" In both cases a spot-corruption will stand out, because it appears in a particular combination of parity checks or introduces a value that can't be as easily/smoothly represented with a finite combination of basis functions.\n",
+    "\n",
+    "## Filtering with More Exotic Bases\n",
+    "\n",
+    "To avert Gibbs phenomenon or achieve better compression (more energy represented in fewer modes), we may prefer to use a basis of Chebyshev polynomials, PCA modes via SVD, wavelets, etc. to represent a signal.\n",
+    "\n",
+    "All bases have \"lower frequency\" and \"higher frequency\" elements, meaning some modes have fewer transitions between low and high values, and some have more. As in Fourier basis representations, signal energy empirically tends to cluster in lower modes, and noise tends to be scattered across modes.\n",
+    "\n",
+    "However, we have to be a bit more conscientious when using these modes to filter noise: If a basis function is higher-frequency over part of its domain, as Chebyshev polynomials are toward the edges of $[-1, 1]$, then that basis function is better at representing high-frequency noise in those regions, and we don't get band-separation quite as cleanly as with a uniform basis like Fourier."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "34931a9d-dd64-4867-8c71-6a910730c98e",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}