Spectral Data Preprocessing Methods

Spectral Data Preprocessing Methods

Overview

The raw spectra acquired by a spectrometer contain not only the sample's own absorption information but also superimposed interference signals such as baseline drift, high-frequency noise and scattering effects. Before building quantitative or qualitative analysis models, appropriately preprocessing the raw spectra is a key step for improving model prediction accuracy and robustness. This article introduces the three most commonly used preprocessing methods in near-infrared and fiber-optic spectral analysis: smoothing denoising, derivative processing and scatter correction.

1. Smoothing Denoising

1.1 Moving Average Smoothing

The moving average is the simplest and most intuitive smoothing method. For each pixel $i$ in the spectrum, it takes the intensity values of $p$ neighboring pixels on each side and averages them:

$$I_s[i] = \frac{\sum_{j=\max(0, i-p)}^{\min(N-1, i+p)} I[j]}{\min(N-1, i+p) - \max(0, i-p) + 1}$$

Parameter description:
- $i$: current pixel index, $0 \le i < N$
- $p$: smoothing window radius (number of smoothing passes); the larger $p$ is, the stronger the smoothing effect
- $N$: total number of spectrum pixels (e.g. 2048)

Edge handling: When a pixel lies at either end of the spectrum ($i - p < 0$ or $i + p \ge N$), the window range is automatically reduced to the valid boundary, and the denominator is correspondingly adjusted to the number of pixels actually involved in the calculation.

1.2 Savitzky-Golay Smoothing

SG smoothing is an improvement over the moving average. Instead of simply averaging, it fits a polynomial within each sliding window and replaces the original value with the fitted value. SG smoothing is superior to the ordinary moving average in preserving the spectral peak shape (peak position, peak height and peak width), and is the most recommended smoothing method in near-infrared spectral analysis.

Typical parameters: window width 7-15 points, polynomial order 2-3.

1.3 Choosing Smoothing Parameters

Tip: Start with a smaller window parameter and increase it gradually, until the noise is visibly reduced on the spectral curve while the peak shape is not noticeably distorted.

2. Derivative Processing

2.1 First Derivative

The first-derivative spectrum eliminates the constant baseline offset (i.e. the additive baseline) in the spectrum, making absorption peaks superimposed on a gentle background more prominent.

Mathematical principle: The derivative of a constant term is zero, so wavelength-independent baseline offsets are completely eliminated in the first derivative.

2.2 Second Derivative

The second derivative further removes linear baseline drift and enhances the resolution of spectral peaks — originally overlapping broad peaks may be separated into independent negative peaks in the second-derivative spectrum.

Typical applications:
- Eliminating the overall vertical shift of spectra caused by differences in sample particle size
- Resolving overlapping absorption peaks
- Improving the prediction accuracy of PLS models

2.3 Precautions

Derivative processing amplifies high-frequency noise. It is usually applied after SG smoothing, or a one-step SG convolution derivative method is used to perform smoothing and differentiation simultaneously.

3. Scatter Correction

In the diffuse reflectance spectra of solid powders or granular samples, physical factors such as particle size and packing density cause an overall shift and tilt of the spectrum — the so-called scattering effect. These variations are unrelated to analyte concentration but can seriously interfere with modeling.

3.1 Standard Normal Variate (SNV)

SNV processes each spectrum independently:

$$x_{SNV} = \frac{x - \bar{x}}{\sigma}$$

where $\bar{x}$ is the mean of all wavelength points of that spectrum, and $\sigma$ is the standard deviation. The processed spectrum has a mean of 0 and a variance of 1, eliminating the overall vertical shift of the spectrum.

Applicable scenarios: Eliminating baseline shifts caused by particle-size differences; commonly used for preprocessing near-infrared diffuse reflectance spectra.

3.2 Multiplicative Scatter Correction (MSC)

MSC assumes that the scattering effects of all samples can be approximated by a linear transformation of an "ideal spectrum" (usually the mean spectrum of the calibration set). For each sample spectrum $x$, a regression is established:

$$x = a + b \cdot x_{ref}$$

The corrected spectrum is:

$$x_{MSC} = \frac{x - a}{b}$$

where $a$ (intercept) corrects the additive scattering (baseline shift) and $b$ (slope) corrects the multiplicative scattering (optical path variation).

Comparison between MSC and SNV:

4. Combined Preprocessing Strategies

In practice, multiple preprocessing methods are usually combined to achieve the best modeling results:

5. Evaluating Preprocessing Effectiveness

Whether preprocessing is effective must be verified through subsequent modeling. Evaluation metrics include:

• RMSEC (root mean square error of calibration): prediction error on the calibration set
• RMSEP (root mean square error of prediction): prediction error on an independent validation set
• $R^2$ (coefficient of determination): the closer to 1, the better

Core principle: The purpose of preprocessing is not to make the spectrum "look good", but to make the model predict more accurately. A reduction in RMSEP is the final criterion for evaluating preprocessing effectiveness.

This article was compiled by Pynect.