06 — Data quality and gap filling

Learning objectives: visualize the 3/4/48/49-hour boundaries, distinguish linear and daily-profile estimates, understand optional missing data, and interpolate wind direction circularly.

Network: none. Expected runtime: under one minute. Synthetic periodic data makes the method visible without claiming to reproduce a real weather event.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

from smhi2epw import processing
from smhi2epw.errors import DataGapError

index = pd.date_range("2023-06-01", periods=24 * 16, freq="h", tz="UTC")
hour = np.arange(len(index))
baseline = 16 + 6 * np.sin(2 * np.pi * (hour - 7) / 24) + 0.03 * hour
base = pd.Series(baseline, index=index, name="dry_bulb")

def fill_example(length, start=120, required=True):
    """Insert one gap and return its frame and processing report."""
    frame = pd.DataFrame({"dry_bulb": base.copy()})
    frame.iloc[start : start + length, 0] = np.nan
    report = processing.impute(
        frame,
        ["dry_bulb"] if required else [],
        optional=[] if required else ["dry_bulb"],
    )
    return frame, report


examples = {length: fill_example(length) for length in (3, 4, 48)}
pd.DataFrame(
    {
        length: {
            "linear_hours": report.linear_filled_hours["dry_bulb"],
            "diurnal_hours": report.diurnal_filled_hours["dry_bulb"],
            "maximum_gap": report.max_gap_hours["dry_bulb"],
        }
        for length, (_, report) in examples.items()
    }
).T

fig, axes = plt.subplots(3, 1, figsize=(10, 8), sharex=True)
window = slice(95, 195)
for ax, (length, (frame, _)) in zip(axes, examples.items()):
    ax.plot(base.iloc[window].index, base.iloc[window], label="known synthetic truth")
    ax.plot(
        frame.iloc[window].index,
        frame.dry_bulb.iloc[window],
        linestyle="--",
        label=f"filled {length} h",
    )
    ax.set_ylabel("Temperature (°C)")
    ax.legend()
axes[-1].set_xlabel("UTC timestamp")
fig.tight_layout()

The temporal reconstruction ceiling

The low-level processing.impute() helper raises for a required 49-hour gap and leaves the same optional gap as NaN. The 48-hour limit is a bounded engineering policy, not a universal accuracy threshold. Automatic compile_epw() can recover unresolved required gaps with assessed same-year donors or ERA5 without extending temporal filling beyond this ceiling. This lesson isolates temporal filling; it does not demonstrate the entire automatic pipeline.

try:
    fill_example(49)
except DataGapError as error:
    print("Required field:", error)
optional_frame, optional_report = fill_example(49, required=False)
print("Optional hours still missing:", int(optional_frame.dry_bulb.isna().sum()))

wind = pd.DataFrame({"wind_direction": [359.0, np.nan, 1.0]})
processing.impute(wind, ["wind_direction"])
wind  # midpoint is near north (0°), not south (180°)

Key takeaways and exercise

Linear filling is local and smooth; daily profiles preserve a diurnal shape. Both are estimates, so reports quantify their use.

Try it: move a 12-hour gap to the beginning of the series. Which one-sided reference is available, and what happens if you remove that entire reference day?