01 — Inspect an EPW

Learning objectives: load all 35 EPW fields, distinguish expected unavailable fields from real data gaps, inspect provenance, understand hour-ending Local Standard Time, and make a labelled weather plot.

Prerequisite: run notebook 00 first. Network: none. Expected runtime: under one minute.

[ ]:

import hashlib
import json
import os
from pathlib import Path

import matplotlib.pyplot as plt
import pandas as pd

from smhi2epw import EPW_COLUMNS, read_epw

OUTPUT_DIR = Path("examples/output") if Path("examples").exists() else Path("output")
EPW_PATH = Path(os.environ.get("SMHI2EPW_EPW_PATH", OUTPUT_DIR / "gothenburg_2023.epw"))
if not EPW_PATH.exists():
    raise FileNotFoundError("Run 00_getting_started.ipynb before this notebook.")
weather = read_epw(EPW_PATH)
weather.shape

Headers and provenance

The reader keeps the original eight headers and parses the LOCATION record into DataFrame.attrs. Missing EPW sentinels are NaN by default, so pandas statistics do not mistake a sentinel such as 9999 for a measurement.

[ ]:

print(weather.attrs["location"])
print(*weather.attrs["header"], sep="\n")
assert tuple(weather.columns) == EPW_COLUMNS
weather.head(3)

Read the reconstruction receipt

Automatic compilation normally writes OUTPUT.epw.json. The receipt includes source fractions, warnings, code identity, consumed-response hashes, and an EPW checksum. A missing receipt or a checksum mismatch means the audit trail needs investigation. Headers alone do not describe replacement extent. Custom provenance_path settings may put the receipt elsewhere.

[ ]:

receipt_path = Path(str(EPW_PATH) + ".json")
if receipt_path.exists():
    receipt = json.loads(receipt_path.read_text(encoding="utf-8"))
    print("Receipt sections:", sorted(receipt))
    if receipt["epw_sha256"] != hashlib.sha256(EPW_PATH.read_bytes()).hexdigest():
        raise ValueError("Receipt checksum does not match this EPW.")
    display(receipt["result"]["report"])
else:
    print("No adjacent receipt; check the compilation policy and provenance path.")

Why are several columns 100% missing?

This is expected for a smhi2epw file. EPW has 35 fixed fields, but the selected SMHI MetObs and STRÅNG sources do not provide every secondary field. The compiler writes the official field-specific missing token instead of inventing values. read_epw() then displays those tokens as NaN.

The table of non-zero percentages can therefore look alarming even when all core thermal, moisture, pressure, wind, and solar inputs are complete. Illuminance, visibility, ceiling height, present-weather codes, precipitable water, aerosol, snow, albedo, and precipitation are currently expected to be unavailable. opaque_sky_cover is also unavailable because observed total cloud cover is not the same physical quantity as opaque cover.

total_sky_cover is optional. It can use observed SMHI clouds or ERA5 clouds at hours where ERA5 replaces required meteorology or GHI. Missing cloud data at other hours remain missing. A value of 100% missing for Gothenburg 2023 means no qualifying cloud series was available in that compilation; it does not mean the entire weather file is missing.

[ ]:

CORE_FIELDS = [
    "dry_bulb",
    "dew_point",
    "relative_humidity",
    "pressure",
    "horizontal_ir",
    "ghi",
    "dni",
    "dhi",
    "wind_direction",
    "wind_speed",
]
OPTIONAL_FIELDS = {"total_sky_cover"}

missing = (100 * weather.isna().mean()).sort_values(ascending=False)
missing_table = missing[missing > 0].rename("missing_percent").to_frame()
missing_table["meaning"] = [
    (
        "optional cloud data were unavailable"
        if field in OPTIONAL_FIELDS
        else "not supplied by the smhi2epw source set"
    )
    for field in missing_table.index
]
missing_table

[ ]:

core_missing = missing.loc[CORE_FIELDS].rename("missing_percent").to_frame()
display(core_missing)
if (core_missing.missing_percent > 0).any():
    raise ValueError("A core EPW field has missing hours; investigate this file.")
print("All core weather and solar fields are complete.")

Build interval-start timestamps

EPW hour 1 represents 00:00–01:00 and hour 24 represents 23:00–24:00. For analysis, construct an interval-start timestamp with hour - 1; do not parse hour 24 directly as a clock hour.

[ ]:

timestamp = pd.to_datetime(
    {"year": weather.year, "month": weather.month, "day": weather.day}
) + pd.to_timedelta(weather.hour - 1, unit="h")
week = weather.iloc[: 24 * 7]
fig, axes = plt.subplots(2, 1, figsize=(10, 6), sharex=True)
axes[0].plot(timestamp.iloc[: len(week)], week.dry_bulb, color="#0072B2")
axes[0].set_ylabel("Dry-bulb temperature (°C)")
axes[1].plot(timestamp.iloc[: len(week)], week.ghi, color="#E69F00")
axes[1].set_ylabel("GHI (W/m²)")
axes[1].set_xlabel("Local Standard Time")
fig.suptitle("First week of the Gothenburg AMY")
fig.tight_layout()

Key takeaways and exercise

A 100% value in the filtered missing-data table describes one optional EPW column, not the whole file. Always assess completeness field by field and prioritize the variables required by your downstream model. The calendar stays explicit because a TMY may use a different representative year for each month.

Try it: plot relative humidity and wind speed for the same week. Which variables vary smoothly, and which are bounded by definition? Then inspect whether total_sky_cover is present in an AMY for another year.