08 — Reproducible batch generation
Learning objectives: compile a small location/year matrix safely, reuse the cache, handle expected failures, create deterministic filenames, and save a provenance summary.
Network: SMHI and, if recovery needs it, Open-Meteo. Expected runtime: depends on the matrix; begin with one year. Batch work increases API load, so preserve .smhi_cache and avoid unnecessary refreshes.
[ ]:
import json
import os
from datetime import datetime, timezone
from pathlib import Path
import pandas as pd
import smhi2epw
from smhi2epw import EPWConfig, Smhi2EpwError, compile_epw
OUTPUT_DIR = Path("examples/output") if Path("examples").exists() else Path("output")
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
LOCATIONS = {
"gothenburg": (57.7089, 11.9746),
"stockholm": (59.3293, 18.0686),
}
YEARS = [2023] # expand only after the small run succeeds
Read the saved completeness runs without network access
The included examples/data/completeness_2026-10-05.json is a compact summary of the frozen EPSM national municipality/year jobs (2017–2024), not all Swedish stations. It contains the original artifact hashes and the current output checks; large EPWs and raw caches are kept outside the example package. Run the next cells to inspect the evidence without fetching weather.
V1/v2 counts are historical manifests, whose EPWs are no longer locally available. V3 uses 5930c07 and all its outputs were independently checked. The v1 denominator is 85; preparation expanded v2/v3 to the same 128 jobs. Compare policy and denominator before interpreting completion percentages.
[ ]:
example_dir = Path(
os.environ.get(
"SMHI2EPW_EXAMPLES_DIR", "examples" if Path("examples").is_dir() else "."
)
)
evidence_path = example_dir / "data/completeness_2026-10-05.json"
evidence = json.loads(evidence_path.read_text(encoding="utf-8"))
completion = pd.DataFrame(
{
"run": f"v{number}",
"policy": stage["policy"],
"source_commit": stage["smhi2epw_git_commit"],
"requested": stage["requested_weather_jobs"],
"completed": stage["completed_weather_jobs"],
"failed": stage["failed_weather_jobs"],
}
for number, stage in enumerate(evidence["stages"], start=1)
)
completion["completion_percent"] = 100 * completion.completed / completion.requested
completion
[ ]:
audit = evidence["stages"][-1]["current_output_audit"]
display(pd.Series(audit["jobs_using_source"]["required_meteorology"], name="jobs"))
display(pd.Series(audit["jobs_with_warning_code"], name="jobs_with_warning"))
display(pd.Series(audit["required_reconstructed_fraction"], name="reconstruction"))
source_hours = pd.DataFrame(audit["exported_hours_by_variable_source"]).fillna(0)
source_percent = 100 * source_hours.div(source_hours.sum(axis=0), axis=1)
source_percent.round(3)
All 128 current jobs completed and passed calendar/hash/required-field checks, but all are mixed_reconstructed: 101 use donor meteorology, 35 use ERA5 and 126 use temporal filling. These groups overlap; do not add them. Twenty-five exceed 5% reconstructed required meteorological cells; one reconstructs all five meteorological variables while retaining primary solar inputs. This fraction counts buffered ingestion cells and excludes GHI; the source-share table counts exported hours separately
for each variable.
Five percent is a reporting aid, not a validated acceptance threshold. Complete EPWs do not establish local weather accuracy or preserve every extreme. Inspect warnings and overlap diagnostics, and assess sensitivity for heavily reconstructed cases before calibration. The archived files predate the solar and actual-year header corrections; regenerate them for the current EnergyPlus gate. The evidence-reading cells below stay offline; the later batch cells contact providers.
Read the corrected-source sensitivity check offline
The 6 October check rebuilt references after the solar corrections and ran nine annual simulations for three weather-years and one illustrative 100 m² building. All completed with zero warnings. The table shows signed annual sensible-cooling changes and temperature error in the masked hours; it does not measure HVAC electricity.
The 168-hour solar outages used daily profiles after physical nighttime zeros split the missing daylight blocks. The 48-hour meteorology outages used assessed fixed donors and temporal recovery for rejected wind direction. None of these six cases selected ERA5. The references include modelled solar radiation; these are conditional paired results, not site-accuracy validation or national uncertainty bounds.
[ ]:
sensitivity = json.loads(
(example_dir / "data/solar_validation_2026-10-06.json").read_text()
)
checks = sensitivity["validation"]
assert checks["total_engine_warnings"] == 0
assert checks["all_annual_calendars_complete_and_ordered"]
rows = []
for weather_year in sensitivity["weather_years"]:
for comparison in weather_year["comparisons"]:
cooling = comparison["load_errors"]["cooling"]
temperature = comparison["weather_errors"].get("dry_bulb", {})
rows.append(
{
"weather_year": weather_year["name"],
"outage": comparison["case"],
"cooling_change_kWh_m2": cooling["signed_annual_error_kwh_m2"],
"cooling_change_percent": cooling["signed_annual_error_percent"],
"hidden_temperature_MAE_C": temperature.get("mae"),
}
)
pd.DataFrame(rows).set_index(["weather_year", "outage"])
[ ]:
records = []
for city, (latitude, longitude) in LOCATIONS.items():
for year in YEARS:
path = OUTPUT_DIR / f"{city}_{year}.epw"
try:
result = compile_epw(
EPWConfig(
year=year,
output_path=str(path),
city=city.title(),
latitude=latitude,
longitude=longitude,
utc_offset=1,
cache_dir=".smhi_cache",
refresh=False,
weather_policy="automatic",
provenance_path=str(path) + ".json",
)
)
except Smhi2EpwError as error:
records.append(
{"city": city, "year": year, "status": "failed", "error": str(error)}
)
continue
records.append(
{
"city": city,
"year": year,
"status": "ok",
"path": str(path),
"rows": result.rows,
"station_id": result.station.station_id if result.station else None,
"station_distance_km": result.coordinate_distance_km,
"solar_source": result.report.solar_source,
"interpolated_fraction": result.interpolated_fraction,
"weather_policy": "automatic",
"weather_classification": result.report.weather_classification,
"required_reconstructed_fraction": result.report.required_reconstructed_fraction,
"source_fractions": json.dumps(
result.report.source_fractions, sort_keys=True
),
"donor_filled_hours": json.dumps(
result.report.cross_station_filled_hours, sort_keys=True
),
"reanalysis_filled_hours": json.dumps(
result.report.reanalysis_filled_hours, sort_keys=True
),
"warning_codes": ";".join(
sorted({w["code"] for w in result.report.warnings})
),
"warnings": json.dumps(result.report.warnings, sort_keys=True),
"provenance_path": (
str(path) + ".json"
if not any(
w["code"] == "provenance_write_failed"
for w in result.report.warnings
)
else None
),
"compiled_at_utc": datetime.now(timezone.utc).isoformat(),
"package_version": smhi2epw.__version__,
}
)
summary = pd.DataFrame(records)
summary
[ ]:
manifest = OUTPUT_DIR / "batch_manifest.csv"
summary.to_csv(manifest, index=False)
manifest.resolve()
Safe batch practices
Do not use except Exception to hide programming errors. Catch Smhi2EpwError, record the failure, and leave the missing output visible. Keep configuration, diagnostics, package version, compilation time, and the JSON receipt with simulation inputs. The receipt carries actual response hashes and code identity. interpolated_fraction excludes donors and ERA5. required_reconstructed_fraction counts missing/invalid primary cells across five required meteorological variables on the
buffered UTC grid; it excludes GHI. Per-variable filled-hour counts can include buffer hours and optional clouds, so do not sum them as unique exported recovered hours. source_fractions describes exported-hour meteorology and GHI. A provenance_write_failed warning leaves no verified current receipt. Atomic output means a failed item cannot leave a partial EPW under its final name.
Key takeaways and exercise
Caching is both a performance feature and responsible API behavior. Deterministic filenames and a manifest make downstream simulation traceable.
Try it: add 2020 and sort successful jobs by required reconstructed fraction and review their source shares and warnings. Before scaling further, estimate the number of API requests and decide whether the scientific question needs every location/year combination.