Version Checks (indicate both or one)
The Issue
Description
When attempting to build renewable profiles for the eastern interconnect using historical 2019 data, the workflow fails at the build_renewable_profiles step with exit status 1 but no Python traceback. Western interconnect works successfully with identical configuration.
Environment
Platform: Seawulf HPC (Stony Brook University)
OS: Rocky Linux 9.6
Python: 3.11.9
PyPSA: 0.30.2
PyPSA-USA: Repository-based installation (~/pypsa-usa/)
Conda Environment: pypsa-usa
Solver: Gurobi 13.0.1 (academic license)
Hardware: AMD EPYC (Milan) nodes, 257GB RAM, 20TB scratch space
yaml config:
`scenario:
interconnect: [eastern]
clusters: [30]
simpl: [75]
opts: [REM-3h]
ll: [v1.0]
sector: ""
planning_horizons: [2019]
renewable_weather_years: [2019]
snapshots:
start: "2019-01-01"
end: "2020-01-01"
inclusive: "left"
renewable_scenarios: ["historical"]`
Steps To Reproduce
Steps to Reproduce
Set up PyPSA-USA with eastern interconnect configuration
Configure for 2019 historical data (as shown above)
Run workflow: snakemake -j16 --configfile config/config.npcc_baseline.yaml
Observe failure at build_renewable_profiles step
Expected Behavior
Renewable profiles should build successfully for eastern interconnect, similar to how western interconnect completes without errors.
Error Message
Error Output
RuleException:
CalledProcessError in file /gpfs/home/glarangeira/pypsa-usa/workflow/rules/build_electricity.smk, line 235:
Command 'set -euo pipefail; /gpfs/home/glarangeira/.conda/envs/pypsa-usa/bin/python3.11 /gpfs/home/glarangeira/pypsa-usa/workflow/.snakemake/scripts/tmp*.build_renewable_profiles.py' returned non-zero exit status 1.
Profile Log Output
All three renewable profile logs (solar, onwind, offwind_floating) show:
INFO:__main__:Loading godeeep renewable data...
INFO:__main__:Using historical year: 2019 (from renewable_weather_years)
INFO:__main__:Using renewable snapshots for year 2019: 2019-01-01 to 2019-12-31 (inclusive end)
[Then crashes with no further output]
Behavior
Job completes in 6-8 minutes (too fast for actual processing)
No Python traceback in logs
Script exits immediately after logging "Loading godeeep renewable data"
All three renewable technologies fail (solar, onwind, offwind_floating)
Anything else?
Working Comparison: Western Interconnect
Configuration that WORKS:
yamlscenario:
interconnect: [western]
clusters: [33] # Required for western
planning_horizons: [2019]
renewable_weather_years: [2019]
snapshots:
start: "2019-01-01"
end: "2019-01-08" # 1 week test
inclusive: "left"
renewable_scenarios: ["historical"]
Result: Successfully completes in ~4 hours with optimization results
Troubleshooting Attempted
✅ Verified all config parameters point to 2019 historical data
✅ Confirmed renewable_scenarios: ["historical"] is set
✅ Cleared cache: rm -rf resources/NYPA_NPCC_Baseline/
✅ Increased memory from 64GB to 128GB
✅ Moved data to scratch space (20TB available)
✅ Verified Python 3.11.9 is active in conda environment
✅ Confirmed zenodo_downloader.py has solar_historical and wind_100m_historical records
✅ Checked GODEEEP data availability:
python scenario_records = {
"solar_historical": 18293999, # ✓ Available
"wind_100m_historical": 18331699, # ✓ Available
"solar_rcp85cooler_2020_2059": None, # Not available
# All future scenarios: None
}
✅ Ran snakemake --unlock
❌ Issue persists
Additional Context
Purpose
Attempting to run PyPSA-USA for NPCC (Northeast) region to identify marginal generators for battery charging analysis and environmental justice impact assessment.
Timeline
Western interconnect test: Successful (Feb 24, 2026)
Eastern interconnect attempts: Multiple failures (Feb 24-25, 2026)
Pattern: Consistent failure at same step across all attempts
Zenodo Data Verification
Checked scripts/zenodo_downloader.py and confirmed:
Only solar_historical (ID: 18293999) and wind_100m_historical (ID: 18331699) have actual Zenodo record IDs
All RCP future scenarios are None
Configuration is correctly using historical scenarios
System Resources
Home directory: 25GB quota (managed with symlinks to scratch)
Scratch space: 20TB available at /gpfs/scratch/$USER/
RAM per job: 128GB allocated
CPUs: 16 cores allocated
Questions
Is eastern interconnect renewable profile building a known limitation?
Does eastern interconnect require different configuration than western?
Are there specific data dependencies for eastern that aren't documented?
Can we use pre-downloaded GODEEEP profiles instead of building from scratch?
Is there a way to get more detailed error output from the build_renewable_profiles.py script?
Version Checks (indicate both or one)
This bug exists on the master branch of PyPSA-USA.
This bug exists on the develop branch of PyPSA-USA.
The Issue
Description
When attempting to build renewable profiles for the eastern interconnect using historical 2019 data, the workflow fails at the build_renewable_profiles step with exit status 1 but no Python traceback. Western interconnect works successfully with identical configuration.
Environment
Platform: Seawulf HPC (Stony Brook University)
OS: Rocky Linux 9.6
Python: 3.11.9
PyPSA: 0.30.2
PyPSA-USA: Repository-based installation (~/pypsa-usa/)
Conda Environment: pypsa-usa
Solver: Gurobi 13.0.1 (academic license)
Hardware: AMD EPYC (Milan) nodes, 257GB RAM, 20TB scratch space
yaml config:
`scenario:
interconnect: [eastern]
clusters: [30]
simpl: [75]
opts: [REM-3h]
ll: [v1.0]
sector: ""
planning_horizons: [2019]
renewable_weather_years: [2019]
snapshots:
start: "2019-01-01"
end: "2020-01-01"
inclusive: "left"
renewable_scenarios: ["historical"]`
Steps To Reproduce
Steps to Reproduce
Set up PyPSA-USA with eastern interconnect configuration
Configure for 2019 historical data (as shown above)
Run workflow: snakemake -j16 --configfile config/config.npcc_baseline.yaml
Observe failure at build_renewable_profiles step
Expected Behavior
Renewable profiles should build successfully for eastern interconnect, similar to how western interconnect completes without errors.
Error Message
Anything else?
Working Comparison: Western Interconnect
Configuration that WORKS:
yamlscenario:
interconnect: [western]
clusters: [33] # Required for western
planning_horizons: [2019]
renewable_weather_years: [2019]
snapshots:
start: "2019-01-01"
end: "2019-01-08" # 1 week test
inclusive: "left"
renewable_scenarios: ["historical"]
Result: Successfully completes in ~4 hours with optimization results
Troubleshooting Attempted
✅ Verified all config parameters point to 2019 historical data
✅ Confirmed renewable_scenarios: ["historical"] is set
✅ Cleared cache: rm -rf resources/NYPA_NPCC_Baseline/
✅ Increased memory from 64GB to 128GB
✅ Moved data to scratch space (20TB available)
✅ Verified Python 3.11.9 is active in conda environment
✅ Confirmed zenodo_downloader.py has solar_historical and wind_100m_historical records
✅ Checked GODEEEP data availability:
python scenario_records = {
"solar_historical": 18293999, # ✓ Available
"wind_100m_historical": 18331699, # ✓ Available
"solar_rcp85cooler_2020_2059": None, # Not available
# All future scenarios: None
}
✅ Ran snakemake --unlock
❌ Issue persists
Additional Context
Purpose
Attempting to run PyPSA-USA for NPCC (Northeast) region to identify marginal generators for battery charging analysis and environmental justice impact assessment.
Timeline
Western interconnect test: Successful (Feb 24, 2026)
Eastern interconnect attempts: Multiple failures (Feb 24-25, 2026)
Pattern: Consistent failure at same step across all attempts
Zenodo Data Verification
Checked scripts/zenodo_downloader.py and confirmed:
Only solar_historical (ID: 18293999) and wind_100m_historical (ID: 18331699) have actual Zenodo record IDs
All RCP future scenarios are None
Configuration is correctly using historical scenarios
System Resources
Home directory: 25GB quota (managed with symlinks to scratch)
Scratch space: 20TB available at /gpfs/scratch/$USER/
RAM per job: 128GB allocated
CPUs: 16 cores allocated
Questions
Is eastern interconnect renewable profile building a known limitation?
Does eastern interconnect require different configuration than western?
Are there specific data dependencies for eastern that aren't documented?
Can we use pre-downloaded GODEEEP profiles instead of building from scratch?
Is there a way to get more detailed error output from the build_renewable_profiles.py script?