Time Series Clustering with cluster()¶
Accelerate investment optimization using typical periods (clustering).
This notebook demonstrates:
- Typical periods: Cluster similar time segments (e.g., days) and solve only representative ones
- Weighted costs: Automatically weight operational costs by cluster occurrence
- Two-stage workflow: Fast sizing with clustering, accurate dispatch at full resolution
- Segmentation: Reduce timesteps within each cluster for further compression
!!! note "Requirements" This notebook requires the tsam and tsam_xarray packages. Install with: pip install "flixopt[full]"
!!! tip "tsam_xarray" flixopt uses tsam_xarray for clustering, which wraps tsam. For advanced clustering options (custom algorithms, weights, tuning), see the tsam_xarray documentation.
import timeit
import pandas as pd
import xarray as xr
import flixopt as fx
fx.CONFIG.notebook()
flixopt.config.CONFIG
Create the FlowSystem¶
We use a district heating system with real-world time series data (one month at 15-min resolution):
flow_system = fx.tutorials.load_example('district_heating')
flow_system.connect_and_transform()
timesteps = flow_system.timesteps
flow_system
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm
FlowSystem ========== Timesteps: 744 (irregular) [2020-01-01 to 2020-01-31] Periods: None Scenarios: None Status: ✓ Components (9 items) -------------------- * Boiler * CHP * CoalSupply * ElecDemand * GasGrid * GridBuy * GridSell * HeatDemand * Storage Buses (4 items) --------------- * Coal * Electricity * Gas * Heat Effects (2 items) ----------------- * CO2 * costs Flows (13 items) ---------------- * Boiler(Q_fu) * Boiler(Q_th) * CHP(P_el) * CHP(Q_fu) * CHP(Q_th) * CoalSupply(Q_Coal) * ElecDemand(P_el) * GasGrid(Q_Gas) * GridBuy(P_el) * GridSell(P_el) ... (+3 more)
# Visualize input data
input_ds = xr.Dataset(
{
'Heat Demand': flow_system.components['HeatDemand'].inputs[0].fixed_relative_profile,
'Electricity Price': flow_system.components['GridBuy'].outputs[0].effects_per_flow_hour['costs'],
}
)
input_ds.plotly.line(x='time', facet_row='variable', title='One Month of Input Data')
Method 1: Full Optimization (Baseline)¶
First, solve the complete problem with all 2976 timesteps:
solver = fx.solvers.HighsSolver(mip_gap=0.01)
start = timeit.default_timer()
fs_full = flow_system.copy()
fs_full.name = 'Full Optimization'
fs_full.optimize(solver)
time_full = timeit.default_timer() - start
Writing constraints.: 0%| | 0/70 [00:00<?, ?it/s]
Writing constraints.: 36%|███▌ | 25/70 [00:00<00:00, 242.17it/s]
Writing constraints.: 73%|███████▎ | 51/70 [00:00<00:00, 249.14it/s]
Writing constraints.: 100%|██████████| 70/70 [00:00<00:00, 244.53it/s]
Writing continuous variables.: 0%| | 0/57 [00:00<?, ?it/s]
Writing continuous variables.: 89%|████████▉ | 51/57 [00:00<00:00, 504.42it/s]
Writing continuous variables.: 100%|██████████| 57/57 [00:00<00:00, 495.59it/s]
Writing binary variables.: 0%| | 0/7 [00:00<?, ?it/s]
Writing binary variables.: 100%|██████████| 7/7 [00:00<00:00, 547.92it/s] Method 2: Clustering with cluster()¶
The cluster() method:
- Clusters similar days using the TSAM (Time Series Aggregation Module) package
- Reduces timesteps to only typical periods (e.g., 8 typical days = 768 timesteps)
- Weights costs by how many original days each typical day represents
- Handles storage with configurable behavior via
storage_mode
!!! warning "Peak Forcing" Always use extremes=ExtremeConfig(max_value=[...]) to ensure extreme demand days are captured. Without this, clustering may miss peak periods, causing undersized components.
!!! note "Which variables get clustered?" By default, all time-varying inputs influence cluster assignments with equal weight (1.0). To see which variables cluster() will feed to tsam, call list(flow_system.transform.cluster_inputs()) — it lists every variable with a time dim (constants included).
To restrict clustering to a subset, pass explicit weights via
`ClusterConfig(weights={...})`. Variables listed with weight `0` are still
aggregated but don't influence cluster assignments; variables not listed
keep the default weight of `1.0`. Example:
```python
from tsam import ClusterConfig
cols = list(flow_system.transform.cluster_inputs())
target = 'HeatDemand(Q_th)|fixed_relative_profile'
weights = {target: 1, **{v: 0 for v in cols if v != target}}
fs_clustered = flow_system.transform.cluster(
n_clusters=8, cluster_duration='1D',
cluster=ClusterConfig(weights=weights),
extremes=ExtremeConfig(method='new_cluster', max_value=[target]),
)
``` from tsam import ExtremeConfig
start = timeit.default_timer()
# IMPORTANT: Force inclusion of peak demand periods!
peak_series = ['HeatDemand(Q_th)|fixed_relative_profile']
# Create reduced FlowSystem with 8 typical days
fs_clustered = flow_system.transform.cluster(
n_clusters=8, # 8 typical days
cluster_duration='1D', # Daily clustering
extremes=ExtremeConfig(method='new_cluster', max_value=peak_series), # Capture peak demand day
)
fs_clustered.name = 'Clustered (8 days)'
time_clustering = timeit.default_timer() - start
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/tsam_xarray/_core.py:432: FutureWarning: aggregate() result columns are sorted alphabetically in v3 but will follow the input DataFrame's order in v4; your columns are not alphabetical, so this output will change. To keep the v3 order, sort before (aggregate(data.sort_index(axis=1), ...)) or after (result.cluster_representatives.sort_index(axis=1)); to adopt v4 now, index results by column name, not position. To silence this warning: warnings.filterwarnings('ignore', category=FutureWarning, message='.*sorted alphabetically.*').
tsam_result = tsam.aggregate(
# Optimize the reduced system
start = timeit.default_timer()
fs_clustered.optimize(solver)
time_clustered = timeit.default_timer() - start
Writing constraints.: 0%| | 0/79 [00:00<?, ?it/s]
Writing constraints.: 38%|███▊ | 30/79 [00:00<00:00, 293.25it/s]
Writing constraints.: 76%|███████▌ | 60/79 [00:00<00:00, 286.89it/s]
Writing constraints.: 100%|██████████| 79/79 [00:00<00:00, 284.67it/s]
Writing continuous variables.: 0%| | 0/58 [00:00<?, ?it/s]
Writing continuous variables.: 93%|█████████▎| 54/58 [00:00<00:00, 536.64it/s]
Writing continuous variables.: 100%|██████████| 58/58 [00:00<00:00, 529.51it/s]
Writing binary variables.: 0%| | 0/7 [00:00<?, ?it/s]
Writing binary variables.: 100%|██████████| 7/7 [00:00<00:00, 569.65it/s] Understanding the Clustering¶
Access clustering metadata via fs.clustering. For full access to the underlying tsam_xarray ClusteringResult, use fs.clustering.clustering_result.
# Clustering overview
fs_clustered.clustering
Clustering( 31 periods → 9 clusters timesteps_per_cluster=24 dims=[] )
Apply Existing Clustering¶
When comparing design variants or performing sensitivity analysis, you often want to use the same cluster structure across different FlowSystem configurations. Use apply_clustering() to reuse a clustering from another FlowSystem:
# First, create a reference clustering
fs_reference = flow_system.transform.cluster(n_clusters=8, cluster_duration='1D')
# Modify the FlowSystem (e.g., different storage size)
flow_system_modified = flow_system.copy()
flow_system_modified.components['Storage'].capacity_in_flow_hours.maximum_size = 2000
# Apply the SAME clustering for fair comparison
fs_modified = flow_system_modified.transform.apply_clustering(fs_reference.clustering)
This ensures both systems use identical typical periods for fair comparison.
Method 3: Two-Stage Workflow (Recommended)¶
The recommended approach for investment optimization:
- Stage 1: Fast sizing with
cluster() - Stage 2: Fix sizes (with safety margin) and dispatch at full resolution
!!! tip "Safety Margin" Typical periods aggregate similar days, so individual days may have higher demand than the typical day. Adding a 5-10% margin ensures feasibility.
# Apply safety margin to sizes
SAFETY_MARGIN = 1.05 # 5% buffer
sizes_with_margin = {name: float(size.item()) * SAFETY_MARGIN for name, size in fs_clustered.stats.sizes.items()}
# Stage 2: Fix sizes and optimize at full resolution
start = timeit.default_timer()
fs_dispatch = flow_system.transform.fix_sizes(sizes_with_margin)
fs_dispatch.name = 'Two-Stage'
fs_dispatch.optimize(solver)
time_dispatch = timeit.default_timer() - start
# Total two-stage time
total_two_stage = time_clustering + time_clustered + time_dispatch
Writing constraints.: 0%| | 0/65 [00:00<?, ?it/s]
Writing constraints.: 45%|████▍ | 29/65 [00:00<00:00, 284.10it/s]
Writing constraints.: 89%|████████▉ | 58/65 [00:00<00:00, 274.17it/s]
Writing constraints.: 100%|██████████| 65/65 [00:00<00:00, 272.67it/s]
Writing continuous variables.: 0%| | 0/57 [00:00<?, ?it/s]
Writing continuous variables.: 96%|█████████▋| 55/57 [00:00<00:00, 549.52it/s]
Writing continuous variables.: 100%|██████████| 57/57 [00:00<00:00, 543.26it/s]
Writing binary variables.: 0%| | 0/5 [00:00<?, ?it/s]
Writing binary variables.: 100%|██████████| 5/5 [00:00<00:00, 554.04it/s] Compare Results¶
results = {
'Full (baseline)': {
'Time [s]': time_full,
'Cost [€]': fs_full.solution['costs'].item(),
'CHP': fs_full.stats.sizes['CHP(Q_th)'].item(),
'Boiler': fs_full.stats.sizes['Boiler(Q_th)'].item(),
'Storage': fs_full.stats.sizes['Storage'].item(),
},
'Clustered (8 days)': {
'Time [s]': time_clustering + time_clustered,
'Cost [€]': fs_clustered.solution['costs'].item(),
'CHP': fs_clustered.stats.sizes['CHP(Q_th)'].item(),
'Boiler': fs_clustered.stats.sizes['Boiler(Q_th)'].item(),
'Storage': fs_clustered.stats.sizes['Storage'].item(),
},
'Two-Stage': {
'Time [s]': total_two_stage,
'Cost [€]': fs_dispatch.solution['costs'].item(),
'CHP': sizes_with_margin['CHP(Q_th)'],
'Boiler': sizes_with_margin['Boiler(Q_th)'],
'Storage': sizes_with_margin['Storage'],
},
}
comparison = pd.DataFrame(results).T
baseline_cost = comparison.loc['Full (baseline)', 'Cost [€]']
baseline_time = comparison.loc['Full (baseline)', 'Time [s]']
comparison['Cost Gap [%]'] = ((comparison['Cost [€]'] - baseline_cost) / abs(baseline_cost) * 100).round(2)
comparison['Speedup'] = (baseline_time / comparison['Time [s]']).round(1)
comparison.style.format(
{
'Time [s]': '{:.1f}',
'Cost [€]': '{:,.0f}',
'CHP': '{:.1f}',
'Boiler': '{:.1f}',
'Storage': '{:.0f}',
'Cost Gap [%]': '{:.2f}',
'Speedup': '{:.1f}x',
}
)
| Time [s] | Cost [€] | CHP | Boiler | Storage | Cost Gap [%] | Speedup | |
|---|---|---|---|---|---|---|---|
| Full (baseline) | 12.3 | -148,910 | 167.6 | 0.0 | 1000 | 0.00 | 1.0x |
| Clustered (8 days) | 4.8 | -138,770 | 171.0 | 0.0 | 1000 | 6.81 | 2.6x |
| Two-Stage | 9.4 | -150,105 | 179.5 | 0.0 | 1050 | -0.80 | 1.3x |
Expand Solution to Full Resolution¶
Use expand() to map the clustered solution back to all original timesteps. This repeats the typical period values for all days belonging to that cluster:
# Expand the clustered solution to full resolution
fs_expanded = fs_clustered.transform.expand()
# Compare heat production: Full vs Expanded
heat_flows = ['CHP(Q_th)|flow_rate', 'Boiler(Q_th)|flow_rate']
# Create comparison dataset
comparison_ds = xr.Dataset(
{
name.replace('|flow_rate', ''): xr.concat(
[fs_full.solution[name], fs_expanded.solution[name]], dim=pd.Index(['Full', 'Expanded'], name='method')
)
for name in heat_flows
}
)
comparison_ds.plotly.line(x='time', facet_col='variable', color='method', title='Heat Production Comparison')
Visualize Clustered Heat Balance¶
fs_clustered.stats.plot.storage('Storage')
fs_expanded.stats.plot.storage('Storage')
API Reference¶
transform.cluster() Parameters¶
| Parameter | Type | Default | Description |
|---|---|---|---|
n_clusters | int | - | Number of typical periods (e.g., 8 typical days) |
cluster_duration | str \| float | - | Duration per cluster ('1D', '24h') or hours |
cluster | ClusterConfig | None | Clustering algorithm and weights. Use weights={var: 0} to exclude variables. |
extremes | ExtremeConfig | None | Essential: Force inclusion of peak/min periods |
segments | SegmentConfig | None | Intra-period segmentation (variable timestep durations) |
**tsam_kwargs | - | - | Additional tsam parameters |
Clustering Object Properties¶
After clustering, access metadata via fs.clustering:
| Property | Description |
|---|---|
n_clusters | Number of clusters |
n_original_clusters | Number of original time segments (e.g., 31 days) |
timesteps_per_cluster | Timesteps in each cluster (e.g., 96 for daily at 15 min) |
cluster_assignments | xr.DataArray mapping original segment to cluster ID |
cluster_occurrences | How many original segments each cluster represents |
clustering_result | Full tsam_xarray ClusteringResult |
aggregation_result | Full tsam_xarray AggregationResult (pre-IO only) |
Storage Behavior¶
Each Storage component has a cluster_mode parameter:
| Mode | Description |
|---|---|
'intercluster_cyclic' | Links storage across clusters + yearly cyclic (default) |
'intercluster' | Links storage across clusters, free start/end |
'cyclic' | Each cluster is independent but cyclic (start = end) |
'independent' | Each cluster is independent, free start/end |
For a detailed comparison of storage modes, see 08c2-clustering-storage-modes.
Summary¶
You learned how to:
- Use
cluster()to reduce time series into typical periods - Apply peak forcing with
ExtremeConfigto capture extreme demand days - Use two-stage optimization for fast yet accurate investment decisions
- Expand solutions back to full resolution with
expand() - Access clustering metadata via
fs.clustering - Apply existing clustering to other FlowSystems using
apply_clustering()
Key Takeaways¶
- Always use peak forcing (
extremes=ExtremeConfig(max_value=[...])) for demand time series - Add safety margin (5-10%) when fixing sizes from clustering
- Two-stage is recommended: clustering for sizing, full resolution for dispatch
- Storage handling is configurable via
cluster_mode - Use
apply_clustering()to apply the same clustering to different FlowSystem variants - For advanced clustering options (weights, algorithms, segmentation, tuning), see tsam_xarray and tsam
Next Steps¶
- 08c2-clustering-storage-modes: Compare storage modes using a seasonal storage system