🗺️ Chronos-2 for Specific Domains

[abdulfatir]

Starting this thread to discuss the use cases of Chronos-2 for specific domains such as:

• Retail and logistics forecasting — exogenous variables such as promotions and holidays which are important for accurate demand forecasting are natively supported.
• Energy forecasting — exogenous variables such as historical weather & forecasts are natively supported.
• CloudOps and IoT metrics forecasting — multivariate forecasting allows Chronos-2 to jointly predict related metrics and the long context allows accurate forecasting for high-frequency metrics.
• and more.

On our cases studies for energy and retail domain, Chronos-2 strongly outperformed existing models as shown in the figure below.

[beriberikix]

🙋 definitely interested in the IoT domain! A large part of IoT includes TS data. The most obvious is sensor data (ex. environmental readings) but also somewhat structured data from other systems (ex. automotive CAN data or OT data like SCADA.)

Since this is a pretty broad topic, what would be the best way to contribute?

[abdulfatir]

@beriberikix Sorry, I missed this. The best way to contribute here would be to share your results and finding with the community here/

[ChernovAndrey]

Hello @abdulfatir,

Thanks for sharing the results on specific domains. I find them very interesting.
I’ve tried to achieve a similar performance boost from covariates on several retail datasets, but I wasn’t able to replicate the same improvements. Would it be possible for you to share the code used to reproduce the results on the retail and energy datasets with covariates?

Thank you, and best regards,
Andrei

[abdulfatir]

@ChernovAndrey these results are just subsets of fev-bench. You can run the model on the specific tasks by following this script: https://github.com/autogluon/fev/tree/main/examples/chronos-2

[ChernovAndrey]

Thanks for the answer!

I ran the Chronos2 evaluation on the Rossmann dataset without covariates and with all possible combinations of covariates. The results suggested that adding only the Promo and Open covariates improves the forecast. When using the past Customers covariate and future holiday covariates, the metrics get worse. Including all possible covariates also results in worse CRPS and WAPE compared to the forecast without covariates.

However, I computed the metrics per store rather than on total sales, and then averaged them across stores. I also filtered out stores with many zero values. When evaluating total sales instead, the overall metric is better and matches your results.

I’ve also tried other datasets and noticed that adding seasonal covariates, such as holidays or non-working days, tends to worsen performance. I’m wondering whether you’ve experienced the same behavior.

[vpanchyk]

@ChernovAndrey, I observe the same Chronos-2 behavior when adding holiday covariates (US holidays) worsens the performance. I wonder whether it comes from Chronos' inherent ability to catch some holidays, or my holiday covariates are not of a good quality, misleading the model. I used to use these holiday covariates for a tree-based model and it improved the model, so I was surprised it wasn't the case with Chronos-2. Any thoughts?

[ChernovAndrey]

@vpanchyk, from my understanding, there is also different behavior depending on whether you pass covariates as integers/floats or as strings. When you pass holidays as strings to the model, it uses a TargetEncoder, but when you use integers/floats, it passes the raw covariates into the model.

However, for me, seasonal features did not work in either configuration. Only after fine-tuning Chronos2 with seasonal features did I see any improvement.

[Miao04]

Hi @abdulfatir ,

Thank you for your excellent work on Chronos-2. I’m working on atmospheric and ecosystem studies, and I’ve been validating Chronos-2 on several sites—so far the results look very promising.

I’m curious whether it’s possible to fine-tune Chronos-2 to better capture the relationships between covariates and the target, so that we can make high-quality forecasts with as short a history window as possible (or even in the case, all historical target values in a site is nan).
Because in practice, we may have many long-term observation in covariates, but the obervation in target may be not long enough.

Thank you in advance, and best regards,
Miao

[abdulfatir]

@Miao04 Yes, you can fine-tune the model. There are two options:

• Using this package. Check the fine-tuning section here: https://github.com/amazon-science/chronos-forecasting/blob/main/notebooks/chronos-2-quickstart.ipynb
• Using AutoGluon. This allows you to easily compare and ensemble with other models. See: https://auto.gluon.ai/stable/tutorials/timeseries/forecasting-chronos.html

as short a history window as possible

This is something you would need to experiment with. Typically, if the time series has some natural seasonality, you would at least need to provide history of a few seasons for the model to capture the seasonality.

[saif-a-a]

@abdulfatir
I have been utilising Chronos-2 for transport modelling Domain.
Particularly for road segment volumes (15-min interval).
This prediction below is for one road segment, where the weekends have been masked by NaNs.
Day 1 and 2 are weekends so can be ignored, the other days seem to perform well.

While I did pass multiple road segments (multiple time series) at once, I am interested in understanding how I could make it spatial-aware? Adding features would make it better, but am concerned of one-hot encoded features for potentially 500 road segments

[abdulfatir]

@saif-a-a can you elaborate what you mean by spatial aware in this context?

[saif-a-a]

An adjacency matrix (of sorts), I suppose. So, the model can predict better by attending more to values within proximity, rather than those further away