🔍 Read the full analysis: Is Your Fine-Tuning Job Taking Longer Than Expected? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Hugging Face reports that a CatBoost model predicted the wall-clock duration of 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. Its best tested FLOP-based estimate had a 12.41-hour error, but the results come from one production environment and do not show whether using the predictor saves GPU time or reduces delays.
Hugging Face says a CatBoost model predicted the wall-clock runtime of 272 supervised fine-tuning jobs on shared NVIDIA H200 GPUs with a mean absolute error of 1.71 hours in five-fold cross-validation. The result was more accurate on this dataset than the best tested FLOP-based estimate, but it has not yet been validated across other clusters or in a live scheduling trial.
The jobs ran in a corporate production environment and accounted for about 7,382 GPU-hours. Their durations ranged from 1.6 to 172.4 hours. The data covered models up to 15 billion parameters, both LoRA and full fine-tuning, different dataset sizes, and jobs assigned one, two or four GPUs. The shared pool contained 8 to 32 H200s across one to four nodes, with training and inference workloads running alongside the jobs.
The prediction target was wall-clock time from GPU allocation to job completion; queue time was excluded. The model used 24 selected features based on information available before dispatch, including training configuration, evaluation cadence, checkpoint settings and per-GPU token volumes. It also included a FLOP-based duration estimate as one input.
Hugging Face reports an R² of 0.98, root mean squared error of 3.52 hours and median absolute error of 0.51 hours for CatBoost in stratified five-fold cross-validation. The best tested FLOP baseline, using an assumed 30% Model FLOPs Utilization, had a mean absolute error of 12.41 hours and an R² of 0.31. Baselines at 45% and 60% MFU had errors of 13.01 and 14.62 hours, respectively. These figures describe performance within the study dataset, not a demonstrated improvement in a deployed scheduler.
Why Fine-Tuning Takes Longer Than FLOPs Say
Hugging Face tested runtime predictions on 272 production fine-tuning jobs. Configuration-aware estimates were more accurate in this dataset, but results from one production pool do not establish performance elsewhere.
What the estimates capture
The target is time from GPU allocation to job completion; queue time is excluded.
The training loop
FLOP estimates approximate compute using an assumed Model FLOPs Utilization, or MFU.
The whole job
CatBoost uses job settings and model metadata, including evaluations, checkpointing and token volumes.
Other runtime costs
Setup, data handling, network and storage can add time alongside training.
Prediction error in the tested data
Lower error means predicted duration was closer to observed duration on average.
What this could mean for GPU scheduling
More useful duration estimates could help teams plan scarce GPU capacity.
- GPUs may sit idle until another job starts.
- Operators can misjudge when capacity will be free.
- Other work can be delayed.
- Delivery dates may shift.
How far the result reaches
The preview reports cross-validation on one corporate production environment.
Runs used shared NVIDIA H200 GPUs and varied configurations.
Accuracy on other clusters, software stacks and workload mixes is not established.
The preview reports prediction error, not savings in GPU time or scheduling delays.
More Useful Estimates for GPU Schedules
Runtime estimates affect when operators dispatch queued work and how they allocate scarce GPU capacity. If a job finishes earlier than expected, GPUs may sit unused until another job can start; if it runs longer, it can delay later jobs and affect delivery plans. More accurate estimates could help cluster operators coordinate work, though this study measured prediction error rather than changes to scheduling outcomes.
The difference between the reported errors suggests that a formula based mainly on compute may not capture enough of the factors shaping a full fine-tuning job. A predictor that incorporates configuration and operational details could offer more useful planning information for workloads similar to those studied. The practical value remains conditional: the preview does not quantify GPU hours saved, lower costs or fewer delays.
Products Worth Considering

Amazon Kindle Scribe (16GB) with Premium Pen, Tungsten

DJI Mini 4K Drone with 4K UHD Camera, Under 249 g

Jackery Explorer 1000 v2 Portable Power Station, 1024Wh LiFePO4 Battery
As an affiliate, we earn on qualifying purchases.
Why FLOP Estimates Can Miss Runtime
A common estimate approximates dense Transformer training compute using parameter count and training-token count, then divides by GPU count, theoretical throughput and an assumed MFU. MFU is the share of theoretical peak performance achieved in practice. The preview says real workloads lose time to memory traffic, communication and kernel inefficiencies, and places typical MFU across a broad 30% to 60% range.
That calculation focuses on the training loop, while the study’s runtime target includes other work after GPUs are allocated. Setup, model loading, tokenization, evaluations, checkpoint uploads and final storage can add time. On a multi-tenant cluster, these steps may also compete for network and storage resources. The researchers compared several MFU assumptions and included scheduled evaluation tokens in the FLOP estimate, according to the preview.
The distinction matters because the reported task was not simply to estimate theoretical training compute. It was to forecast a job’s end-to-end wall-clock duration after allocation. Hugging Face says the model used submitted configuration and model metadata that were available before dispatch.
“Everything is known before dispatch.”
— Hugging Face, in its study preview
How Far the Results Extend
The reported scores come from cross-validation on 272 runs from one production environment. That evaluation does not establish how the model would perform on a different GPU type, cluster, software stack or workload mix. The preview also does not report a test on a later, unseen period or a live scheduling trial, so it is unclear whether the model’s accuracy would persist as configurations and shared-cluster conditions change.
The study measures prediction accuracy, not operational effects. It does not report whether using CatBoost reduced idle GPU intervals, delayed jobs, compute costs or delivery times. The full paper has not yet been released, and the preview provides no publication date; further details on the data and evaluation design may become available there.
Full Paper and External Tests
Hugging Face says it plans to post the full version on arXiv, but has not announced when. The paper may provide more detail on feature selection, dataset composition and the evaluation procedure.
To show whether the result applies beyond this production pool, subsequent testing would need runs from different time periods or computing environments. A trial within an operational scheduler could then measure whether better forecasts lead to fewer delays or idle GPU intervals. Until such evidence is reported, the 1.71-hour figure should be read as a cross-validation result on this dataset, not a general runtime guarantee.
Key Questions
What did the CatBoost model predict?
It predicted wall-clock time from GPU allocation until a supervised fine-tuning job finished. Queue time was not included.
How accurate was the prediction?
Hugging Face reports a 1.71-hour mean absolute error and a 0.51-hour median absolute error in five-fold cross-validation on 272 jobs. Those scores apply to the study dataset.
How did it compare with the FLOP-based estimate?
The best tested FLOP baseline, using a 30% MFU assumption, had a 12.41-hour mean absolute error. The preview reports higher errors at the tested 45% and 60% MFU settings.
Does the study show that GPU time or costs were saved?
No. It reports prediction accuracy, but does not describe a controlled deployment measuring GPU time saved, cost reductions or scheduling delays avoided.
When will the full paper be available?
Hugging Face says the full version will be posted on arXiv, but the preview gives no publication date.
Primary source: Hugging Face · via ThorstenMeyerAI.com
