AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: Is Your Fine-Tuning Job Taking Longer Than Expected? on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

TL;DR

Hugging Face reports that a CatBoost model predicted the wall-clock duration of 272 supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. Its best tested FLOP-based estimate had a 12.41-hour error, but the results come from one production environment and do not show whether using the predictor saves GPU time or reduces delays.

Hugging Face says a CatBoost model predicted the wall-clock runtime of 272 supervised fine-tuning jobs on shared NVIDIA H200 GPUs with a mean absolute error of 1.71 hours in five-fold cross-validation. The result was more accurate on this dataset than the best tested FLOP-based estimate, but it has not yet been validated across other clusters or in a live scheduling trial.

The jobs ran in a corporate production environment and accounted for about 7,382 GPU-hours. Their durations ranged from 1.6 to 172.4 hours. The data covered models up to 15 billion parameters, both LoRA and full fine-tuning, different dataset sizes, and jobs assigned one, two or four GPUs. The shared pool contained 8 to 32 H200s across one to four nodes, with training and inference workloads running alongside the jobs.

The prediction target was wall-clock time from GPU allocation to job completion; queue time was excluded. The model used 24 selected features based on information available before dispatch, including training configuration, evaluation cadence, checkpoint settings and per-GPU token volumes. It also included a FLOP-based duration estimate as one input.

Hugging Face reports an R² of 0.98, root mean squared error of 3.52 hours and median absolute error of 0.51 hours for CatBoost in stratified five-fold cross-validation. The best tested FLOP baseline, using an assumed 30% Model FLOPs Utilization, had a mean absolute error of 12.41 hours and an R² of 0.31. Baselines at 45% and 60% MFU had errors of 13.01 and 14.62 hours, respectively. These figures describe performance within the study dataset, not a demonstrated improvement in a deployed scheduler.

At a glance
reportWhen: Study preview published; full paper and…
The developmentHugging Face has previewed a runtime prediction study comparing a CatBoost model with FLOP-based estimates across 272 production fine-tuning jobs.
AI Training · Briefing

Why Fine-Tuning Takes Longer Than FLOPs Say

Hugging Face tested runtime predictions on 272 production fine-tuning jobs. Configuration-aware estimates were more accurate in this dataset, but results from one production pool do not establish performance elsewhere.

1.71 hours
CatBoost mean absolute error
12.41 hours
Best tested FLOP baseline error
272
Production jobs evaluated
7,382
Approximate GPU-hours

What the estimates capture

The target is time from GPU allocation to job completion; queue time is excluded.

Compute

The training loop

FLOP estimates approximate compute using an assumed Model FLOPs Utilization, or MFU.

Configuration

The whole job

CatBoost uses job settings and model metadata, including evaluations, checkpointing and token volumes.

Shared systems

Other runtime costs

Setup, data handling, network and storage can add time alongside training.

Prediction error in the tested data

Lower error means predicted duration was closer to observed duration on average.

CatBoost1.71 hours
FLOP baseline, 30% MFU12.41 hours
FLOP baseline, 45% MFU13.01 hours
FLOP baseline, 60% MFU14.62 hours

What this could mean for GPU scheduling

More useful duration estimates could help teams plan scarce GPU capacity.

When jobs finish early
  • GPUs may sit idle until another job starts.
  • Operators can misjudge when capacity will be free.
When jobs run over
  • Other work can be delayed.
  • Delivery dates may shift.

How far the result reaches

The preview reports cross-validation on one corporate production environment.

1
Tested on one pool

Runs used shared NVIDIA H200 GPUs and varied configurations.

2
Generalization remains open

Accuracy on other clusters, software stacks and workload mixes is not established.

3
Operational impact remains unmeasured

The preview reports prediction error, not savings in GPU time or scheduling delays.

What to watchThe full paper is forthcoming; validation on independent runs and a live scheduler trial would show whether the result carries over.
The takeaway
Configuration-aware prediction beat the tested FLOP estimates on these jobs, but teams need broader validation before relying on the same accuracy for their own clusters.

More Useful Estimates for GPU Schedules

Runtime estimates affect when operators dispatch queued work and how they allocate scarce GPU capacity. If a job finishes earlier than expected, GPUs may sit unused until another job can start; if it runs longer, it can delay later jobs and affect delivery plans. More accurate estimates could help cluster operators coordinate work, though this study measured prediction error rather than changes to scheduling outcomes.

The difference between the reported errors suggests that a formula based mainly on compute may not capture enough of the factors shaping a full fine-tuning job. A predictor that incorporates configuration and operational details could offer more useful planning information for workloads similar to those studied. The practical value remains conditional: the preview does not quantify GPU hours saved, lower costs or fewer delays.

Products Worth Considering

Jackery Explorer 1000 v2 Portable Power Station, 1024Wh LiFePO4 Battery

Jackery Explorer 1000 v2 Portable Power Station, 1024Wh LiFePO4 Battery

As an affiliate, we earn on qualifying purchases.

Why FLOP Estimates Can Miss Runtime

A common estimate approximates dense Transformer training compute using parameter count and training-token count, then divides by GPU count, theoretical throughput and an assumed MFU. MFU is the share of theoretical peak performance achieved in practice. The preview says real workloads lose time to memory traffic, communication and kernel inefficiencies, and places typical MFU across a broad 30% to 60% range.

That calculation focuses on the training loop, while the study’s runtime target includes other work after GPUs are allocated. Setup, model loading, tokenization, evaluations, checkpoint uploads and final storage can add time. On a multi-tenant cluster, these steps may also compete for network and storage resources. The researchers compared several MFU assumptions and included scheduled evaluation tokens in the FLOP estimate, according to the preview.

The distinction matters because the reported task was not simply to estimate theoretical training compute. It was to forecast a job’s end-to-end wall-clock duration after allocation. Hugging Face says the model used submitted configuration and model metadata that were available before dispatch.

“Everything is known before dispatch.”

— Hugging Face, in its study preview

How Far the Results Extend

The reported scores come from cross-validation on 272 runs from one production environment. That evaluation does not establish how the model would perform on a different GPU type, cluster, software stack or workload mix. The preview also does not report a test on a later, unseen period or a live scheduling trial, so it is unclear whether the model’s accuracy would persist as configurations and shared-cluster conditions change.

The study measures prediction accuracy, not operational effects. It does not report whether using CatBoost reduced idle GPU intervals, delayed jobs, compute costs or delivery times. The full paper has not yet been released, and the preview provides no publication date; further details on the data and evaluation design may become available there.

Full Paper and External Tests

Hugging Face says it plans to post the full version on arXiv, but has not announced when. The paper may provide more detail on feature selection, dataset composition and the evaluation procedure.

To show whether the result applies beyond this production pool, subsequent testing would need runs from different time periods or computing environments. A trial within an operational scheduler could then measure whether better forecasts lead to fewer delays or idle GPU intervals. Until such evidence is reported, the 1.71-hour figure should be read as a cross-validation result on this dataset, not a general runtime guarantee.

Key Questions

What did the CatBoost model predict?

It predicted wall-clock time from GPU allocation until a supervised fine-tuning job finished. Queue time was not included.

How accurate was the prediction?

Hugging Face reports a 1.71-hour mean absolute error and a 0.51-hour median absolute error in five-fold cross-validation on 272 jobs. Those scores apply to the study dataset.

How did it compare with the FLOP-based estimate?

The best tested FLOP baseline, using a 30% MFU assumption, had a 12.41-hour mean absolute error. The preview reports higher errors at the tested 45% and 60% MFU settings.

Does the study show that GPU time or costs were saved?

No. It reports prediction accuracy, but does not describe a controlled deployment measuring GPU time saved, cost reductions or scheduling delays avoided.

When will the full paper be available?

Hugging Face says the full version will be posted on arXiv, but the preview gives no publication date.

Primary source: Hugging Face · via ThorstenMeyerAI.com

You May Also Like

Could Anthropic’s New Tool Let Your Boss Review Claude Chats?

Futurism reports Anthropic introduced a tool for employers to monitor workers’ Claude chats, but its features, availability and access rules remain unverified.

Game 3: Both Teams Destroy Barracks?

In Game 3, both teams destroyed each other’s barracks, marking a rare strategic move in the match. Details are confirmed, but implications remain uncertain.

A Free GTA 6 Competitor Is Blowing Up On Steam

Kefir Games’ free-to-play crime sandbox ROB RUN reached about 140,440 concurrent Steam players, but reviews are Mixed amid performance complaints.

Anthropic Joins Forces With OpenAI On Markdown Instructions Standard For AI

Anthropic has reportedly decided to support OpenAI’s markdown instructions specification, potentially easing cross-platform AI instruction sharing.