V-Nutri: Dish-Level Nutrition Estimation from Egocentric Cooking Videos
V-Nutri fuses final-dish and cooking keyframes; HD-EPIC experiments show process cues can improve nutrition estimation.
Key Findings
Methodology
V-Nutri is a staged pipeline. Cooking Keyframes Selection uses VideoMamba-Middle on 2-second, 16-frame clips to identify ingredient-addition moments. A frozen ResNet-101, ViT-B/16, or ViT-L/16 backbone pretrained on Nutrition5K encodes the selected process frames and final dish frame. Learned attention pooling produces a process representation, which is combined with the dish representation through concatenation or gated projection and mapped by an MLP to calories, protein, fat, and carbohydrates.
Key Results
- The constructed HD-EPIC benchmark contains 80 recipe instances from 69 unique recipes; 52 are regression-ready and 22 also have complete add-event timestamps. Manual refinement updated 372 ingredient-addition timestamps and 52 final-dish timestamps, creating a structured basis for video nutrition evaluation.
- On Nutrition5K, ViT-L/16 achieved the lowest validation loss, 2.203. ResNet-101 achieved the best calorie MAE, 73.2 kcal, compared with 88.0 kcal for ViT-B/16 and 77.7 kcal for ViT-L/16. Its fat MAE was 5.60 g, versus 7.00 and 5.76 g.
- Process frames provided complementary evidence under controlled conditions, but gains depended strongly on representation capacity and event quality. VideoMamba ingredient-addition detection reached 62.7% F1 on the fold-0 test split; the paper does not claim universal improvement across all settings.
Significance
The study moves nutrition estimation from a plate-only setting toward process-aware observation. Oils, sauces, dairy products, and mixed ingredients can disappear visually through absorption, melting, or blending, whereas preparation videos may expose their addition and intermediate states. This connects food understanding, efficient long-video modeling, and digital health, offering a more realistic evidence source for dietary logging, personalized nutrition, and automated health monitoring.
Technical Contribution
V-Nutri combines sparse temporal evidence selection with static nutrition regression in a modular architecture, avoiding expensive dense processing of long videos. VideoMamba-Middle proposes candidate events; Nutrition5K pretraining preserves a nutrition-oriented visual prior; attention pooling compresses sparse process features; and gated fusion learns how much to trust dish versus process evidence. Frozen backbones and recipe-level cross-validation reduce overfitting risk and isolate the effect of process information.
Novelty
The authors present the first benchmark for dish-level nutrition estimation from cooking video and the first systematic framework using egocentric cooking keyframes as nutritional complements to the final dish image. Unlike Nutrition5K-style static methods, the central innovation is not a larger regressor but recovery of otherwise hidden evidence through ingredient-addition events and temporal context.
Limitations
- The benchmark is small: only 52 regression-ready instances come from nine participants and nine kitchens. Highly skewed targets, including calories from 0 to 4,792 kcal, make broad generalization uncertain.
- Only 22 instances have complete add-event timestamps, and the selector reaches 62.7% F1 on fold 0. Missed or incorrectly selected events can directly remove the process evidence on which the model relies.
- The system does not explicitly model ingredient quantity, cooking-related nutrient loss, eaten portion size, or cultural recipe variation.
Future Work
Future work should expand across kitchens, regions, cuisines, and recording devices while completing timestamps and quantity labels. Joint learning of event localization, ingredient recognition, and nutrition regression could reduce pipeline error. RGB-D, audio, recipe text, and nutrition databases may improve grounding. Uncertainty estimation, user correction, privacy protection, and efficient on-device inference are necessary for real-world dietary monitoring.
AI Executive Summary
People would like to point a camera at a meal and obtain estimates of calories, protein, fat, and carbohydrates. Yet most visual nutrition systems inspect only the finished dish. Oil, sauces, dairy, and mixed ingredients may be absorbed, melted, or hidden during cooking, leaving a final image that cannot reveal the full recipe or quantities. V-Nutri asks whether the preparation process can restore some of this missing evidence.
The framework first applies VideoMamba-Middle to egocentric cooking video, scanning 2-second clips containing 16 frames to locate candidate ingredient-addition events. Nutrition5K-pretrained ResNet-101, ViT-B/16, or ViT-L/16 backbones then encode the selected process frames and the final dish frame. Attention pooling compresses process evidence, while concatenation or gated projection fuses it with the dish representation before a lightweight MLP predicts four nutritional targets. This staged design avoids dense computation over long, redundant recordings.
The authors also reorganize HD-EPIC into an 80-instance benchmark: 52 instances have complete nutritional targets and 22 have complete add-event timestamps. Manual annotation refined 372 addition events. On Nutrition5K, ViT-L/16 achieved the best validation loss, 2.203, whereas ResNet-101 achieved the best calorie MAE, 73.2 kcal. VideoMamba reached 62.7% F1 on the fold-0 event-detection test split. The findings support process cues as complementary evidence, but their value depends on backbone quality and selection accuracy. Small data, incomplete temporal labels, and uncertain real-world generalization remain important caveats.
Deep Analysis
Background
Visual nutrition estimation evolved from Im2Calories toward Nutrition5K, which introduced multi-view observations, food weights, and direct nutrient regression. Later work added geometry, ingredient semantics, and external knowledge. However, most systems still rely on the finished dish and cannot observe absorbed oil, hidden sauces, or merged components. HD-EPIC offers egocentric kitchen video, recipe structure, ingredient metadata, and temporal actions, making it a suitable source for process-aware nutrition evidence.
Core Problem
The task is to predict y=[calories, protein, fat, carbohydrates] for an entire dish from cooking video. The challenge is that recordings are long, cluttered, and temporally redundant, while useful events are sparse. Final appearance also conceals ingredient identity, quantity, and transformation. HD-EPIC was originally designed for activity understanding: nutrition values are ingredient-level, and no unified dish-level target or protocol for final-dish and addition frames existed.
Innovation
- �� Benchmark: aggregate ingredient nutrition into dish-level targets and provide refined temporal annotations.
- �� Event selection: use VideoMamba-Middle to surface ingredient-addition candidates instead of densely processing every frame.
- �� Transfer learning: freeze Nutrition5K-trained visual backbones to preserve nutrition-relevant representations on a small target dataset.
- �� Fusion: use learned attention pooling and either concatenation or gated projection to combine temporal process evidence with final appearance.
Methodology
- �� Input: represent video v={f_t} as 2-second sliding clips with 16 frames at 224×224; inference uses a 1-second stride.
- �� Selection: VideoMamba-Middle outputs s_m=q_ω(c_m); the top-K timestamps become process candidates. A separate Final Dish Frame Selector chooses the clip with maximum dish-state confidence.
- �� Encoding: frozen Nutrition5K backbones produce z_d=f_θ(x_d) and z_i=f_θ(x_i). Dimensions are 2048 for ResNet-101, 768 for ViT-B/16, and 1024 for ViT-L/16.
- �� Pooling: attention scores s_i=g_ψ(z_i) yield α_i=exp(s_i)/Σ_j exp(s_j), and z_p=Σ_iα_i z_i.
- �� Fusion: concatenate [z_d;z_p], or compute σ(w)W_dz_d+(1−σ(w))W_ez_p, then regress four outputs with an MLP.
Experiments
HD-EPIC contains approximately 41 hours, 156 videos, nine participants, and nine kitchens. Evaluation uses recipe-level stratified five-fold cross-validation, with roughly 40–43 training and 9–12 test instances per fold. Targets are fold-wise z-score normalized. The study compares Dish-only, Ground Truth, Pred-K, Pred-all, Random-K, and Uniform-K sampling with K∈{20,50,200}. Fusion heads use hidden size 512, Smooth L1 loss, Adam at 10^-3 for 50 epochs; VideoMamba uses Kinetics-400 pretraining, AdamW at 10^-4, weight decay 0.05, and 30 epochs.
Results
Nutrition5K validation loss was 2.203 for ViT-L/16, 2.308 for ResNet-101, and 2.559 for ViT-B/16. Calorie MAE was 77.7, 73.2, and 88.0 kcal respectively; fat MAE was 5.76, 5.60, and 7.00 g; carbohydrate MAE was 6.81, 6.63, and 7.95 g; protein MAE was 5.19, 5.14, and 6.52 g. VideoMamba achieved 62.7% F1 on fold 0. The main conclusion is conditional: process cues help in controlled cases, but detection quality and backbone capacity govern the benefit.
Applications
The method could support smartphone or wearable-camera dietary logs by detecting oil, sauce, and ingredient additions and combining them with the completed dish. Potential users include athletes, diabetes patients, clinicians, dietitians, and food-service annotators. Deployment requires a sufficiently visible cooking stream, reliable recipe or nutrition databases, and user correction. Outputs should include uncertainty because current benchmark scale does not justify treating predictions as clinical measurements.
Limitations & Outlook
Only 52 instances are regression-ready, from limited kitchens and participants, and the nutrient distributions are strongly right-skewed; four instances have zero values across all nutrients. Complete temporal ground truth exists for only 22 instances, while event detection reaches 62.7% F1 on fold 0. The pipeline does not explicitly infer weight, portion consumed, nutrient loss, occlusion, or cross-cultural recipe variation. Larger datasets, multimodal inputs, calibration, and longitudinal user studies are needed.
Plain Language Accessible to non-experts
Imagine a kitchen detective estimating the cost of a finished meal. An ordinary system enters the kitchen only after dinner is ready and examines the plate. It can see the final result, but not the oil poured into the pan, the sauce mixed into the noodles, or the cheese that melted underneath. V-Nutri lets the detective watch the whole cooking session and mark moments such as “oil added,” “sauce poured,” or “meat dropped in.”
The detective does not study every second equally. VideoMamba acts like a quick assistant that circles the most informative moments in a long recording. Other trained visual helpers examine those moments and the finished plate. The system gives more influence to clues that seem useful, then combines the cooking history with the final appearance, much as someone estimates a household bill from both receipts and the items finally on the table.
The output is an estimate of calories, protein, fat, and carbohydrates. In HD-EPIC, the researchers organized 80 cooking instances and manually refined 372 addition moments. The results suggest that watching the process can help, but only when the camera captures the action and the assistant selects the right moments. It is promising evidence, not a perfect nutritional scale.
ELI14 Explained like you're 14
Think of a food-guessing game. The old version gives you one photo of a finished pizza and asks how much oil, cheese, and meat went into it. Tricky, right? Some cheese is melted, some sauce is hidden, and you cannot tell how much oil was used just by staring at the crust.
V-Nutri gives the player the whole cooking video. A helper called VideoMamba scans it for important moments: pouring oil, adding sauce, dropping in meat, and so on. It does not treat every frame as equally important, because that would be like taking a screenshot of every second of a three-hour game stream. Most screenshots would show nothing useful!
Next, vision models that practiced on the Nutrition5K food dataset look at the selected moments and the final dish. The system decides which clues matter most and combines them to predict calories, protein, fat, and carbohydrates. On Nutrition5K, ResNet-101 had a calorie error of 73.2 kcal, while the VideoMamba event detector scored 62.7% F1 on one test split.
But it is not a mind reader. If a hand blocks the ingredients, the video misses an addition, or the helper chooses the wrong moment, the estimate can fail. Bigger and more varied datasets could make it much better. The exciting idea is simple: sometimes the story of how food was made tells you more than the final picture!
Glossary
Egocentric video
A recording captured from the cook’s first-person viewpoint. It preserves actions and their temporal order.
HD-EPIC provides the cooking videos used by V-Nutri.
Nutrition5K
A dataset of roughly 5,000 real dishes with visual streams, ingredient or weight information, and nutritional annotations. It supports nutrition-oriented visual pretraining.
Its fine-tuned backbones are frozen during HD-EPIC training.
VideoMamba
A video sequence model based on selective state-space modeling, designed for efficient long-range temporal processing. It avoids the computational pattern of full dense attention.
VideoMamba-Middle selects candidate ingredient-addition clips.
Keyframe selection
Choosing a small set of informative video frames rather than processing the entire sequence. It reduces computation and irrelevant temporal noise.
Cooking Keyframes Selection identifies process evidence.
Attention pooling
A weighted averaging operation in which learned scores determine each feature’s contribution. More relevant frames receive larger weights.
It converts multiple process embeddings into z_p.
MAE
Mean absolute error, the average absolute difference between predictions and targets; lower is better. It is more stable than percentage error when targets can be zero.
The paper reports separate MAE values for calories and macronutrients.
Open Questions Unanswered questions from this research
- 1 Will process cues remain useful across cuisines, cultures, kitchens, cameras, and users? The 52 regression-ready instances are too limited to establish robust cross-domain generalization.
- 2 How can a video system accurately infer ingredient weight, cooking loss, and the portion actually eaten? This likely requires weighing data, depth sensing, structured recipes, or external nutrition knowledge.
- 3 How should event-detection confidence become calibrated uncertainty for nutrition predictions? Clinical and health applications need trustworthy intervals, not only single-point estimates.
Applications
Immediate Applications
Automated dietary logging
A smartphone or wearable camera could record cooking, detect oil, sauce, and ingredient additions, and combine them with the final dish to estimate four nutrients. Practical use requires a clear view, recipe or nutrition databases, and a way for users to correct missed ingredients.
Fitness and chronic-disease monitoring
Athletes, diabetes patients, dietitians, and clinicians could obtain richer meal records without manually entering every ingredient. Because the benchmark is small, predictions should be advisory, calibrated, and accompanied by uncertainty rather than presented as clinical measurements.
Long-term Vision
Personalized nutrition platform
A future system could combine video, spoken recipes, user history, and physiological data to learn personal oil and portion habits and provide adaptive dietary advice. Major barriers include privacy, cultural generalization, standardized labels, and reliable validation over long-term use.
Abstract
Nutrition estimation of meals from visual data is an important problem for dietary monitoring and computational health, but existing approaches largely rely on single images of the finally completed dish. This setting is fundamentally limited because many nutritionally relevant ingredients and transformations, such as oils, sauces, and mixed components, become visually ambiguous after cooking, making accurate calorie and macronutrient estimation difficult. In this paper, we investigate whether the cooking process information from egocentric cooking videos can contribute to dish-level nutrition estimation. First, we further manually annotated the HD-EPIC dataset and established the first benchmark for video-based nutrition estimation. Most importantly, we propose V-Nutri, a staged framework that combines Nutrition5K-pretrained visual backbones with a lightweight fusion module that aggregates features from the final dish frame and cooking process keyframes extracted from the egocentric videos. V-Nutri also includes a cooking keyframes selection module, a VideoMamba-based event-detection model that targets ingredient-addition moments. Experiments on the HD-EPIC dataset show that process cues can provide complementary nutritional evidence, improving nutrition estimation under controlled conditions. Our results further indicate that the benefit of process keyframes depends strongly on backbone representation capacity and event detection quality. Our code and annotated dataset is available at https://github.com/K624-YCK/V-Nutri.