Gayanuka's Space
←Back to all work
Machine Learning· 2025· Research & implementation, team of 2Shipped

FastVLM Ablations

Extending Apple's FastVLM (CVPR 2025) with video fine-tuning and a real-time Windows inference engine

FastVLM Ablations

Vision-language models are hard to run anywhere near real time, and the bottleneck is token count: a CLIP-based encoder spends 576 tokens on a single image, so eight frames of video already cost 4,608 tokens before the model has read a word of the prompt. FastVLM: Efficient Vision Encoding for Vision Language Models (Apple, CVPR 2025) attacks exactly this problem - its FastViTHD hybrid encoder emits roughly 100 tokens per image instead of 576.

We wanted that efficiency for video understanding on ordinary hardware, and Apple's release left two gaps in the way:

  • No video training path - the released code fine-tunes on images only, with no support for video-text pairs
  • No non-Apple inference story - the demos target Apple Silicon, with no way to point the model at a live webcam on a Windows machine

Closing those two gaps became the two halves of this project.

1. Video Fine-tuning Adaptation

Choosing the approach

There are two ways to make an image VLM understand video. The heavyweight option changes the architecture: add a temporal module or token resampler so frames exchange information before reaching the language model. The lightweight option treats video as a "flipbook" - sample frames uniformly, encode each one independently, and concatenate the tokens.

We chose the flipbook, and the deciding factor was token arithmetic. Naive frame concatenation is normally ruled out by cost - 8 frames through CLIP is 4,608 tokens - but through FastViTHD the same 8 frames cost ~800 tokens, cheap enough that the simple approach becomes viable. That decision bought us:

  • No architectural changes: the vision encoder and language model stay untouched, so Apple's pretrained checkpoints remain fully usable
  • Backward compatibility: the same model still handles single-image tasks after video fine-tuning
  • An explicit trade-off: frames never attend to each other inside the encoder, so temporal reasoning must be learned by the language model - which is why we monitored temporal coherence during validation instead of assuming it
Video Fine-tuning Training Metrics
Training metrics showing loss convergence and performance improvements during video fine-tuning

What it took

  1. Video Loading Functions: Added utility functions to detect and process video files
  2. Dataset Modifications: Extended LazySupervisedDataset to handle both images and videos
  3. Token Expansion Strategy: Modified preprocess_multimodal() to expand <image> tokens per frame
  4. Configuration Parameters: Added video-specific training arguments

Training configuration:

  • Sparse temporal sampling (4-8 frames per video)
  • Batch processing with mixed precision training
  • Validation on video-text datasets
  • Monitored metrics: loss convergence, BLEU scores, temporal coherence
Note: Fine-tuning implementation is available at EdgeVLM-Labs/fastvlm-adaptation as it requires modifications to core training code in the llava/ module.

Results

  • Training Stability: Consistent loss convergence across video datasets
  • Temporal Understanding: The model learned to process sequential visual information
  • Backward Compatibility: Performance on image-only tasks was maintained after video fine-tuning

2. Windows Inference Engine

The second gap was practical: a fine-tuned model is only useful to us if we can watch it respond to a live camera on the machines we actually own. We built a Gradio-based inference application with two modes - static image analysis and real-time webcam streaming.

Windows Inference Engine Interface
Interactive inference interface with real-time performance monitoring

In Chat Mode, users upload images for detailed analysis using custom or preset prompts - describing scenes, counting objects - with full-length responses and performance metrics like TTFT and TPS. In Live Mode, the system runs continuous capture and inference on a webcam stream, with responses kept short so they keep pace with the video.

Making it real-time

Each optimization pairs a bottleneck with its fix:

  • Prompt caching - re-tokenizing the same prompt on every frame is wasted work, so tokenized prompts are cached and invalidated only when the prompt changes
  • Frame skipping - inference cannot keep up with full frame rate, so an adjustable 1-10x skip rate trades visual coverage for responsiveness
  • TF32 + FP16 - mixed precision on Ampere-class GPUs, with explicit memory cleanup to keep long sessions stable
  • Response shaping - live-mode outputs are cleaned, sentence-completed, and length-limited so replies never lag behind the stream

System requirements

  • Python 3.10+
  • CUDA-enabled GPU (recommended for real-time performance)
  • 8GB+ VRAM for Stage 3 model
  • Windows/Linux compatible
The net result: an image-only research release turned into a video-capable model with a live inference loop on commodity hardware - without touching the architecture that made it fast in the first place.

References

[1] P. K. A. Vasu, F. Faghri, C.-L. Li, C. Koc, N. True, A. Antony, G. Santhanam, J. Gabriel, P. Grasch, O. Tuzel, and H. Pouransari, "FastVLM: Efficient Vision Encoding for Vision Language Models," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. [Online]. Available: https://www.arxiv.org/abs/2412.13303

Contributors

For detailed documentation, setup instructions, and code examples, visit the GitHub repository.