MJ Mujadded Al Rabbani Alif

20 Aug 2026 · Notes

Lightweight by constraint

Four of my applied papers have the word “lightweight” in the title. That is not a stylistic tic. It’s what happens when the deployment target picks the model.

Look at the applied half of my publication list and a word keeps appearing. A lightweight convolutional network with integrated attention for missing bolt detection in railways. A lightweight CNN for exudate detection in retinal fundus images. A lightweight convolutional network for helmet detection on construction sites. Different domains, different collaborators, same adjective.

I didn’t set out to specialise in small models. The constraint arrived first and the architecture followed, every time. This post is about what that constraint actually consists of, because “it has to run on the edge” is where most discussions stop and where the interesting part starts.

The budget is the frame, not the model

The number that matters is not how long inference takes. It is how long you have per frame, minus everything that is not inference.

The per-frame budget on an edge device A horizontal bar divided into four segments: capture and decode, preprocess, model inference, and postprocess including non-maximum suppression and tracking. Inference is one segment among four, not the whole bar. A bracket beneath marks the total as the per-frame budget, and an arrow shows that whatever the camera frame rate allows is the fixed ceiling. No measured values are shown; the segment widths are illustrative of structure, not of timing. capture + decode preprocess model inference postprocess NMS, tracking one frame’s budget fixed by the camera — miss it and you drop the frame structure only; segment widths are not measured timings
Figure 1: inference is one segment of the frame budget, not the budget itself. Shrinking the model only moves the accent-coloured block.

This is why “we got the model down to n milliseconds” can be true and still not ship. If capture, preprocessing and non-maximum suppression eat most of the frame, halving inference buys you a fraction of what the headline suggests. Measure the loop, not the layer.

Average FPS is the wrong summary

A detector that averages comfortably inside its budget and occasionally spikes past it does not degrade gracefully. It drops frames, and it drops them in bursts, because the conditions that cause a spike tend to persist for more than one frame. Crowded scenes produce more candidate boxes, and more candidate boxes make postprocessing slower exactly when the scene is hardest.

So the useful summary is not the mean. It is the tail: the p95, and how far past the budget it sits. Two models with identical average throughput can behave completely differently at the ninety-fifth percentile, and only one of them is deployable.

Small is a design decision, not a compromise

Once the budget is the constraint, capacity has to be spent rather than simply added. That is the question behind BoltVision, where we compared a convolutional network, a compact convolutional transformer and a vision transformer on the same missing-bolt classification task. The three differ mainly in how they get from pixels to tokens and how much data they need before that choice pays off.

Three ways to get from pixels to a prediction Three stacked pipelines. A convolutional network runs a stack of convolutions into pooling and a classifier head. A compact convolutional transformer uses a convolutional tokeniser, then a transformer encoder, then sequence pooling. A vision transformer splits the image into fixed patches, embeds them, then runs a transformer encoder with a class token. The diagram shows structure only; it makes no claim about which performs better. CNN conv stack pooling classifier head CCT conv tokeniser transformer encoder sequence pooling ViT fixed patches transformer encoder class token Structure only — no performance claim is made by this diagram.
Figure 2: the three architectures compared in BoltVision differ mainly in how pixels become tokens. That choice, more than raw parameter count, drives how much data each one needs.

The convolutional tokeniser in the middle row is the interesting one for inspection work: it keeps the local inductive bias that convolutions give you for free, which matters when your dataset is a few thousand images of one component rather than a web-scale corpus.

Quantisation is a trade, and small objects pay for it

Dropping to FP16 or INT8 is the standard move for edge throughput, and it is close to free on large, well-separated objects. It isn’t free on the things inspection cares about. A defect occupying a few dozen pixels has little signal to lose before it stops being detectable, and INT8 in particular can quietly cost you recall on exactly that slice while aggregate metrics barely move.

The mitigation is not to avoid quantisation. It is to calibrate on data that contains the hard cases, and to re-measure per slice afterwards rather than trusting the overall number. I wrote about why aggregate metrics hide this in Edge CV under domain shift.

The device gets hot, and then it gets slower

A benchmark runs for thirty seconds. An inspection rig runs for a shift. Sustained throughput on a thermally-constrained device is not the number you measured in the first minute, and a model sized to fit the budget when cold will miss it when warm. Benchmark long, benchmark warm, and leave headroom you do not think you need.

“Lightweight” in a title is not modesty. It is the deployment target showing through into the architecture.

What this means in practice

None of this makes for an exciting leaderboard entry. It is, in my experience, most of the difference between a model that demos and a model that runs.

Written by Mujadded Al Rabbani Alif

More notes Papers Case studies Email me