Edge AI and on-device ML quiz

Question 1

What is the primary motivation for running ML inference on-device (edge) rather than in the cloud?

Accepted Answer

Lower latency, offline capability, and data privacy

Answer

Unlimited compute and memory compared to the cloud

Answer

Guaranteed higher model accuracy than cloud models

Answer

Elimination of the need for model quantization

Question 2

In post-training quantization to INT8, what role does the calibration dataset play?

Accepted Answer

It estimates activation ranges to compute scale and zero-point

Answer

It retrains the model weights end-to-end

Answer

It compresses weights via Huffman coding

Answer

It is required only for floating-point inference

Question 3

What is the key difference between post-training quantization (PTQ) and quantization-aware training (QAT)?

Accepted Answer

QAT simulates quantization during training; PTQ quantizes after training without gradients

Answer

PTQ requires labeled data while QAT is unsupervised

Answer

QAT only works for weights, PTQ only for activations

Answer

There is no accuracy difference between them

Question 4

Why is per-channel (per-axis) quantization often preferred over per-tensor quantization for convolution weights?

Accepted Answer

Channels have differing weight ranges, so per-channel scales preserve resolution

Answer

Per-channel uses fewer bits than per-tensor

Answer

Per-tensor quantization is not supported on any hardware

Answer

Per-channel removes the need for a zero-point

Question 5

What is structured vs. unstructured pruning, and why does structured pruning typically yield real edge speedups?

Accepted Answer

Structured removes whole channels/filters so dense kernels speed up directly

Answer

Unstructured pruning always gives larger speedups on edge CPUs

Answer

Both require dedicated sparse-tensor hardware to gain anything

Answer

Structured pruning increases the parameter count

Question 6

In knowledge distillation for edge deployment, what is the purpose of the temperature parameter in the softmax?

Accepted Answer

It softens teacher logits to reveal inter-class similarity (dark knowledge)

Answer

It increases the learning rate of the student

Answer

It controls the dropout probability during distillation

Answer

It quantizes the teacher's outputs to INT8

Question 7

What does the 'roofline model' tell you when optimizing an edge ML kernel?

Accepted Answer

Whether a kernel is memory-bound or compute-bound via arithmetic intensity

Answer

The optimal learning rate schedule for training

Answer

The exact INT8 accuracy drop expected

Answer

The number of model parameters that fit in flash

Question 8

Why are depthwise separable convolutions (as in MobileNet) efficient for edge devices?

Accepted Answer

They factorize conv into depthwise + pointwise, cutting FLOPs roughly k^2x

Answer

They increase the receptive field without any compute cost

Answer

They eliminate the need for activation functions

Answer

They are only useful during training, not inference

Question 9

What is operator (kernel) fusion and why does it matter on edge accelerators?

Accepted Answer

Merging ops to avoid DRAM round-trips for intermediate tensors

Answer

Splitting one op across multiple devices

Answer

Converting integer ops back to floating point

Answer

Doubling the batch size automatically

Question 10

What is the function of a 'delegate' in TensorFlow Lite (LiteRT) on mobile devices?

Accepted Answer

It routes supported subgraphs to GPU/NPU/DSP backends with CPU fallback

Answer

It compresses the .tflite file on disk

Answer

It performs over-the-air model updates

Answer

It collects user telemetry for retraining

Question 11

On Apple devices, what is the Core ML compute units setting 'all' likely to use, and what is a risk of forcing the Neural Engine?

Accepted Answer

ANE has a limited op set; relying on it can cause fallbacks and transfer overhead

Answer

The ANE supports every op so forcing it is always fastest

Answer

ANE only runs FP64 models

Answer

Core ML cannot use the GPU at all

Question 12

For deploying a quantized LLM on a phone, what does 4-bit weight-only quantization (e.g., GPTQ/AWQ-style) optimize, and why does it help edge inference?

Accepted Answer

Shrinks memory and boosts bandwidth-bound decode by storing weights in 4 bits

Answer

Reduces FLOPs to zero by removing matmuls

Answer

Trains the model faster on the device

Answer

Eliminates the KV cache entirely

Question 13

Why does the KV cache dominate memory during on-device LLM generation, and what mitigations exist?

Accepted Answer

It grows with sequence length x layers x heads; mitigated by GQA, KV quant, windowing

Answer

It is fixed-size regardless of context length

Answer

It only exists during training, not inference

Answer

It is eliminated by INT8 weight quantization

Question 14

What is the difference between latency and throughput optimization for an edge inference workload, and when do they conflict?

Accepted Answer

Batching boosts throughput but increases single-request latency

Answer

They are always optimized by the same techniques

Answer

Throughput only matters on edge, latency only in cloud

Answer

Lowering latency always raises throughput proportionally

Question 15

What is 'cold start' latency in on-device ML and how can it be reduced?

Accepted Answer

First-inference load/compile cost; mitigated by warm-up and precompiled graphs

Answer

The time to download the app from the store

Answer

The energy used while the device is asleep

Answer

The accuracy drop after quantization

Question 16

In edge TinyML on microcontrollers (e.g., TFLite Micro), why is there typically no dynamic memory allocation at inference time?

Accepted Answer

A fixed pre-allocated arena avoids fragmentation and OOM on tiny RAM

Answer

MCUs have unlimited heap so allocation is free

Answer

Dynamic allocation is required for INT8 inference

Answer

Static memory is only used during training

Question 17

What metric should you optimize to maximize battery life for a continuously running edge model, and why is it not just latency?

Accepted Answer

Energy per inference (joules), since power x time and memory traffic vary

Answer

Only raw FLOP count, which fully determines energy

Answer

Model file size on disk

Answer

Only inference latency in milliseconds

Question 18

What is the role of NNAPI on Android and why is its deprecation relevant for ML engineers?

Accepted Answer

NNAPI abstracted HW acceleration but is deprecated; use vendor/LiteRT delegates

Answer

NNAPI is the only supported path on Android 15+

Answer

NNAPI handles model training on-device

Answer

NNAPI is Apple's neural acceleration API

Question 19

What is federated learning and what core problem does it solve for edge ML?

Accepted Answer

Training on-device with only model updates shared, keeping raw data local

Answer

Sending all user data to the cloud for centralized training

Answer

Compressing models to 4-bit precision

Answer

Running inference without any model on the device

Question 20

In federated learning, why is secure aggregation important even though raw data stays on device?

Accepted Answer

Individual updates can leak data; aggregation hides per-client contributions

Answer

It speeds up training by skipping encryption

Answer

It is only needed when data is sent to the cloud

Answer

It reduces the model size for transfer

Question 21

What is the straight-through estimator (STE) and why is it needed in quantization-aware training?

Accepted Answer

It approximates the rounding gradient as identity so gradients can flow

Answer

It increases precision back to FP32 during inference

Answer

It is a pruning criterion for removing channels

Answer

It encrypts gradients during federated training

Question 22

Why can converting a model to a fixed input shape (static shapes) improve edge inference performance?

Accepted Answer

Static shapes enable memory planning, kernel selection, and NPU acceleration

Answer

Dynamic shapes are always faster on NPUs

Answer

Static shapes increase model accuracy

Answer

Shape information has no effect on performance

Question 23

What is the difference between weight-only quantization and full-integer (W8A8) quantization, and where is each used?

Accepted Answer

Weight-only keeps FP compute; W8A8 does integer compute for NPUs/MCUs

Answer

They are identical in implementation

Answer

Weight-only requires activation calibration but W8A8 does not

Answer

W8A8 only quantizes the bias terms

Question 24

For on-device speech wake-word detection, why is a two-stage (cascade) architecture commonly used?

Accepted Answer

A tiny always-on model gates a larger verifier to save power

Answer

One huge model runs continuously for best accuracy

Answer

Cascades are used only to increase latency

Answer

Both stages run in the cloud

Question 25

What is the impact of memory bandwidth on transformer decode latency on edge devices, and how does speculative decoding help?

Accepted Answer

Decode is bandwidth-bound; speculative decoding verifies multiple draft tokens at once

Answer

Decode is compute-bound and speculative decoding adds more FLOPs

Answer

Speculative decoding changes the output distribution to save memory

Answer

It eliminates the need for the KV cache

Question 26

Why might INT8 quantization of a model with many residual/skip connections or attention require special handling?

Accepted Answer

Large dynamic ranges/outliers need mixed precision or outlier handling (e.g., SmoothQuant)

Answer

Residual connections cannot be quantized at all

Answer

INT8 always overflows so the model must stay FP32

Answer

Attention layers are immune to quantization error

Question 27

What is the purpose of the ONNX format and ONNX Runtime in an edge deployment pipeline?

Accepted Answer

A portable model format/runtime targeting multiple hardware execution providers

Answer

A quantization algorithm for INT4 weights

Answer

A federated learning aggregation protocol

Answer

An Android-only inference API

Question 28

When profiling an edge model you find ops are split between an NPU delegate and CPU with frequent fallbacks. Why is this often slower than CPU-only?

Accepted Answer

Cross-device tensor transfers and graph fragmentation outweigh per-op speedups

Answer

NPUs are always slower than CPUs

Answer

Fallbacks improve accuracy but cost latency

Answer

CPU cannot run any quantized ops

Question 29

What is LoRA and why is it attractive for personalizing/adapting LLMs on or for edge devices?

Accepted Answer

Low-rank adapters give tiny, swappable fine-tunes over a frozen base model

Answer

It quantizes the base model to 1-bit

Answer

It removes attention layers to shrink the model

Answer

It is a federated aggregation method

Question 30

Why is FP16 (or BF16) commonly used on mobile GPUs/NPUs instead of FP32, and what is a key numerical caveat?

Accepted Answer

Half precision halves bandwidth but risks overflow/underflow needing FP32 accumulation

Answer

FP16 has more precision than FP32

Answer

BF16 and FP16 are bit-identical

Answer

Half precision is only used during training

Test yourself