1. Introduction
The ubiquity of Internet-of-Things (IoT) ecosystems and autonomous systems has dramatically amplified the demand for real-time deep neural network (DNN) processing directly on localized edge devices. Conventional architectures rely heavily on offloading computational workloads to remote cloud server clusters. However, variable network bandwidth, stringent data sovereignty regulations, and unpredictable transmission latencies render cloud-dependent offloading sub-optimal for mission-critical scenarios such as autonomous drone navigation, robotic surgery, and distributed industrial telemetry.
Modern convolutional networks and transformer vision models possess tens of millions of floating-point parameters, consuming substantial DRAM bandwidth and inducing significant thermal throttling on constrained embedded micro-architectures. Model compression strategies—predominantly post-training quantization and structured channel pruning—offer promising alternatives to reduce arithmetic complexity. However, naive low-bit integer representations often produce unacceptable drops in task-specific generalization, especially when operating on non-uniform sensor input streams.
2. System Architecture & Formal Quantization Model
To overcome precision degradation while optimizing throughput, we introduce a structured hierarchical quantization pipeline designed for heterogeneous embedded platforms. Our formulation decomposes the tensor weight parameter space into localized sensitivity tiers, dynamically assigning 8-bit integer (INT8) configurations to resilient convolutional kernels while reserving 16-bit floating-point (FP16) precision for sensitive projection layers.
We combine this non-linear quantization function with iterative structured channel pruning, removing redundant computational kernels according to Taylor-expansion sensitivity estimates. This pruning mechanism reduces memory footprints without disrupting contiguous memory addresses, ensuring high SIMD parallelization on ARM Cortex-A and embedded GPU accelerators.
3. Experimental Setup & Distributed Testbeds
We evaluated the proposed framework across an extensive distributed hardware cluster comprising three heterogeneous edge hardware tiers:
- Tier 1: High-Performance Edge Accelerator: NVIDIA Jetson Orin Nano (6-core ARM Cortex-A78AE, 1024-core Ampere GPU, 8 GB LPDDR5).
- Tier 2: Mid-Range Single-Board Computer: Raspberry Pi 5 (Quad-core Broadcom BCM2712 ARM Cortex-A76 at 2.4 GHz, 8 GB RAM).
- Tier 3: Constrained Industrial Micro-controller: STM32MP157 Microprocessor (Dual ARM Cortex-A7 with dedicated DSP co-processors).
Evaluations were conducted across standardized computer vision benchmarks, including ImageNet-1K (1.28 million training images, 50,000 validation images) and the MS COCO 2017 detection database. Inference execution latency was measured using high-precision hardware oscilloscopes and hardware performance counters over 1,000 consecutive test cycles.
4. Empirical Results & Latency Analysis
Empirical measurements confirmed substantial efficiency improvements across all participating testbed nodes. Compared to baseline 32-bit floating-point (FP32) baselines, our hybrid quantization framework achieved an aggregate 3.4× speedup in end-to-end inference frame rates on Tier 1 hardware, reducing 99th-percentile tail latency from 38.4 ms to 11.2 ms.
Importantly, the classification degradation was exceptionally minimal: Top-1 ImageNet accuracy decreased by only 0.42% under full INT8 deployment, compared to over 4.8% degradation observed in standard uniform quantization baselines. Thermal profiling revealed a 44% reduction in peak power consumption (Joules per frame), preventing thermal frequency scaling during prolonged continuous operation.
5. Discussion & Future Research Directions
The findings substantiate our core hypothesis: model compression frameworks that combine sensitivity-aware clipping with structured hardware-aligned pruning can preserve state-of-the-art accuracy while delivering sub-15ms real-time latency on low-cost hardware. Unlike unstructured weight pruning—which yields sparse matrices that contemporary mobile hardware cannot accelerate efficiently—our structured approach yields dense tensor shapes directly aligned with ARM NEON and Tensor Core SIMD architectures.
Future extensions will expand this quantization paradigm to autoregressive multimodal foundational models and explore decentralized Federated Learning convergence on resource-constrained embedded networks.
6. Declarations & Disclosures
Funding: This research was supported in part by the Advanced Systems Research Grant (Grant ID: ASR-2025-9411) and the Consortium for Distributed Edge Computing.
Data Availability: Benchmark validation scripts, configuration files, and quantization models are archived and publicly accessible via the open repository at DOI: 10.55555/rs.eng-tech.2026.1.1.001-data.
Conflicts of Interest: The authors declare that they have no competing financial or commercial conflicts of interest that could influence the results reported in this paper.