A Context-aware Benchmarking Approach for Scenario-based Evaluation of Smart Devices

Abstract

Understanding how smart devices behave under real-world conditions remains challenging, as existing benchmarking tools often isolate metrics like energy consumption or protocol compliance without considering realistic interactions or coordinated use. This paper presents a scenario-driven benchmarking framework tailored for smart devices which integrates control, orchestration, instrumentation, and monitoring into a unified, reproducible workflow. Our framework enables structured specification of interaction scenarios, precise scheduling of device actions, and synchronized measurement of energy usage and network activity. We validate our approach through a smart home testbed comprised of off-the shelf speakers, lights, and fans across scenarios that reflect typical user interactions and adversarial network conditions. Results reveal behavioral differences not evident from device specifications alone, highlighting the importance of context-rich, scenario-based evaluation for understanding smart device performance in practice.