HANS: A framework for distributed ANN inference on heterogeneous edge devices
Due to large networks of sensors and actuators, thanks to the flourishing field of IoT, a lot of data would be created and consumed. Most of such data would be utilized to train artificial neural network models to realize real-time inference for environmental analysis. Most of the time, such inference will be done on the edge, i.e. a computing node that is near to the end device. These edge devices can be heterogeneous, i.e. they can be any device like General Purpose Processors (GPPs), Graphical Processing Units (GPUs), Field Programmable Gate Arrays (FPGAs), etc. These devices might differ in their computation power, adaptability to run ANN models, and the ease of programming.
Therefore, it is necessary to have a framework that provides a higher-level API to map a given trained neural network and present a trade-off analysis between the communication and the computation overhead. Given the network description, we would generate a Synchronous Data Flow Graph which would be fine-grained to the basic operation level (like addition and multiplication). We would then group these operations into clusters such that the optimal mapping of these operations onto the provided resources, given their compute power and parallelization capabilities, is achieved. The capabilities might or might not be beneficial in a particular configuration when the related inter-device communication is taken into account. At the operation node level, target architecture-specific implementation also has to be taken care of, along with the inter-device communication which might include intermediate data. Depending on the user’s requirements and the resource constraints provided by them, the optimal mapping of the individual operations within or across the layers of the ANN onto the heterogeneous devices can be done.
- Funded by the Software Campus, supported by the BMBF and co-mentored by Huawei.
Research Partner (TU Dresden):
- Prof. Dr. Akash Kumar (Head of Supervision)
- Mr. Shubham Rai (PhD Supervisor-Management)
- Mr. Ansh Rupani (Master Student: Software modelling and implementation)
- Mr. Yuhao Liu (PhD Employee: Hardware modelling and implementation)
Industry Partner (Huawei-Munich/Dresden)
- Dr. Osama Abboud (Management and industrial supervision)