LapiDude: Biometric-Authenticated Multilingual Framework for Secure Desktop Automation

Over the years, voice-based interaction has become an important interface paradigm in today’s computing systems. Voice interactions has become widespread with mobile devices and structured on smart assistants, desktop environments still depends on manual input via keyboards and pointing devices. Additionally, since most desktop automation tools do not even check whether the user sending commands is who he/she claims to be, there are security implications when performing operations on a system level are executed through automated processes. Also, another gap of current systems which makes them not much efficient is that they do not provide multilingual corpus since a majority of available voice interaction framework are designed to be helpful only for English based users. This work proposes a new biometric-authenticated multilingual framework, LapiDude for secure voice-driven desktop automation. It utilizes speaker verification models on top of multi-lingual command recognition to allow users to control desktop applications with natural spoken phrases, but only allows authorized speakers to issue commands. We use deep speaker embeddings from a pretrained voice representation to perform speaker verification. It uses cosine similarity to match incoming voice samples against stored administrator voiceprints in order to authenticate identity. The spoken command gets transformed to text post successful authentication and subsequently passed through a multilingual