The ERMIS project has conducted a systematic analysis of speech and facial input signals, in separate, as well as in common; the aim was to extract parameters and features which can provide human computer interaction (HCI) systems with the ability to recognize the basic emotional state of their users and interact with them in a more natural and user friendly way. Testbed applications have been selected for testing and evaluating the ERMIS system performance, referring to everyday interaction of users with their PCs and with service or information providing call centers, with successful developments prospecting a large market size.
Linguistic and paralinguistic analysis of speech have been investigated systematically, creating a prototype that is able to analyze and respond to its users' commands, taking into account the cues about their emotional state. Analysis of facial expressions, specially in the framework of the MPEG-4 standard, constitutes another input for retrieving cues about the user's emotional state. Facial expression analysis has been applied separately, or combined with emotional speech analysis.
The ERMIS system is able to rely on prior knowledge related to the emotional analysis of speech and/or facial expressions, and to accommodate for the different expressive styles of humans. The continuity of emotion space, the uncertainty involved in the feature estimation process and the required ability of the system to use prior knowledge, while being also capable of adapting its behaviour to its users' characteristics, is handled by using intelligent hybrid, neurofuzzy, approaches.
A crucial criterion for the success of the project developments has to do with the effectiveness and the improvement of user friendliness achieved by the proposed system when compared with the current state of Human Computer Interaction. User research and testing has been carried out by the project industrial partners and subcontractors, evaluating user acceptance and related ethical issues.