Neuroadaptive technology (NAT) utilizes real-time measures of neurophysiological activity within a closed control loop to create intelligent software adaptation. The Neuroadaptive Technology Conference 2022 took place in Lübbenau (Spreewald) from October 10th-11th 2022. Members of the KD2School were present at the conference and presented research on adaptive systems:

Ivucic, G., Putze, F., Cai, S., Li, H., and Schultz T. (2022). Interpretable Deep Neural Networks for EEG-based Auditory Attention Detection with Layer-Wise Relevance Propagation

Deep Neural Networks (DNNs) recently found their way into cognitive neuroscience serving as powerful computational models. However, the complexity of deep learning models results in an uninterpretable black box, preventing neurophysiological insight into processes behind the decision of the model. In this work, we present an explanation approach for a DNN in spatial auditory attention detection (AAD) with electroencephalography (EEG), based on Layer-Wise Relevance Propagation (LRP). LRP decomposes the prediction of the DNN into relevance heatmaps that represent the importance of the spectro-spatial image features regarding the decision of the network, illustrated in Figure 1. To validate the LRP explanation for the DNN, (1) the relation between relevance heatmaps and the output of the network is examined via relevance-guided input perturbation. Further, (2) structural features and potential prediction strategies in the LRP heatmaps are investigated by spectral clustering of relevance heatmaps. The results indicate that explanation heatmaps generated by LRP highlight areas in the cortical activation images that predominantly impact the decision of the network. The clustering approach found distinct patterns in relevance maps, individually for each subject, revealing the importance of neuro-physiologically plausible frontal, lateral, and rear brain areas for auditory attention. This work demonstrates that LRP can fill the interpretability gap in the development of DNNs for EEGbased AAD. The relevance heatmaps of single input samples combined with the knowledge of global prediction strategies open up the ability to investigate sample groups of interest at will, which renders LRP as a tool to reveal potential neural- or decisional processes underlying the deep learning model.

Seitz, J., Reuscher, T. F., Jacob, S., and Maedche, A. (2022). Towards an Awkward Silence-Adaptive Virtual Meeting System.

In times of worldwide confinements and global crisis, virtual meetings are increasingly an alternative to in-person meetings. Within virtual meetings, moments of silence can occur due to several reasons like connection issues, several persons starting to speak or none starts to speak (see e.g. [1]). As participants in meetings follow the obligation to avoid interaction gaps, silent moments might cause a state of anxiety referred to “awkward silence” and lead to unconscious behavior such as laughing or being embarrassed [2]–[6]. This so-called state anxiety differs from anxiety disorders and is a shortterm emotional state [7]. According to [8], people tend to feel awkward after a certain duration of silence. However, this does not necessarily mean that everyone feels awkward at the same time as individual and contextual factors play an important role. Existing approaches to encounter this awkward silence have created conversational agents or topic proposals that aim to break the silence (see e.g. [9]–[12]). However, recent literature suggests that silence itself is not necessarily bad as it helps to increase creative solutions and can ultimately lead to better meeting outcomes [13]. To incorporate the idea of embracing the silence, we aim to build an adaptive system that can detect and respond to moments of awkward silence by reducing the feeling of state anxiety. In this work-inprogress paper, we report on a first design prototype and a pre-tested experimental design to create silent moments while collecting and analyzing participants’ physiological data.

A recent publication of one of KD2School's associates has been published focusing on the emerging topic of ear-based wearable sensors (short: earables). Earables have emerged as a unique platform for ubiquitous computing by augmenting ear-worn devices with state-of-the-art sensing. This new platform has spurred a wealth of new research exploring what can be detected on a wearable, small form factor. As a sensing platform, the ears are less susceptible to motion artifacts and are located in close proximity to a number of important anatomical structures including the brain, blood vessels, and facial muscles which reveal a wealth of information.

They can be easily reached by the hands and the ear canal itself is affected by mouth, face, and head movements. We have conducted a systematic literature review of 271 earable publications from the ACM and IEEE libraries. These were synthesized into an open-ended taxonomy of 47 different phenomena that can be sensed in, on, or around the ear. Through analysis, we identify 13 fundamental phenomena from which all other phenomena can be derived, and discuss the different sensors and sensing principles used to detect them. We comprehensively review the phenomena in four main areas of (i) physiological monitoring and health, (ii) movement and activity, (iii) interaction, and (iv) authentication and identification. This breadth highlights the potential that earables have to offer as a ubiquitous, general-purpose platform.

With June we see the first acceptance series of conference articles by several KD2School members and associates. At the 12th annual retreat on neuro-information systems (NeuroIS), an international group of scholars is gathering to discuss conceptual and empirical works, as well a theoretical and design science research. Thereby, NeuroIS work includes research based on all types of neuroscience and physiological methods. With numerous contributions we will participate in this timely discussion revolving around how adaptive systems can influence economic decision making and everyday experience. The submissions to be presented are:

• Seitz, J., and Maedche, A. (2022) Biosignal-based Recognition of Cognitive Load: A Systematic Review of Public Datasets and Classifiers.
• Langner, M. et al. (2022) Recognizing Polychronic-Monochronic Tendency of Individuals using Eye Tracking and Machine Learning.
• Bartholomeyczik, K. et al. (2022) Flow in Knowledge Work: An Initial Evaluation of Flow Psychophysiology Across Three Cognitive Tasks.
• Nikolajevic, N., et al. (2022). Facilitating NeuroIS Research using Natural Language Processing: Towards Automated Recommendations.
• Greif-Winzrieth, A., et al. (2022) A Participant’s View on the Potential of Conducting NeuroIS Studies in the Wild.

Information dashboards are a critical capability in contemporary business intelligence & analytics systems. Despite coming with strong potential to support better decision making, the huge amount of information provided challenges their users when they perform data exploration tasks. Accordingly, dashboard users face difficulties in managing their limited attentional resources when processing the presented information on dashboards. Also, recent studies show that the amount of concentrated time humans can spend on a task is reduced massively and there is a need for designing user interfaces that support their users' attention management. Therefore, in this design science research project, we propose attentive information dashboards that provide individualized visual attention feedback (VAF) as an innovative artifact to solve this problem. We articulate theoretically grounded design principles and instantiate a software artifact leveraging users' eye movement gaze data in real-time to provide individualized VAF. The instantiated artifact was evaluated in a controlled lab experiment with 92 participants.

The results from analyzing users' eye movement after receiving individualized VAF reveal that our proposed design has a positive effect on users' attentional resource allocation, attention shift rate, and attentional resource management. We contribute a system architecture for attentive information dashboards that support data exploration and two theoretically grounded design principles that provide prescriptive knowledge on how to provide individualized VAF. Further, practitioners can leverage the prescriptive knowledge derived from our research and design innovative systems that support users' information processing by managing their limited attentional resources.

With the advent of physiological computing systems, new avenues are emerging for the field of learning analytics related to the potential integration of physiological data. To this end, we developed a physiological computing infrastructure to collect physiological data, surveys, and browsing behavior data to capture students’ learning journey in remote learning. Specifically, our solution is based on the Raspberry Pi minicomputer and Polar H10 chest belt.

In this work-in-progress paper, we present preliminary results and experiences we collected from a field study with medical students using our developed infrastructure. Our results do not only provide a new direction for more effectively capturing different types of data in remote learning by addressing the underlying challenges of remote setups, but also serve as a foundation for future work on developing a less obtrusive, (near) real-time measurement method based on the classification of cognitive-affective states such as flow or other learning-relevant constructs with the captured data using supervised machine learning.

Leading teams effectively is an essential factor for team performance. Main events where leaders practice leadership are video-meetings which are core elements of remote teamwork. However, leading teams is challenging. In particular, in video-meetings, leadership is complex due to limitations such as aggravated understanding of social signals and team dynamics.

Voice-based smart personal assistants (SPA) such as Amazon’s Alexa can provide real-time support and have the potential to maintain effective leadership in video-meetings. Therefore, we developed the SPA LeadBoSki for real-time leadership support in video-meetings. LeadBoSki detects leadership potentials in video-meetings and sends leadership potential advice to the leader. Pilot study results with two real-world teams (n=9) show a positive leader experience and stimulating behavior by the leader for the team members’ motivation. Our work contributes with the SPA LeadBoSki as the first SPA for leadership support in video-meetings of its kind and as a potential use-case for future workplaces.