Neuroadaptive technology (NAT) utilizes real-time measures of neurophysiological activity
within a closed control loop to create intelligent software adaptation. The
Neuroadaptive Technology Conference 2022
took place in Lübbenau (Spreewald) from October 10th-11th 2022.
Members of the KD2School were present at the conference and presented research on adaptive systems:
Ivucic, G., Putze, F., Cai, S., Li, H., and Schultz T. (2022). Interpretable Deep Neural Networks for EEG-based Auditory Attention Detection with Layer-Wise Relevance Propagation
Deep Neural Networks (DNNs) recently found their way into cognitive neuroscience serving as
powerful computational models. However, the complexity of deep learning models results in an
uninterpretable black box, preventing neurophysiological insight into processes behind the decision
of the model. In this work, we present an explanation approach for a DNN in spatial auditory attention detection
(AAD) with electroencephalography (EEG), based on Layer-Wise Relevance Propagation (LRP). LRP
decomposes the prediction of the DNN into relevance heatmaps that represent the importance of
the spectro-spatial image features regarding the decision of the network, illustrated in Figure 1. To
validate the LRP explanation for the DNN, (1) the relation between relevance heatmaps and the
output of the network is examined via relevance-guided input perturbation. Further, (2) structural
features and potential prediction strategies in the LRP heatmaps are investigated by spectral
clustering of relevance heatmaps. The results indicate that explanation heatmaps generated by LRP highlight areas in the cortical
activation images that predominantly impact the decision of the network. The clustering approach
found distinct patterns in relevance maps, individually for each subject, revealing the importance of
neuro-physiologically plausible frontal, lateral, and rear brain areas for auditory attention.
This work demonstrates that LRP can fill the interpretability gap in the development of DNNs for EEGbased AAD. The relevance heatmaps of single input samples combined with the knowledge of global
prediction strategies open up the ability to investigate sample groups of interest at will, which
renders LRP as a tool to reveal potential neural- or decisional processes underlying the deep learning
Seitz, J., Reuscher, T. F., Jacob, S., and Maedche, A. (2022). Towards an Awkward Silence-Adaptive Virtual Meeting System.
In times of worldwide confinements and global crisis, virtual meetings are increasingly an alternative
to in-person meetings. Within virtual meetings, moments of silence can occur due to several reasons
like connection issues, several persons starting to speak or none starts to speak (see e.g. ). As
participants in meetings follow the obligation to avoid interaction gaps, silent moments might cause a
state of anxiety referred to “awkward silence” and lead to unconscious behavior such as laughing or
being embarrassed –. This so-called state anxiety differs from anxiety disorders and is a shortterm emotional state . According to , people tend to feel awkward after a certain duration of
silence. However, this does not necessarily mean that everyone feels awkward at the same time as
individual and contextual factors play an important role. Existing approaches to encounter this
awkward silence have created conversational agents or topic proposals that aim to break the silence
(see e.g. –). However, recent literature suggests that silence itself is not necessarily bad as it
helps to increase creative solutions and can ultimately lead to better meeting outcomes . To
incorporate the idea of embracing the silence, we aim to build an adaptive system that can detect and
respond to moments of awkward silence by reducing the feeling of state anxiety. In this work-inprogress paper, we report on a first design prototype and a pre-tested experimental design to create
silent moments while collecting and analyzing participants’ physiological data.
A recent publication of one of KD2School's associates has been published focusing
on the emerging topic of ear-based wearable sensors (short: earables).
Earables have emerged as a unique platform for ubiquitous computing by augmenting
ear-worn devices with state-of-the-art sensing. This new platform has spurred a
wealth of new research exploring what can be detected on a wearable, small form factor.
As a sensing platform, the ears are less susceptible to motion artifacts and are
located in close proximity to a number of important anatomical structures including
the brain, blood vessels, and facial muscles which reveal a wealth of information.
They can be easily reached by the hands and the ear canal itself is affected by mouth,
face, and head movements. We have conducted a systematic literature review of
271 earable publications from the ACM and IEEE libraries. These were synthesized
into an open-ended taxonomy of 47 different phenomena that can be sensed in, on,
or around the ear. Through analysis, we identify 13 fundamental phenomena from
which all other phenomena can be derived, and discuss the different sensors and
sensing principles used to detect them. We comprehensively review the phenomena
in four main areas of (i) physiological monitoring and health, (ii) movement and
activity, (iii) interaction, and (iv) authentication and identification.
This breadth highlights the potential that earables have to offer
as a ubiquitous, general-purpose platform.
June 01, 2022
KD2School Goes to NeuroIS
With June we see the first acceptance series of conference articles by several KD2School members and associates.
At the 12th annual retreat on neuro-information systems (NeuroIS), an international group of
scholars is gathering to discuss conceptual and empirical works, as well a theoretical and design science research.
Thereby, NeuroIS work includes research based on all types of neuroscience and physiological methods.
With numerous contributions we will participate in this timely discussion revolving around how adaptive
systems can influence economic decision making and everyday experience. The submissions to be presented are:
- Seitz, J., and Maedche, A. (2022) Biosignal-based Recognition of Cognitive Load: A Systematic Review of Public Datasets and Classifiers.
- Langner, M. et al. (2022) Recognizing Polychronic-Monochronic Tendency of Individuals using Eye Tracking and Machine Learning.
- Bartholomeyczik, K. et al. (2022) Flow in Knowledge Work: An Initial Evaluation of Flow Psychophysiology Across Three Cognitive Tasks.
- Nikolajevic, N., et al. (2022). Facilitating NeuroIS Research using Natural Language Processing: Towards Automated Recommendations.
- Greif-Winzrieth, A., et al. (2022) A Participant’s View on the Potential of Conducting NeuroIS Studies in the Wild.
Information dashboards are a critical capability in contemporary business intelligence
& analytics systems. Despite coming with strong potential to support better decision
making, the huge amount of information provided challenges their users when they perform
data exploration tasks. Accordingly, dashboard users face difficulties in managing their
limited attentional resources when processing the presented information on dashboards.
Also, recent studies show that the amount of concentrated time humans can spend on a
task is reduced massively and there is a need for designing user interfaces that support
their users' attention management. Therefore, in this design science research project,
we propose attentive information dashboards that provide individualized visual attention
feedback (VAF) as an innovative artifact to solve this problem. We articulate
theoretically grounded design principles and instantiate a software artifact leveraging
users' eye movement gaze data in real-time to provide individualized VAF.
The instantiated artifact was evaluated in a controlled lab experiment with 92
The results from analyzing users' eye movement after receiving
individualized VAF reveal that our proposed design has a positive effect on users'
attentional resource allocation, attention shift rate, and attentional resource
management. We contribute a system architecture for attentive information dashboards
that support data exploration and two theoretically grounded design principles that
provide prescriptive knowledge on how to provide individualized VAF.
Further, practitioners can leverage the prescriptive knowledge derived from our research
and design innovative systems that support users' information processing by managing
their limited attentional resources.
With the advent of physiological computing systems, new avenues are emerging for the
field of learning analytics related to the potential integration of physiological data.
To this end, we developed a physiological computing infrastructure to collect
physiological data, surveys, and browsing behavior data to capture students’ learning
journey in remote learning. Specifically, our solution is based on the Raspberry Pi
minicomputer and Polar H10 chest belt.
In this work-in-progress paper, we present
preliminary results and experiences we collected from a field study with medical
students using our developed infrastructure. Our results do not only provide a new
direction for more effectively capturing different types of data in remote learning
by addressing the underlying challenges of remote setups, but also serve as a foundation
for future work on developing a less obtrusive, (near) real-time measurement method
based on the classification of cognitive-affective states such as flow or other
learning-relevant constructs with the captured data using supervised machine learning.
Leading teams effectively is an essential factor for team performance.
Main events where leaders practice leadership are video-meetings which are core
elements of remote teamwork. However, leading teams is challenging.
In particular, in video-meetings, leadership is complex due to limitations such as
aggravated understanding of social signals and team dynamics.
Voice-based smart personal assistants (SPA) such as Amazon’s Alexa can provide
real-time support and have the potential to maintain effective leadership in
video-meetings. Therefore, we developed the SPA LeadBoSki for real-time leadership
support in video-meetings. LeadBoSki detects leadership potentials in video-meetings
and sends leadership potential advice to the leader. Pilot study results with two
real-world teams (n=9) show a positive leader experience and stimulating behavior
by the leader for the team members’ motivation. Our work contributes with the SPA
LeadBoSki as the first SPA for leadership support in video-meetings of its kind and
as a potential use-case for future workplaces.