Are you paying attention when you read? PhD candidates Yoon Lee and Xuan Liu are researching a way to detect that and provide feedback.
Xuan Liu (center) and Yoon Lee (right) were demonstrators at the Hybrid Intelligence Consortium. (Photo: Bart Vastenhouw)
Yoon: “I’m from South Korea and I’ve been at TU Delft for five years. I did a master’s at the Faculty of Industrial Design Engineering and then I started my PhD at EEMCS (Faculty of Electrical Engineering, Mathematics & Computer Science) in 2019. I’m in my third year so I have one and half years left if everything goes smoothly.”
Xuan: “I’m a visiting PhD student at TU Delft. I have been here almost a year and my visit is almost over so I will leave in May. I come from China, from the Harbin Institute of Technology. I’m a third year PhD student and I mainly work on facial expression recognition and detection. I’m working at EEMCS in the interactive intelligence group where we focus on making robots more socially interactive with people.”
Yoon: “We are working on a collaborative project that has two dimensions. Both are based on multimodality which, if we think of human senses, is like multiple traits that we can get from visual, auditory and tactile stimuli. We are combining those multimodal cues to detect the attention status of readers. At the same time, we aim for managing human attention and learning behaviour by providing multimodal feedback. Xuan is in charge of the detection part and I’m in charge of learning theories and feedback design.”
Xuan: “Now we are applying this to a reading scenario by building a reading companion. When people are reading something, we can use this device to detect their reading status, like their attention level, if their mind is wandering or if they are bored. With social robots we can detect things and react by giving some suggestions or asking some questions.”
‘If people don’t need to drive any more in some cars, what will they do?’
Yoon: “To give you a little bit of context, we started from the idea of human attention during computer mediated study. In lectures during the pandemic and with the current transition of education, computer mediated education is more prevalent. Whether it’s reading or watching lectures, people’s attention levels are different than in person. If you’re in a lecture hall, an educator can measure if students are focussed or not and then they can adjust their interaction or change the content to make it better. For computer mediated study it’s more difficult because there’s no detection mechanism. It’s a whole different architecture than a real life setting so that’s why we started this project.
For now, this is being applied in the education field, but I’m interested in applying it in different fields as well. For example, electric cars are becoming more popular, and if people don’t need to drive any more in some cars, what will they do? They might be involved in entertainment or education, and I think this kind of application can be used in this scenario as well.”
Xuan: “And even with current cars where a driver is needed, we can use this application to detect their attention level and give feedback. For example, if it detects they are tired it could tell them to rest a bit. Or if there is an emergency the robot could offer some help.
Our goal is to collect more data before I leave and when I return back to China we can continue working on the analysis of this data.
When I arrived at TU Delft there was still a lockdown, but when things opened, I experienced the working atmosphere here and I liked it. The thing I like most about TU Delft is that people collaborate with each other very easily and often. We all are from different disciplines and we are collaborating. This is a very good experience. For research, I think we need to collaborate more and help each other.”
Want to be featured in Humans of TU Delft? Or do you know someone with a good story to tell? Send us an e-mail at humansoftudelft@gmail.com
UPDATE 11-5
After receiving a number of questions about the research discussed in this article, Delta asked the researchers for clarification on the funding of this research, the use of the collected data, and the potential for undesirable knowledge transfer to the Harbin Institute of Technology, a university known to have close ties with the Chinese army.
According to the Faculty of Electrical Engineering, Mathematics and Computer Science (EEMCS), the research is funded by the Faculty itself and co-funded by the China Scholarship Council, an institution that gives many young Chinese students grants to cover their expenses, visas and board and lodgings when they study abroad.
The Faculty says that the data is used for research into learning support for individual students. “It is handled according to our data-management plan devised in connection with the TU Delft ethics clearance. No other entities beyond TU Delft will have access to it.” Regarding the potential use or misuse of the research and its data, and the risk of undesirable knowledge transfer, EEMCS also refers to the ethical clearance it went through. “We follow the policies of TU Delft on these matters.”
Heather Montague / Freelance writer
Comments are closed.