Machine Learning Techniques for Inferring the Museum Visitors’ Behavior

The awareness of the visitor’s role centrality in museums has led to the constant development of techniques and methodologies aimed at improving the visitor’s experience. In this context, Augello et al. suggest the four pillars to enhance the user experience: User Localization, User Understanding, Multimodal Interaction, and Gamification. As part of our research activities, we have focused on the latter two pillars. More specifically, User Localization allows museum curators and staff to understand the user movements in the museum and how much time she spends in front of the artworks. This information can help personalize the user experience to make it more and more satisfying. To collect this information, we believe that rather than using traditional techniques, such as beacons or IR, it is better to leverage deep learning techniques using cameras, thus supporting subsequent analyzes such as an automatic analysis of the user’s face. In particular, analyzing the user’s face allows us to grasp her reactions in front of the artworks and establish whether she may have liked that artwork or not, and if she may want to see similar artworks. The aim of this research is to understand how to use localization systems and automatic face analysis to build a user model that could allow museum staff to customize her experience. The first results obtained regarding an innovative localization system, based on deep learning, show the potential of this technique, which permits the extraction of information from faces through the same cameras. As regards this second part, techniques based on deep neural networks, such as Convolutional and Recurrent Neural Networks, are being tested to identify the best approach to extract implicit feedback from users’ faces without making them fill out long questionnaires that require active participation and steal time from the visit.