Explainable Interactive Projections of Images
Published in Machine Vision and Aplications, 2023
Authors:Huimin Han Rebecca Faust, Brian Felipe Keith Norambuena, Jiayue Lin, Song Li, and Chris North.
Dimension reductions (DR) help people make sense of image collections by organizing images in the 2D space based on similarities. However, they provide little support for explaining why images were placed together or apart in the 2D space. Additionally, they do not provide support for modifying and updating the 2D representation to explore new relationships and organizations of images. To address these problems, we present an interactive DR method for images that uses visual features extracted by a deep neural network to project the images into 2D space and provides visual explanations of image features that contributed to the 2D location. In addition, it allows people to directly manipulate the 2D projection space to define alternative relationships and explore subsequent projections of the images. With an iterative cycle of semantic interaction and explainable-AI feedback, people can explore complex visual relationships in image data. Our approach to human–AI interaction integrates visual knowledge from both human-mental models and pre-trained deep neural models to explore image data. We demonstrate our method through examples with collaborators in agricultural science and other applications. Additionally, we present a quantitative evaluation that assesses how well our method captures and incorporates feedback.
Recommended citation: H. Han, R. Faust, B. Keith Norambuena, J. Lin, and C. North. Explainable interactive projections of images. Machine Vision and Applications 34, 100 (2023). https://doi.org/10.1007/s00138-023-01452-9
Prior Conference Version: Explainable Interactive Projections for Image Data. Presented at the International Symposium on Visual Computing (ISVC).