A method of sensing an interaction with an object using a semitransparent tactile surface (STS) sensor having an image sensor and a semitransparent membrane includes capturing, by the image sensor, a stream of images; separating the stream of images into a visual stream including light traveling through the semitransparent membrane and a tactile stream including light reflecting off of the semitransparent membrane; and processing the visual stream and tactile stream through a multimodal deep neural network.