A Haptic Memory Game using the STReSS2 Tactile Display

Qi Wang, Vincent Levesque, Jerome Pasquero and Vincent Hayward
Haptics Laboratory, McGill University, Montreal, Canada
CHI '06 extended abstracts on Human factors in computing systems.

Other formats: ACM Digital Library, PDF, Video


A computer implementation of a classic memory card game was adapted to rely on touch rather than vision. Instead of memorizing pictures on cards, players explore tactile graphics on a computer-generated virtual surface. Tactile sensations are created by controlling dynamic, distributed lateral strain patterns on a fingerpad in contact with an electronic tactile display called STReSS2. The tactile graphics are explored by moving the device within the workspace of a 2D planar carrier. Three tactile rendering methods were developed and used to create distinct tactile memory cards. The haptic memory game showcases the capabilities of this novel tactile display technology. 

1. Introduction

The memory card game is played by randomly placing a set of cards face down on a table. Players turn over cards two at a time. If the pictures on the cards match, a point is scored and the pair is removed from the playing area. Pairs that do not match are turned back face down. To succeed, players must memorize the location of previously exposed cards. We revisited this classic game to explore the potential of a novel tactile display named STReSS2 (pronounced "stress-square") [9]. We replaced the pictures with computer-generated virtual tactile graphics. Using carefully designed stimulation patterns, we produced 12 distinct tactile cards. Instead of seeing the pictures on the cards, players explore their tactile equivalent with a finger.

2. Technology

Tactile displays are computer-driven transducers able to create tactile sensations on the fingerpad [1]. The STReSS2 is distinct from most other displays in that it takes advantage of the skin's sensitivity to distributed lateral deformation. The device has an active area of 12.0 x 10.8 mm, slightly larger than a fingerpad. It deforms the skin using a 10 x 6 array of piezoelectric bending motors (see Figure 1).

The display is mounted on a Pantograph haptic device used as a passive 2D planar carrier [2]. Players explore a 11.3 x 6.0 cm virtual surface by moving the display within the carrier's workspace with a finger. The fingerpad remains fixed on the display's active area. The skin deformation patterns are updated according to the exploratory movements, creating the sensation of sliding over embossed or textured virtual surfaces. Alternatively, the display can produce distributed vibratory patterns.

Figure 1a: A picture of the 10 by 6 actuators forming the active area of the STReSS2 tactile display.
Figure 1b: A picture of the tactile display mounted on a passive plannar carrier.
Figure 1c: A picture of a player with his finger on the tactile display.
Figure 1. STReSS2 tactile display: (a) active area, (b) display on carrier that allows movement in the horizontal plane, and (c) player's left hand with index on the display.

3. Tactile Rendering

The specification of programmed spatiotemporal actuator deflection patterns may be termed tactile rendering by analogy to graphics rendering. Three types of tactile rendering methods were developed. They are illustrated in Figure 2. In each case, tactile patterns are specified as grayscale images (e.g. Figure 2a), making it possible to quickly draw complex tactile graphics using standard painting software.

The first method divides the virtual surface into 94 x 33 cells, each containing a 1.2 x 1.8 mm tactile feature resembling an embossed dot (e.g. Figure 2b). The perceived height, or intensity, of each dot is controllable, giving rise to the tactile equivalent of a coarse grayscale image. The deflection of each actuator is continuously updated based on its location within the surface. As an actuator traverses a dot, it smoothly sweeps its entire range of motion. The experience of a dot results from the local skin stretch and compression patterns occurring locally between adjacent actuators. A similar method was previously used to display virtual Braille dots [5]. This mode is particularly adequate for the display of contours, edges, and letters. It produces tactile graphics comparable to those obtained with Braille printers [7].

The second type of rendering fills areas with a spatial texture resembling an embossed horizontal grating (e.g. Figure 2c). The grating is produced from a traveling wave moving on the display in response to exploratory movements. As the wave travels in the direction opposite to the finger movement, one has the sensation of sliding over a rippled surface. The amplitude of the texture is modulated over the virtual surface to form simple shapes. Tactile maps for the blind commonly make a similar use of texture to mark regions such as bodies of water [3].

The third approach replaces the spatial texture with an amplitude-modulated vibrotactile stimulus (e.g. Figure 2d). The vibrotactile stimulus is produced by driving each actuator with a 50 Hz sinusoidal signal. The phase is inverted between adjacent actuators to maximize strain. Unlike the two previous methods, vibration provides strong stimulation even in the absence of exploratory movements. The resulting sensation, however, is difficult to relate to a natural tactile stimulus. Vibration was found to reliably convey thick line drawings and contours. This mode of stimulation is similar to the one used by the Optacon, a vibrotactile reading aid for the blind [6]. The STReSS2, however, vibrates laterally instead of tapping against the skin, and allows more control over the stimulation frequency and amplitude. A similar use of image-based vibrotactile stimuli has also been made for the rendering of texture [4].

Figure 2a: A grayscale pattern consisting in a circle inside a square with rectangles of decreasing intensity on the sides.
Figure 2b: The same pattern illustrated with discrete dots.
Figure 2c: The same pattern illustrated with a grating texture.
Figure 2d: The same pattern illustrated with a time-varying texture representing vibrations.
Figure 2. (a) A pattern specified by a grayscale mask and a pictorial representation of its tactile rendering using (b) dots, (c) a grating texture or (d) vibration.

4. Haptic Memory Game

The game was implemented on a personal computer by replacing the pictures on the cards with tactile graphics. When the user clicks on a card using the mouse, the card becomes activated (highlighted) but its content is not revealed visually (see Figure 3). Instead, the player must explore an invisible tactile drawing.

The three modes of stimulation were used to design a set of 12 tactile memory cards represented pictorially in Figure 4. After a short training period, the cards can be distinguished from one another using tactile stimuli alone.

Figure 3a: A screenshot of the memory game showing an array of 6 by 4 cards. Two cards are shown face up, with the currently selected card highlighted in green. Both exposed cards are blank.
Figure 3b: Identical screenshot with the two exposed cards sporting illustrations.
Figure 3. Screenshots of the (a) haptic memory game and (b) its visual equivalent. The currently selected card is highlighted.

Figure 4: The set of 12 cards used for the game, with illustrations representing the corresponding tactile graphics. The six cards rendered with dots consist in an ellipse, a diamond, a triangle, a sequence of vertical lines, and the letters X and M. The three cards rendered with a grating texture consist in a sequence of 3 filled circles of varying size, a sequence of 3 filled squares of the same size, and a filled triangle. The three cards rendered with vibration consist in a set of six thick circles, a large circle, and the letter X.
Figure 4. Pictorial representation of 12 tactile cards selected for the memory game: dots, gratings, and vibration.

5. Conclusion

This paper introduced a new version of the memory card game that uses tactile feedback. The main purpose of the game is to exemplify the capabilities of the STReSS2 tactile display. The game demonstrates that the display can be used to produce convincing tactile graphics. Although some visual feedback is currently required to select cards, this game could also easily be adapted for visually impaired players [8].

The three rendering methods introduce basic building blocks for simple shapes, and eventually for more complex drawings. One could consider, for example, drawing a bicycle using vibrating wheels, a dotted frame and grating-textured ground. The Pantograph could also be used to provide additional force feedback. We expect future work to yield a wider range of expressive capabilities as well as applications to other areas of human-computer interaction.


The authors would like to thank the members of McGill's Haptics Laboratory as well as Danny Lynch and Prasun Lala for their help and support. This research was supported by NSERC, the Natural Sciences and Engineering Research Council of Canada.


[1] Benali-Khoudja, M., Hafez, M., Alexandre, J.-M. and Kheddar, A. Tactile Interfaces: a State-of-the-Art Survey. In Proc. ISR 2004 (2004), 721-726.

[2] Campion, G., Wang, Q., Hayward, V. The Pantograph Mk-II: A Haptic Instrument. In Proc. IROS 2005 (2005), 723-728.

[3] Edman, P. K. Tactile Graphics. American Foundation for the Blind (1992).

[4] Ikei, Y., Wakamatsu, K. and Fukuda, S. Vibratory tactile display of image-based textures. IEEE Computer Graphics and Applications 17, 6 (1997), 53-61.

[5] Levesque, V., Pasquero, J., Hayward, V. and Legault, M. Display Of Virtual Braille Dots By Lateral Skin Deformation: Feasibility Study. ACM Transactions on Applied Perception 2, 2 (2005), 132-149.

[6] Linvill, J.G. and Bliss, J.C. A direct translation reading aid for the blind. Proceedings of the IEEE 54, 1 (1966), 40-51.

[7] Nater, P. Tactile Graphics with the Aid of a Conventional Braille Printer. Journal of Microcomputer Applications 16 (1993), 307-314.

[8] Sjöström, C. Designing Haptic Computer Interfaces for Blind People. In Proc. ISSPA 2001 (2001), 1-4.

[9] Wang, Q. and Hayward, V. Compact, Portable, Modular, High-performance, Distributed Tactile Display Device Based on Lateral Skin Deformation. In Proc. HAPTICS 2006 (2006), 67-72.