Braille Display by Lateral Skin Deformation with the STReSS2 Tactile Transducer
Vincent Levesque, Jerome Pasquero and Vincent HaywardHaptics Laboratory, McGill University, Montreal, Canada
Proceedings of World Haptics 2007, March 22-24, Tsukuba, Japan.
Other formats: pdf
Abstract
Earlier work with a 1-D tactile transducer demonstrated that lateral skin deformation is sufficient to produce sensations similar to those felt when brushing a finger against a line of Braille dots. Here, we extend this work to the display of complete 6-dot Braille characters using a general purpose 2-D tactile transducer called STReSS2. The legibility of the produced Braille was evaluated by asking seven expert Braille readers to identify meaningless 5-letter strings as well as familiar words. Results indicate that reading was difficult but possible for most individuals. The superposition of texture to the sensation of a dot improved performance. The results contain much information to guide the design of a specialized Braille display operating by lateral skin deformation. They also suggest that rendering for contrast rather than realism may facilitate Braille reading when using a weak tactile transducer.
1. Introduction
The technology used to display Braille to visually-impaired computer users has not changed significantly for decades. Following the approach described in early patents [10], today's commercial Braille displays still rely on arrays of electromechanical Braille cells that control the protrusion of vertical pins with cantilevered piezoelectric bending actuators. This method has proved to be robust and effective, but the need for one actuator per pin pushes the price of these devices beyond that of a personal computer and makes multi-line displays economically unviable.
Several attempts have been made to reduce Braille displays to a few cells moving with the scanning finger. Fricke mounted a single cell on a slider and generated waveforms in an effort to produce sensations of friction on the skin [2]. Ramstein similarly used Braille cells in conjunction with a force-feedback planar carrier [8]. The success of this approach has been limited due to the difficulty of producing the sensation of brushing against Braille dots by indentation of the skin. These and other recent efforts at improving the refreshable display of Braille [1,6,9,14] have largely focused on reducing the cost of actuation without reconsidering the necessity of creating dot-like protrusions out of a surface. Other methods of skin stimulation can be found in the literature on tactile displays [7].
We recently investigated the possibility of displaying Braille by lateral deformation rather than indentation of the skin [3]. This principle, which we name laterotactile, assumes that critical aspects of skin deformation patterns occurring during tactile exploration can be reproduced with lateral stimulation alone. A single array of lateral stimulators in fixed contact with the fingerpad may thus produce the sensation of brushing against tactile features as it deforms the skin in response to exploratory movements.
The feasibility of this approach was evaluated using a laterotactile display (see Fig. 1a) mounted on an instrumented slider [3]. This Virtual Braille Display (VBD) prototype reproduced the sensation of brushing against a single line of Braille dots with an array of twelve piezoelectric bending motors that caused programmable, dynamic lateral skin deformation patterns along the axis of motion of the slider. When appropriately synchronized with the exploratory movements of the finger, waves traveling across the tactile display resulted in the perception of static features resembling Braille dots along the virtual surface. Braille readers who participated in the study were able to identify sequences of four Braille dots with an average rate of success of 90% after personalization of the sensation.
Efforts to design a general purpose 2-D laterotactile display have since resulted in the STReSS2 [11], a third generation display with a 10-by-6 matrix of laterally-moving actuators (see Fig. 1b). The device shows promise for a number of application areas including the refreshable display of tactile graphics [13].
The work presented in this paper extends our earlier work to the display of complete 6-dot Braille characters using the STReSS2. Although not designed with Braille in mind, the device's six rows of actuators make it possible to experiment with the display of multiple rows of Braille dots. The present study is a step toward the design of a laterotactile transducer optimized for the display of Braille.
2. Virtual Braille Rendering
The Braille rendering system comprised hardware and rendering algorithms which are described below.
2.1. Hardware
The STReSS2 has an array of 10-by-6 independent skin contactors able to apply tangential forces to the skin [11]. It has an active area of 1.2 x 1.1 cm, a spatial resolution of 1.2 x 1.8 mm, and a contactor area of 0.5 x 1.6 mm. Its actuators are restricted to lateral motion, deflecting towards the left or the right up to approximately 0.1 mm when unloaded. They exhibit a maximum force in the order of 0.15 N when prevented from moving. At the time of the present experiments, the leftmost column as well as two actuators near the center were defective. Only half of those were actually used for Braille rendering (see Section 2.2.2).
The STReSS2 was mounted on a linear slider (see Fig. 2) that allowed users to explore a virtual surface as they moved the display laterally. The low-friction slider was connected to an optical encoder giving a spatial resolution of 17 μm. Actuator activation signals were produced with a resolution of 10 bits. The system was controlled by a personal computer running Linux and the Xenomai real-time framework. The deflection of each actuator was updated at 750 Hz based on encoder readings.
2.2. Skin Deformation Patterns
The realism and the effectiveness of the sensation crucially depend on the specification of actuator activation patterns in response to slider movements, a process that we term tactile rendering by analogy with graphics rendering. The Braille rendering algorithms described below were inspired by prior experience with the VBD [3] and tuned according to the feedback of an expert Braille reader received during an informal preliminary experimentation session. The deflection of unloaded actuators is used here as an approximation of the resulting skin deformation patterns. Actual deformation patterns differ due to the biomechanical properties of the skin [12].
2.2.1. Dot Rendering
The pattern that occurs for each virtual Braille dot was adapted from earlier work with the VBD [3]. Consider first a single row of actuators. The free deflection δi of each actuator i is a function of the position xi of the actuator along the virtual surface. Given a pitch ε and slider position xs, the actuator position is expressed as xi = xs +iε. All actuators thus follow the same deflection function along the virtual surface, but with a phase difference. This results in tactile features that appear to travel along the fingerpad as if fixed on the virtual surface.
The deflection profile for a single smooth dot swings the actuators back and forth as the dot is traversed. This gives a sensation comparable to that of brushing against a single bump. If the actuators swing back and forth many times, a sensation comparable to that of sliding over a rippled surface arises instead. The addition of such a texture to a smooth dot was earlier found to noticeably increase the stimulus strength and to facilitate reading [3]. Defining deflections of -1.0 and +1.0 as the rightmost and leftmost actuator positions, the deflection profile of a complete dot is expressed as a weighed sum of the following two profiles:
 |
(1) |
 |
(2) |
where p=(xi-center)/radius is the relative distance from the dot center and k is an odd number of cycles in the texture waveform. For a texture level T, the combined deflection profile is obtained by superposing the waveforms:
 |
(3) |
The preliminary experimentations indicated that texture level is an important factor for legibility. Dots without texture (T=0%), with low texture (T=25%) and with texture alone (T=100%) were thus selected for further experimentation. The preferred number of textural ripples k was 7. The resulting deflection profiles are illustrated in Fig. 3.
Figure 3. Actuator deflection as a function of position for dots with different texture levels.
2.2.2. Cell Rendering
Lines of Braille dots were formed by combining dot patterns along the virtual surface. The preliminary tests showed that a strict adherence to standard Braille dimensions may not give sufficient separation between dots. Horizontal distances between dots of 3.7 mm within cells (1.6 x standard), and 7.4 mm between cells (2.0 x standard) were found to be reasonable. Dot deflection profiles were set to span 2.6 mm. The horizontal and vertical organizations of dots are illustrated in Fig. 4.
The rendering of Braille cells made use of the multiple rows of actuators available on the STReSS2. Since the slider moves only horizontally, the three rows of dots of the 6-dot Braille cell had to be mapped to deflections along the six rows of actuators of the display. Four possibilities were implemented and evaluated.
The first method maximized the forces applied by mapping each row of dots to pairs of adjacent actuator rows (Fig. 5b). The second method inverted the phase of deflection profiles within a pair of actuator rows so as to maximize shearing of the skin (Fig. 5c). The third method attempted to facilitate the perception of horizontal pairs of dots by showing the two column of dots of a Braille cell on different sets of actuators (Fig. 5d). The fourth and final method left odd rows inactive so as to increase separation between dot rows (Fig. 5e).
Although all methods were usable, the preliminary experimentations led to the selection of the fourth one. The first two methods were found to give slightly stronger sensations but the use of two rows per dot resulted in the perception of two dots, perhaps due to the large area of skin stimulated or the disjunction of actuators. The third method facilitated perception but felt like misaligned Braille. The selected method results in a vertical distance between dots of 3.6 mm (1.4 x standard), as illustrated in Fig. 4.
3. Virtual Braille Legibility
3.1. Method
The legibility of virtual Braille was evaluated by observing the performance of human subjects at reading tasks. Braille readers were asked to identify Braille strings composed of letters of the alphabet encoded according to the 6-dot Braille code (Fig. 6). The strings were either meaningless sequences of letters or familiar words. Following an experimental design similar to that used in [3], the level of texture applied to dots was varied to investigate its contribution to legibility. Dots were presented either without texture (T=0%), with low texture (T=25%) or with texture alone (T=100%). Other parameters were selected as described in the previous section.
Figure 6. Braille code for the alphabet.
3.1.1. Participants
Three female and four male experienced Braille readers volunteered for the study. Their age varied between 28 and 57, with a median of 52. Onset of blindness varied from birth to 19 years of age. The primary reading finger was the right-hand index for five subjects and the left-hand index for the other two. None of the subjects had tried the device prior to the experimental session.
3.1.2. Training
An experiment began with a short supervised training session during which Braille patterns of gradually increasing difficulty were displayed to the subjects. This allowed subjects to familiarize themselves with the device and the sensations it produced, thereby reducing training effects and providing time to adapt to the system's non-adherence to standard Braille dimensions.
3.1.3. Letter Identification
A first experiment was conducted to evaluate the legibility of meaningless strings of five letters. Subjects were asked to read using their dominant reading finger. They placed the slider to the left, waited for an audible signal, and proceeded to read the displayed string. They were instructed to verbally identify the string as soon as possible, with an audible warning after 30 seconds. Answers were logged by the experimenters. Supporting data such as the duration of trials were automatically recorded by the system.
An experimental session consisted of reading 26 five-letter strings for each of the 3 rendering modes, for a total of 390 letters. The order of presentation was randomized over two blocks of 39 readings with a short break in between. A similar difficulty level was maintained across subjects and modes by using the same set of 26 strings in all cases. The strings were randomly assembled such that each letter appeared 5 times. Memorization effects were minimized by randomly reordering the letters of each trial's string. For example, the string epkgn was shown 21 times (7 subjects x 3 modes) but the order of the letters was varied randomly each time.
3.1.4. Word Identification
An additional experiment was conducted to evaluate legibility in a more natural context. Following a procedure similar to the first experiment, subjects were asked to identify 5-letter words in French, their native language. Forty familiar words were selected such that each letter of the alphabet was represented at least four times. Virtual Braille was rendered using the optimal texture level as obtained from the first experiment. Subjects were instructed to give their best guess or spell the letters they were reading in case of difficulty. The experiment was terminated when either all forty words were read or the hour allotted for the session was nearly over.
3.2. Results
3.2.1. Letter Identification
Performance at reading meaningless strings varied greatly between participants (Fig. 7). S1 had great difficulty reading and asked to halt the experiment after only a few trials. Other subjects correctly read 22% to 83% of the letters presented to them, for an average success rate of 57%. Reading appeared slow and laborious for most subjects, taking 12 seconds per string on average.
Figure 7. Results of letter identification experiment.
The texture level appeared to affect legibility. Average legibility increased from 50% to 56%, and then to 64% as the level of texture was increased. A series of paired t-tests confirmed that Braille with texture alone was better read than both with low texture (t=-3.246, p<0.05) and without texture (t=-4.456, p<0.05). The difference between low texture and no texture was not significant (t=-1.745, p>0.05). Reading speed similarly increased with the amount of texture but only the difference between Braille with texture alone and without texture was significant (t=2.688, p<0.05).
An inspection of reading errors shows that legibility decreased with the number of dots in a letter (Fig. 8). This is consistent with other reports about Braille reading [5]. Errors most frequently involved the addition or removal of a single dot (Fig. 8). The complete confusion matrix is shown in Table 1.
Figure 8. Analysis of errors in letter identification experiment.
3.2.2. Word Identification
Word reading experiments were performed with the level of texture most effective during the first experiment: low texture for S7 and texture alone for all others. Due to time constraints, S2 and S4 read 20 and 30 words respectively. The others read the complete set of 40 words. On average, subjects were more accurate and faster than when reading meaningless strings (Fig. 9). They could read 69% of the words but only 57% of the letters in meaningless strings. The mean trial duration was 9 seconds for words and 12 seconds for meaningless strings. Individual performance at word identification was not strongly correlated with that at letter identification. S1, for instance, was unable to read meaningless strings even in a second attempt after reading words nearly flawlessly . S4 also performed much better at word reading, correctly reading 86% of the words but only 50% of the letters in meaningless strings. Overall, word reading rates were poor for two subjects (10%, 27.5%), reasonable for two others (70%, 87%), and excellent for the remaining three (95%, 97.5%, 97.5%).
Figure 9. Results of word identification experiment.
3.2.3. Verbal Reports
All participants felt that reading Braille rendered with the proposed system was difficult and demanded great concentration. Some compared the sensation to that of worn out or erased paper Braille. The participants expressed doubt about their skills and often reported feeling Braille characters other than the displayed letters such as punctuation. Many participants pointed out that reading was increasingly difficult over time, either due to tactile or mental fatigue. They were bothered by the large size of the virtual Braille cell and by the stimulation of parts of their fingertip generally unused for Braille reading.
The participants also confirmed that the use of texture facilitated reading by greatly increasing the strength and contrast of the dot sensation. Most participants slightly preferred the sensation of low texture although they often felt that texture alone had greater contrast. Some also suggested that the randomized presentation order of meaningless strings caused masking effects when a non-textured string followed a textured one.
Despite having difficulty reading, most participants showed genuine interest in the concept and were enthusiastic about its potential for Braille and tactile graphics. Many participants compared the sensations produced by the STReSS2 to those felt with the Optacon, a tactile transducer that enables the blind to read printed text by mapping images from a camera to vibratory patterns [4]. S1's reading difficulties were analogous to the subject's limited experience with the reading aid. S2, the only subject who still uses the Optacon regularly, felt that the STReSS2 produced weaker sensations and had difficulty reading with it. S3 found the sensation of the STReSS2 less irritating. Another subject who used the aid in the past had the reflex of looking for printed character patterns while reading with the STReSS2.
4. Discussion
Prior work with the VBD was focused on achieving realistic sensations that approached those of physical Braille dots [3]. Our results indicated that this was possible for most subjects using the equivalent of the non-textured dot rendering used here. A small textural component, equivalent to the low texture used here, was shown to improve legibility for some users, but at the cost of some realism. The use of texture alone was never considered as it was deemed to unnecessarily stray away from standard Braille sensations.
The results obtained with the STReSS2 point in a different direction. The ability of a laterotactile display to present realistic dot sensations appears to depend on the forces it is able to produce. Because its actuators are weaker than those of the VBD, the STReSS2 produces subtler dot sensations that, taken together with the greater complexity of identifying Braille characters, prevent efficient reading. While texture only improved the contrast of the Braille dots on the VBD, it appears to be critical with the STReSS2. A natural extension of this idea is to produce a purely symbolic representation of Braille patterns that leverages the Braille code without reproducing the sensation of Braille dots. As the experiment with texture alone demonstrates, knowledge of Braille appears to transfer well to this new medium.
This discussion leads to two possible approaches for the continuation of our work on laterotactile display of Braille. A first approach is to remain within a symbolic framework and attempt to design optimal laterotactile symbols to replace Braille dots. The resulting Braille would work on most laterotactile displays but may encounter resistance from Braille readers. A second approach is to aim for realism and design a specialized laterotactile display. Large piezoelectric actuators could be cut so as to produce a dense array of three rows of skin contactors respecting Braille dimensions. The revised design may produce forces strong enough to render realistic Braille. The display could also be widened to accommodate reading with multiple fingers.
5. Conclusion
This paper presented the result of experiments evaluating the legibility of 6-dot Braille rendered by lateral skin deformation with a general purpose tactile transducer. Reading meaningless strings was difficult but possible for most subjects. Reading familiar words was generally faster and more accurate. The addition of a texture to the sensation of Braille dots improved legibility but reduced realism.
These results indicate that a symbolic representation of Braille dots in the form of texture may fare better than a more realistic rendition when a weak laterotactile display is used. Future work will therefore focus either on increasing the forces applied by the tactile transducer or on identifying optimal laterotactile symbols to replace Braille dots. Comments from subjects also suggest that respecting at least vertical dimensions of Braille cells is critical for comfortable reading. Rendering parameters that received attention only during the preliminary experimentation phase of this work, such as the vertical distribution of sensations along the transducer, will also be given a more rigorous treatment.
6. Acknowledgments
This study was approved by the Research Ethics Board of the Centre for Interdisciplinary Research in Rehabilitation of Greater Montreal. The authors thank Marie-Chantal Wanet-Defalque and Chantal Nicole of the Institut Nazareth et Louis-Braille, and Qi Wang of the Haptics Laboratory for their support.
References
[1] D. Blazie. Refreshable Braille Now and in the Years Ahead. Braille Monitor, 43(1), 2000.
[2] J. Fricke. Substituting friction by weak vibration on a tactile pin array. In IEE Colloquium on Developments in Tactile Displays (Digest No. 1997/012), pages 3/1-3/3, 1997.
[3] V. Levesque, J. Pasquero, V. Hayward, and M. Legault. Display of virtual braille dots by lateral skin deformation: Feasibility study. ACM Trans. on Applied Perception, 2(2):132-149, 2005.
[4] J. G. Linvill and J. C. Bliss. A direct translation reading aid for the blind. Proc. IEEE, 54(1):40-51, 1966.
[5] S. Millar. Reading by Touch. Routledge, 1997.
[6] T. Nobels, G. Desmet, J. Van den Keybus, and R. Belmans. Development of a portable braille display using a fast prototyping platform for power electronics. In Proc. PEMD 2006, 2006.
[7] J. Pasquero. Survey on communication through touch. Technical report, McGill University, 2006.
[8] C. Ramstein. Combining haptic and Braille technologies: Design issues and pilot study. In Proc. ACM ASSETS'96, pages 37-44, 1996.
[9] J. Roberts, O. Slattery, D. Kardos, and B. Swope. New technology enables many-fold reduction in the cost of refreshable braille displays. In Proc. ASSETS'00, pages 42-49, 2000.
[11] Q. Wang and V. Hayward. Compact, portable, modular, high-performance, distributed tactile display device based on lateral skin deformation. In Proc. IEEE HAPTICS'06, pages 67-72, 2006.
[12] Q. Wang and V. Hayward. In vivo biomechanics of the fingerpad skin under local tangential traction. J. Biomech., In press.
[13] Q. Wang, V. Levesque, J. Pasquero, and V. Hayward. A haptic memory game using the STReSS2 tactile display. In ACM CHI 2006 Extended Abstracts, pages 271-274, 2006.
[14] L. Yobas, D. M. Durand, G. G. Skebe, F. J. Lisy, and M. A. Huff. A novel integrable microvalve for refreshable braille display system. IEEE J. of Microelectromech. Syst., 12(3):252-263, 2003.