Sonification Enhances Perception of Color Intensity

Sonification Enhances Perception of Color Intensity

Division for Media and Information Technology Link ¨oping University

ABSTRACT

This poster presents an interactive sonification experiment, designed to evaluate possible benefits of sonification in information visual-ization. The aim of the present study was to explore the use of composed and deliberately designed musical sounds to enhance perception of color intensity in visual representations. It was hypoth-esized, that by using musical sounds for sonification perception of color intensity would be improved. In this evaluation, sonification was mapped to color intensity in visual representations, and the par-ticipants had to identify and mark the highest color intensity, as well as answer a questionnaire about their experience. Both quantitative and qualitative preliminary results suggest a benefit of sonification, and indicate that sonification is useful in data exploration.

Index Terms: Sonification, information visualization, color inten-sity, evaluation.

1 INTRODUCTION

Data are produced by many different research disciplines, resulting in data that ranges from static data to dynamic temporal data sets. It is common to use different visual representations to represent this data, however, the large size and complex structures of the data often make analyses of the visual representation a grand challenge.

One of the problems might be the limited space on the visual display, resulting in dense and unintelligible visual representations. To reduce visual clutter and facilitate perception of visual repre-sentations, it is common to employ renderings based on the data density [2]. However, using density information has the drawback that it is difficult to perceive the actual number of blended data points in the representation, as the light level or color intensity in the visual representation might become saturated. Another problem might arise due to simultaneous brightness contrast [15], which occurs when a colored patch with a set luminance is perceived as brighter when it is placed in a region with dark color tones, compared to when it is placed in a region with overall lighter color tones. Yet another difficulty for human visual perception is the Mach band phenomenon [7], which occurs at boundaries between colors, or between shades of colors, and is perceived as a gradient just next to the boundary, even if the actual color is solid. These problems will, in turn, negatively affect the perception of density levels as well as strength encoded as color intensity in visual representations.

Such shortcomings in the comprehension of data visualization, can hardly be managed by the visual modality alone. Instead, these could be addressed by adding sound as a complementary modality to the visual representation. The research area of sonification is rather new and emerging, and aims to use sound to enhance and clarify visual representations of data, and to simplify the understanding of these [4, 5, 9]. The aim of the present study was to investigate whether sound could be used to facilitate the perception of color intensity, and contribute to the research field of sonification as well as information visualization.

*_{e-mail: niklas.ronnberg@liu.se}

Figure 1: Example of the visual representations used in the test setup, showing the complexity of grating. The upper half shows the actual stimuli used in the test setup, while the lower half shows the same stimuli but with enhanced contrast for use in this paper.

2 RELATEDWORK

Even though the concept of sonification and data exploration is not new, the combination of visualization and sonification has been sparsely evaluated, for example in connection with the depth of mar-ket stock data [8], to augment 3D visualization [6], and to enhance visualization of molecular simulations [10]. Some research has con-sidered sonification in connection with data exploration and scatter plots [3, 11], as well as sonification for data exploration in scatter plots and parallel coordinates [12, 13]. All these studies suggest that there is a benefit of sonification in connection with visualization, and data exploration.

3 VISUALREPRESENTATION

The visual representations used in the present study (see Fig. 1) were created in Matlab as a variant of a sine wave grating, by mixing two sinusoids of different frequencies, and then by adding some random ripples with an applied Gaussian filter to the mixed sinusoids, and finally by adding a triangle wave to create a peak level in the combined wave form. A total of 90 images were created, with a balanced difficulty level over sets of ten images. The green color channel was used, as the human visual perception is more sensitive to contrasts within the green parts of the spectrum [1, 14].

4 SONIFICATION ANDMAPPING TOLIGHTLEVEL

The sonification used for the present study was created in Super-Collider, which is an environment and programming language for real-time audio synthesis. The tones used were C2, C3, G3, C4, E4, G4, C5, E5, C6, C7, and C8, thus in combination forming a rather large C major chord. This chord was mixed with pink noise, at a low sound level, to further ensure that a wide frequency band sound was used for the sonification, but still with a pleasant harmonic content. Two aspects of the musical sound were modulated due to the visual representation, amplitude and cutoff frequency.

5 TESTAPPARATUS

The interactive test (see Fig. 2) was divided into four conditions: no sonification (Quiet), sonification with band pass filtering (BPF), sonification with amplitude modulation (AM), and sonification with both band pass filtering and amplitude modulation (BPFAM). In each condition, the participants moved a horizontal slider, using

Figure 2: The test setup, showing the number of the trial (in Swedish), the type of sonification, the visual representation, the interactive slider used to select the vertical pixel row in the visual representation, and the button used to confirm the selection and for starting the next trial.

the computer mouse, to mark the vertical pixel column that had the highest color intensity in the visual representation. The participants also answered a questionnaire about their experience of the different sonification conditions, as well as about their experience of the sonification in general. Answers were given via a 5-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree). The experiment yielded objective measures, accuracy and response time, as well as subjective measures from the questionnaire.

6 RESULTS

Preliminary results (n=15) show that the accuracy was 3.0 (SD=1.9) for Quiet, 1.8 (SD=1.1) for BPF, 1.7 (SD=0.9) for AM, and 1.5 (SD=0.7) for BPFAM. The accuracy measure indicates the highest light level minus the selected light level, hence a lower measure means better performance.

When accuracy was analysed using a repeated measures ANOVA with one within-subject factor, sonification condition (Quiet, BPF, AM, BPFAM), a main effect of sonification condition was found (F(3,42)=9.46, p<0.001). Post-hoc tests with Bonferroni correc-tion for multiple comparisons revealed a significant difference be-tween Quiet and BPF (p=0.017), as well as bebe-tween Quiet and AM (p=0.025), and between Quiet and BPFAM (p=0.031), but there were no differences found between the three sonification conditions. The measures of response time suggested that responses were generally faster in the Quiet condition compared to when sonification was used. The subjective measures from the questionnaire showed that the participants generally experienced sonification as helpful.

7 DISCUSSION ANDFUTURE WORK

The preliminary results presented in this poster suggest that sonifica-tion can improve percepsonifica-tion of a visual representasonifica-tion with regards to color intensity. This was demonstrated by increased accuracy when sonification was used compared to when no sound was used. However, the results also show that the type of sonification did not affect the outcome of the experiment. This assumption was also supported by the subjective measures.

Furthermore, the increase in response time when sonification was used, suggest that the participants used the sonification to refine their selection as also observed, hence the increased accuracy, and that this took longer time. Interestingly, some of the participants stated that sonification improved their response time as well, even if this was not the case according to the objective measure.

Even though a simple mathematical test would be sufficient to mark the highest color intensity in a given data set, the task used in this study is a simplification to enable a controlled examination of sonification benefit. The challenge and problem with density levels, simultaneous brightness contrast, and the Mach band phenomenon exist in visual representations in general. Therefore, knowledge

gained within the present study should be applicable to other visual representations than those used in this study.

For future work, the first step will be to further investigate mu-sical elements other than BPF and AM that might be suitable for sonification, such as pitch, harmony, as well as tempo and rhythm. The second step will be to evaluate sonification in relation to a wider range of visual representations. This should indicate which musical element in the sonification that is most suitable to use interactively, in combination with which kind of visual representation for maximal usefulness.

8 CONCLUSION

The preliminary results suggest that there is a benefit of sonification, showed by increased accuracy, in selecting the vertical pixel column with the highest color intensity in the visual representations. This result was also supported by the subjective measurement.

REFERENCES

[1] CIE. Commission internationale de l’Eclairage proceedings, 1931. Cambridge University Press, Cambridge, 1932.

[2] G. Ellis and A. Dix. A taxonomy of clutter reduction for informa-tion visualisainforma-tion. IEEE Transacinforma-tions on Visualizainforma-tion and Computer Graphics, 13:1216–1223, 2007. doi: 10.1109/TVCG.2007.70535 [3] J. H. Flowers, D. C. Buhman, and K. D. Turnage. Data sonification

from the desktop: Should sound be part of standard data analysis software? ACM Transactions on Applied Perception, 2, 2005. [4] K. Franinovic and S. Serafin. Sonic Interaction Design. MIT Press,

2013.

[5] T. Hermann, A. Hunt, and J. G. Neuhoff. The Sonification Handbook. Logos Publishing House, Berlin, Germany, 1nd_{ed., 2011.}

[6] M. Kasakevich, P. Boulanger, W. F. Bischof, and M. Garcia. Augmen-tation of visualisation using sonification: A case study in computa-tional fluid dynamics. In Proc. IPT-EGVE Symposium, pp. 89–94. The Eurographics Association, Germany, Europe, 2007. doi: 10.2312/PE/ VE2007Short/089-094

[7] R. B. Lotto, S. M. Williams, and D. Purves. Mach bands as empirically derived associations. In Proc. National Academy of Sciences, vol. 96, pp. 5245–5250. National Academy of Sciences of the United States of America, Los Alamitos, 1999.

[8] K. V. Nesbitt and S. Barrass. Evaluation of a multimodal sonification and visualisation of depth of market stock data. In Proc. International Conference on Auditory Display (ICAD), pp. 2–5. International Com-munity on Auditory Display, United States, 2002. doi: 10.1.1.103. 7402

[9] T. Pinch and K. Bijsterveld. The Oxford Handbook of Sound Studies. Oxford University Press, 2011. doi: 10.1093/oxfordhb/9780195388947 .001.0001

[10] B. Rau, F. Frieß, M. Krone, C. Mller, and T. Ertl. Enhancing visual-ization of molecular simulations using sonification. In Proc. IEEE 1st International Workshop on Virtual and Augmented Reality for Molecu-lar Science (VARMS@IEEEVR 2015), pp. 25–30. The Eurographics Association, Arles, France, 2015. doi: 10.1109/VARMS.2015.7151725 [11] E. Riedenklau, T. Hermann, and H. Ritter. Tangible active objects and interactive sonification as a scatter plot alternative for the visually impaired. In Proc. 16th International Conference on Auditory Display (ICAD-2010), pp. 1–7. International Community for Auditory Display, Germany, Europe, 2010.

[12] N. R¨onnberg, G. Hallstr¨om, T. Erlandsson, and J. Johansson. Sonifica-tion support for informaSonifica-tion visualizaSonifica-tion dense data displays. In Proc. IEEE VIS Infovis Posters (VIS2016). IEEE VIS, Baltimore, Maryland, 2016.

[13] N. R¨onnberg and J. Johansson. Interactive sonification for visual dense data displays. In Proc. 5th Interactive Sonification Workshop (ISON-2016), pp. 63–67. CITEC, Bielefeld University, Germany, 2016. [14] G. J. Smith, T. The c.i.e. colorimetric standards and their use.

Trans-actions of the Optical Society, 33:73–134, 1931. doi: 10.1088/1475 -4878/33/3/301

[15] C. Ware. Information Visualization: Perception for Design. Morgan Kaufmann Publishers Inc., San Francisco, 3nd_{ed., 2013.}