SENSORY SOUND OBJECTS

The high speed of modern life and the constant stimulants effect our perception of things and interpersonal communications. Trying to do many things at once, our sensory system starts to ignore some of the stimuli we have to deal with. This project seeks to find a way of understanding our perception of the world by experimentations with inter-sensory translations. Through a series of objects converting Braille into music, users will experience a different point of view of their communication tools and perception abilities.

The high speed of modern life and the constant stimulants effect our perception of things and interpersonal communications. Trying to do many things at once, our sensory system starts to ignore some of the stimuli we have to deal with. This project seeks to find a way of understanding our perception of the world by experimentations with inter-sensory translations. Through a series of objects converting Braille into music, users will experience a different point of view of their communication tools and perception abilities.

Prototyping Process:

In my prototypes, I wanted to experiment with both the form and meaning of language/
words.

  • Prototype 1:

The prototype was coded in Processing and was an interactive experience which could be played by the user as a musical instrument, using the keyboard of the computer. The formulation was based on labeling each dot on a braille symbol and assigning a note to each label. This meant that a letter in braille language would be associated with a number of musical notes ranging from one to six, which is are the minimum and maximum number of dots in a letter in braille language. For example, the letter “a” will be assigned a single note, while the letter “z” will be assigned 4 notes, which are the number of dots on their symbols in braille language.

I had several iterations of the first prototype. The first iteration was based on playing each letter as a chord; which meant playing all the musical notes associated with that letter at the same time. I played around with this iteration by changing the notes assigned to each label, to make each letter sound more harmonious. The second iteration was based on playing each letter as a melody if the letter was associated with more than one musical note. This meant playing the musical notes associated with a letter separately, in an order. I experimented with this iteration with changing the notes assigned to each label and changing the order of the notes played for each letter, in order to obtain a more harmonious melody.

sensory_!.png

In this prototype, I only gave importance to the form; the visual aspect of the language. I did not try to express the meaning of the words and sentences with the audio element.

  • Prototype 2:

My second prototype was also initially based on the form of the braille language. I took each braille letter and placed them to the musical scale, treating each dot in the letter as a musical note.

I used MuseScore software to make different iterations of this prototype. I tried different placements of the letters on the scale, to create different audio effects. My first iteration was only about the form again, giving no importance to the meaning of the words. This gave the ability for the user to interact with it.

In my second iteration, I tried to give some importance to meaning, in addition to form. I tried to manipulate the audio results of each word, according to the meaning of the word and the feeling it is supposed to express. I worked with words that express feelings, such as “happy”, “sad” or “angry”, and tried to manipulate their melodies by changing the placements on the scale, trying to put them into major or minor tones.

  • Prototype 3:

In my third prototype, I wanted to make the experience analog, so I used some musical representations of the words obtained in the second prototype and made an analog music box that would play these simple melodies. I used a musical box kit that would allow me to put musical notes on a scale and play them.


The analog property was very interesting to the users, yet it was not as interesting as an interactive experience.

User testing for initial prototypes:

  • Prototype 4:

In my fourth prototype, I thought about a different approach of looking at the writing in Braille, which would allow me to work with an elementary level of music theory. I first thought about the basic elements in a musical composition and tried to find a common point between musical language and the languages we use to communicate with each other.  The most influential elements that I came up with were “rhythm” and “stress”. These are the elements can change the idea delivered to the audience both in music and any kind of language. 

I decided to break the Braille symbols down into 3 parts, each part being a row consisting of 0, 1 or 2 dots in each letter. I assigned a different instrument to each row.

I used the rhythmic structure of jazz in my experimentations.The top row of each symbol was played by the double bass and it was playing the melody. The middle row was played by the ride cymbal, and the bottom row was played by the hi-hat. I assigned the instruments to each division based on the frequency of dots appearing on the rows. Namely, since all of the Braille symbols have at least one dot on the top row , it is the row with the highest frequency of dots, so I assigned the double bass to it, which will play the melody.  The middle row has the second highest frequency of dots, so I assigned the time keeper ride cymbal to it and to the bottom row, which has the lowest frequency of dots, I assigned the hi-hat, which should appear less frequently in the jazz beat than the others.

For creating the melody, I again assigned a note to each Braille symbol. Since double bass was the only pitched instrument in the group, it was playing the notes assigned. Each dot was played as one beat in the melody, so if the row had one dot, it would be played as one beat of sound and one beat of rest; if it had two dots, it would be two beats of sound. 

  • Prototype 5:

The final prototype is an object with which you can type a sentence and hear the corresponding melody.

I used the same assignment technique of instruments to Braille symbols, but I changed the instruments assigned. I replaced the instruments with vibraphone, alto kalimba and timpani, just to make them sound more natural, rather than electronic. Also, instead of making each dot on the symbols separate beats, I made the durations based on the number of dots in the symbol. Namely, if there were two dots in a row, it would sound for twice as long time than a row with one dot.

Photo Credit: Xing Zhang
Special Thanks to Anna Meriano, Xing Xhang, Maria Helena Lima, Jiaqi Liu, Saima Mohammad, Marta Molina Gomez