Kadenz – version alpha
“Kadenz (tentative name)” is a sound system utilizing gaze of a person who views an abstract painting.The system detects the gaze of a person who views an abstract painting by a gaze detection equipment such as the Eye Tracker.The gaze tends to stay at certain points in the painting for certain durations. At each of such staying points, color information and shape information of figures, is obtained. The diagram in Fig. 1 illustrates the authors’ gaze based music generation system.
- The gaze of a person who views the abstract painting is detected by a gaze detection equipment such as the Eye-Tracker, and each gazed point is tracked in the abstract painting. The tracked points of the gaze are smoothed with a simple averaging that uses the gaze points in the 15 frames prior to the current frame. This could remove noise due to blinking and false recognition.
- Gazed regions (objects or figures) are extracted from the painting using the gazed points. Specifically, as shown Fig. 2, the gazed region, which consists of multiple pixels, is obtained by finding a region with similar colors to that of the gazed point in the neighborhood of the gazed point. Here, the color similarity is obtained by computing the Euclidean distance D between the color P (R, G, B) of the gazed point and the color P’ (R, G, B) of a pixel in the neighborhood. Note that the values of R, G, and B range between 0 and 255 and that if D is smaller than 30, the color of that pixel in the neighborhood is judged to be similar to the color of the gazed point.
- Key, chord and melody are determined by the averaged color of the gazed region. This paper determines the key, chord and melody by using the authors’ proposed method [references ware as follows], which converts color to sounds based on correspondence between tonality and the colors people with synesthesia feel. Music tempo is determined by the area of the gazed region. Sound position between left and right is determined by the centroid of that region.
- The parameters determined in 3 are converted to MIDI (Musical Instrument Digital Interface), and are sent to the software synthesizer so that music (sound series) is generated.
This project is proceeding as a consistent of my research at Waseda university.