Tatsuya Ogusu

Kadenz – version alpha

「Kadenz(仮名)」は,抽象絵画を音と共に鑑賞するための新しいサウンドシステムです.一様に造形的な色と形で表現される抽象絵画は一見何が描かれているかわかりませんが,これには「画家の内面世界を忠実に表現するために,現実世界のモデルを一切借用せずに描く」という試みがありました.そうして一切の客観性を排除して描かれた抽象絵画は,音楽のように直接的な感情効果を持つのではないかと考えられます.本プロジェクトでは,抽象絵画を鑑賞する際の人の視線行動を利用し,注視した箇所の色や形を画像処理で抽出し,音に変換することで,抽象絵画を「聴く」ためのインターフェースを提案します.

※本プロジェクトはまだ研究段階であり,現在作品化に向けて進行中です.

Methods

“Kadenz (tentative name)” is a sound system utilizing gaze of a person who views an abstract painting.The system detects the gaze of a person who views an abstract painting by a gaze detection equipment such as the Eye Tracker.The gaze tends to stay at certain points in the painting for certain durations. At each of such staying points, color information and shape information of figures, is obtained. The diagram in Fig. 1 illustrates the authors’ gaze based music generation system.

  1. The gaze of a person who views the abstract painting is detected by a gaze detection equipment such as the Eye-Tracker, and each gazed point is tracked in the abstract painting. The tracked points of the gaze are smoothed with a simple averaging that uses the gaze points in the 15 frames prior to the current frame. This could remove noise due to blinking and false recognition.
  2. Gazed regions (objects or figures) are extracted from the painting using the gazed points. Specifically, as shown Fig. 2, the gazed region, which consists of multiple pixels, is obtained by finding a region with similar colors to that of the gazed point in the neighborhood of the gazed point. Here, the color similarity is obtained by computing the Euclidean distance D between the color P (R, G, B) of the gazed point and the color P’ (R, G, B) of a pixel in the neighborhood. Note that the values of R, G, and B range between 0 and 255 and that if D is smaller than 30, the color of that pixel in the neighborhood is judged to be similar to the color of the gazed point.
  3. Key, chord and melody are determined by the averaged color of the gazed region. This paper determines the key, chord and melody by using the authors’ proposed method [references ware as follows], which converts color to sounds based on correspondence between tonality and the colors people with synesthesia feel. Music tempo is determined by the area of the gazed region. Sound position between left and right is determined by the centroid of that region.
  4. The parameters determined in 3 are converted to MIDI (Musical Instrument Digital Interface), and are sent to the software synthesizer so that music (sound series) is generated.

This project is proceeding as a consistent of my research at Waseda university.