Sight and Sound

Since pretty much every video has an audio track, the prospect of learning from paired audio-visual data — either with new forms of unsupervised learning, or by simply incorporating sound data into existing vision algorithms — is intuitively appealing, and this workshop will cover recent advances in this direction. But it will also touch on higher-level questions, such as what information sound conveys that vision doesn’t, the merits of sound versus other “supplemental” modalities such as text and depth, and the relationship between visual motion and sound. We’ll also discuss how these techniques are being used to create new audio-visual applications, such as in the fields of speech processing and video editing.