Reverse the auditory processing pathway: Coarse-to-fine audio reconstruction from human brain activity
Brain2Sound Dataset
| Stimulus | LiR | Fine-LDM | C2F-LDM (ours) | |
|---|---|---|---|---|
| Reconstruction |
|
|
|
/sample1.png)
|
| Comparison | C2F-LDM reconstructs the spectral distribution of Stimulus very well (although not cattle calls), significantly better than Fine-LDM and LiR (which always sounds like noise, omitted below). | |||
| Reconstruction |
|
|
|
/sample2.png)
|
| Comparison | C2F-LDM reconstructs the piano sound in Stimulus, while Fine-LDM is the guitar sound. | |||
| Reconstruction |
|
|
|
/sample3.png)
|
| Comparison | C2F-LDM reconstructs the speech in Stimulus, while Fine-LDM reconstructs the singing voice. | |||
| Reconstruction |
|
|
|
/sample5.png)
|
| Comparison | C2F-LDM reconstructs the periodicity of Stimulus (although not bird calls), which is better than Fine-LDM. | |||
| Reconstruction |
|
|
|
/sample6.png)
|
| Comparison | C2F-LDM reconstructs the periodicity of Stimulus (although not bird calls), which is better than Fine-LDM. | |||
Brain2Music Dataset
| Stimulus | LiR | Fine-LDM | C2F-LDM (ours) | |
|---|---|---|---|---|
| Reconstruction |
|
|
|
/sample1.png)
|
| Comparison | The music genre of C2F-LDM is closer to Stimulus. | |||
| Reconstruction |
|
|
|
/sample2.png)
|
| Comparison | The pitch of C2F-LDM has an obvious process of rising first and then falling, which is consistent with Stimulus. | |||
| Reconstruction |
|
|
|
/sample3.png)
|
| Comparison | The rhythm of C2F-LDM is closer to Stimulus. | |||
Brain2Speech Dataset
| Stimulus | LiR | Fine-LDM | C2F-LDM (ours) | |
|---|---|---|---|---|
| Reconstruction |
|
|
|
|
| Comparison | C2F-LDM correctly reconstructs the male voice, while Fine-LDM reconstructs the female voice. | |||
| Reconstruction |
|
|
|
|
| Comparison | C2F-LDM has a faster speaking speed and is closer to Stimulus. | |||
| Reconstruction |
|
|
|
|
| Comparison | C2F-LDM can reconstruct the laughter in Stimulus, while Fine-LDM is closer to applause. | |||