When the Ears Interact with the Brain What we perceive in a complex setting depends on how we focus attention. The ability to attend to one sound and filter out competing sources of noise requires both a robust sensory representation and the ability to direct and control attention through the closely coordinated activity of multiple cortical networks. Presentation Video
Free
Presentation Video  |   November 01, 2016
When the Ears Interact with the Brain
Author Affiliations & Notes
  • Barbara Shinn-Cunningham
    Boston University
  • Presented at the 26th Annual Research Symposium at the ASHA Convention (November 2016).
    Presented at the 26th Annual Research Symposium at the ASHA Convention (November 2016). ×
  • The Research Symposium is hosted by the American Speech-Language-Hearing Association, and is supported in part by grant R13DC003383 from the National Institute on Deafness and Other Communication Disorders (NIDCD) of the National Institutes of Health (NIH).
    The Research Symposium is hosted by the American Speech-Language-Hearing Association, and is supported in part by grant R13DC003383 from the National Institute on Deafness and Other Communication Disorders (NIDCD) of the National Institutes of Health (NIH).×
Article Information
Hearing & Speech Perception / Acoustics / Hearing Disorders / Research Issues, Methods & Evidence-Based Practice / Professional Issues & Training / Attention, Memory & Executive Functions / Clinical Practice Research / Overview, Trends and Environment
Presentation Video   |   November 01, 2016
When the Ears Interact with the Brain
CREd Library, November 2016, doi:10.1044/cred-pvd-c16001
CREd Library, November 2016, doi:10.1044/cred-pvd-c16001

The following is a transcript of the presentation video, edited for clarity.

What I'm going to talk to you about today is sort of an overview giving you a feel for how it is that the brain interacts with the ear when we're in complex settings especially, because that's really where you see the effects of the brain more clearly. And I should say I started out as a psychophysicist, what I was interested in is how sound is localized in space, and over time I realized that one of the main things -- that knowing where things are is a good thing, but we don't really use our sense of sound if we're sighted to do that without a lot of detail. We really rely on the sense of sight for that, but nonetheless if you went to the poster session or go to a poster session today and try to listen with the only one ear you'll have trouble, and I guess I challenge you actually to go into one of the poster sessions and plug one ear and try and converse, because you'll find that having two ears and having a spatial sense of sound really is very helpful in exactly those kinds of settings. So I became very interested in understanding that and decided as I went along that just doing psychophysics and computation wasn't enough, and so what you'll see today is a blend of a kind of an overview and some demonstrations of the effects that I study, but then also some brain data kind of showing us how it is that we use our brain when we're listening.
Everyday acoustic communication is an amazing feat
So it is pretty amazing when you think about it, how well we communicate in noisy settings. And if you go into a crowded cocktail party, because that is sort of the stereotypical example is the cocktail party, one of the things that's kind of interesting is all of the sound that is in that environment adds up acoustically before it ever enters your ear. And so there may be some very interesting interactions going on at some cocktail party and there's some things that are less interesting, because if you've ever seen me talk about this I always do use this slide, although I do update it a little bit depending on the circumstances. But it's kind of funny because invariably I seem to be the person stuck talking to this guy, who is not the most interesting person in the world. He's a little self-involved, he's not maybe the most fun person in the cocktail party. But one of the beautiful things about it is I can stand there and nod politely at the right times while actually throwing my attention elsewhere, and paying attention to the very interesting things that I might be more excited to hear about. And that's pretty neat. If we try to build a machine to do that, we fail. We're not nearly as good at this as any good listener with healthy hearing. And it really involves the brain very, very directly. So I'm very interested in studying those processes and understanding how it is that we can filter out sounds.
One of the things that's kind of interesting is if you think about how psychophysics and psychoacoustics has evolved. We started by worrying about whether things were audible. And certainly if you can't hear that something is there, that's a problem. You're not going to be able to do much with that if you can't hear it. But one of the things that's fun is in a lot of settings, it's not it's not actually - this is even better. It's not that my son Nick is not visible somewhere in the scene, it's that there's a whole bunch of interesting things going on in the scene, and to find him in that scene you have to interrogate each of the faces, one after the other. Now it's kind of interesting you, literally do interrogate each face one after the other because your brain organizes that visual scene into faces. It doesn't investigate each nose, it investigates each face until it finds the right one. So it's not that he's an invisible, the analog of inaudible, but rather that there's clutter. There's other things vying for attention, and you can't make sense of him until you look at him specifically. And that's really what attention is all about.
Now vision is actually -- we're catching up, but we traditionally, the studies of attention actually started in audition at in the turn of the mid-last century. But vision took over because vision is a bigger field and they made much, much greater advances, faster advances in understanding the neuroscience underlying attention. We started to catch up but we're still a little bit behind. And one of the things that's really fun is if you look in vision, you can actually show that if I direct attention like to a particular location in space -- this is my other son Will -- the effect on the brain is literally to turn down the representation of other things so that his face is the thing that gets through and is analyzed in detail. Now I kind of hinted that I've been using these slides for a while. I just I continue to because it's getting funnier and funnier because my son Nick is about to graduate from college, and Will is now a sophomore. So I really have been working on this for quite a while and have been looking at it from all different angles. And like I said I started out looking at it as a behavioral issue, but then I brought in techniques from neuroscience to study how is the brain engaging in these tasks.
So it's early morning everybody stretch. Okay, because I'm about to make you do some work. I'm going to demonstrate -- hopefully these demos work -- I'm going to demonstrate for you the effects of attention and how good you are at it. And what I'm going to do is play a mixture -- and you can see this mixture of two voices, it's clearly two voices, right? That's a joke you can laugh. No it's not. This is time versus energy, and you can see there's stuff going on. But this is two voices, but they're mixed together, and you're going to be able to attend to one if you focus attention on the right features. So I'm going to play this for you, and you're gonna listen for the sentence that starts "her shaky" and to make it easier, these are sentences that are used to do speech testing and they're over articulated, clear speech so it should be really easy. And I'm going to ask for people to raise their hands afterwards about what you hear so try to focus on the demonstration.
How many people felt like they heard the sentence that started "her shaky" -- you want me to play it again? Of course I could play it again, I could play it an infinite number times -- this is not a normal cocktail party. These are hyper articulated speech sounds. I will play it again. Listen again -- you're probably not going to be able to follow the right speaker because these are identical talkers talking nonsense sentences from the same speakers with no spatial cues. There's no linguistic, I guess there's no linguistic or semantic cues that differentiate the two streams and so they mix together. But listen again, and listen to what you can get out. You'll hear words.
Despite the fact that it's a mixture of the same voice, with the same pitch range, from the same direction, how many people have heard words? Everybody could hear words, and that's one of the things our brains are immensely good at: pulling out structure from a mixture of sound. So you're pulling out the words, the syllables the words formed together naturally because of the continuity and time and frequency and expectations about what words look like. But because there's nothing to allow you to latch on to the right words through time that connect them and differentiate one sentence from another, you can't follow one sentence against the other. Here are the two sentences just for completeness:
Her shaky increases will leap on their quarrel
Really no predictability, that's part of the reason this example works. They're nonsense sentences. Here's the other.
Their greedy pull ends at the carpet
And if I put them together, you can follow it if you kind of know, if you're primed a little bit. But the mixture cannot be pulled apart, so it really is more like that picture of blue, it really is like a picture where the organization of the different words or the different messages is to unclear for your brain to separate the sounds. And in order to focus attention on one thing, you have to separate it from everything else first. That's a necessary piece of this puzzle. Now I don't mean to imply that segregating sound sources is a process that happens, and then attention happens. Especially in noisy settings where it's just on the edge of what you can do, you may start to sense that there's something you want to pay attention to, focus on it that will help it get segregated by starting to suppress other things, and it will build up through time. You can actually see this in behavior, that people through time get more and more precise at this focus of attention, if it's not obvious at the very beginning of something. So if I'm trying to read this mess of words it's really quite hard. And I don't know -- many of you are more clinically minded than I am. I sit in my lab all day, but everybody that I know who works with people with hearing impairment, it's really this problem of segregation that they tend to describe. They'll tend to describe being at a restaurant and hearing, when they put in their hearing aid, that all it does is make the clinking sound of the dishes nearby louder. That's really a failure of the process of segregation and attention, they're unable to filter out the unimportant stuff because they don't have a representation that supports the detail that you need to segregate and then select out from a mixture the right thing. So really, this is not the normal situation. For a healthy ear, most people are not talking at the same time as they're identical twin, nonsense sentences from the same direction. Normally in the real world, everything is independent of everything else. And so if one speaker is talking and making some sense, somebody nearby is talking, but there's features in the sound or in the visual image that differentiate the words or the messages. So when you look at this scene versus this scene it's completely different. You can make out words when they're a mush, but it's effortful, it takes time, it's not natural. You see it as a blob that you then have to really analyze.
That's very different than when this the words are differentiated by the feature of color, and its gray scale actually works too if you're colorblind, I was able to get that to work. But if you look at this scene, you now interrogate each word very naturally and the meaning comes very quickly compared to when they're all mixed together. So it's not that you can't get any information out of a scene that is mixed that you can't segregate naturally. It's just that it's slower, less precise, and guess what if you're trying to keep up in a conversation that extra time that it takes to pull stuff out of the mixture, kills you. You cannot keep up with the conversation. And I myself, I'm a middle-aged person about you know, when I was 45-ish I started to really notice myself that I was having more trouble in complex settings, and it's annoying because I'm kind of a smart alec and I like to make smart ass remarks. But now sometimes if it's too noisy I can't be quick enough because I'm still processing this, and just a split-second late and the joke is gone. it's so disappointing.
So it's real problem, and I don't have overt hearing loss. I just have normal middle-aged hearing -- and we can talk about that at the end -- but the process takes time now that it didn't used to, and that a healthy young ear doesn't need in order to make this kind of decision about what to suppress and what not to.
So now, I kind of played with you, I gave you an example, got you all excited -- you're going to tell me and I was going to show you how good you are at listening. Hopefully the acoustics in this room are not so bad that this example will actually work for those of you who are not my age or older, or who have hearing impairments at least. But what I'm going to do now is play an example that really you should be able to segregate. Now I gave this example -- two days ago I gave a talk and I used this example with a roomful of psychologists, and I asked a question and they all raised their hands, but I it may not be that way here because your backgrounds are very different. So how many people here have heard of the seven plus or minus two problem. About half. That did not remove everybody, like everybody. It's a memory thing. Seven plus or minus two is kind of the number of items you can keep in working memory, quickly, you know without memorizing it overtly. So if I'll play something you can rehearse it and keep it in mind for a short time and then recite it back. And guess what a seven digit phone number just is at the limit of what you can do. So I'm about to test your short-term working memory, and I'm going to play you a seven digit phone number. It's spoken in a mixture, but it's segregable. You should be able to hear it and isolate it and focus attention on it so that you can tell me the number. And it's a male voice, its metallic sounding, it's been processed to be metallic sounding -- it's actually, for reasons I won't go into, for an experiment. But it works really well for this demo because it's really very distinct and it has a monotone so it doesn't change pitch. It's really easy to grab and latch onto it. And so is everyone ready, because I really am going to call on someone so and if you want to write the number down if you're worried you'll forget it, you can write it down in real time. I don't mind. This is not really about the memory part, it's just it happens to be what the limit of what you can do usually. Is everyone ready hopefully I'll click on the right thing, here we go. [mixture of voices] How many people got the phone number? Good, a majority. Of those who have your hand up, how many got all seven digits? Let's do it this way -- of those who got all seven, did you -- keep your hand up I'm a teacher of those who have your hands up: How many of you (without speaking out loud because you don't want to ruin it for everybody) how many of you can tell me any of the content of the other speaker that was in the mixture. That's pretty common, that's pretty common. There's one person in the audience of about a hundred two days ago who could do it. And sometimes that's about right. Now, if you've heard me play the example before, you probably could because you knew I was going to do this mean awful trick. I really made you focus attention. I challenged you. I said this is about limit of what you're going to be able to do. And you're overachievers, you're like I'm gonna get this. You really focused. This is not an ordinary audience exactly. And I was depending on that. But I also played phone digits, a set of phone digits. Can anybody guess why I used numbers rather than a sentence? Less context. Because if I'm speaking and making some sense then you don't need every word to know, you still get the gist, which is really what communication is about. But with a phone number, you're at the bar, you meet somebody nice, you missed one digit -- sorry. There's no predictability. So that really focuses you and makes you maintain attention throughout, which is why you were so successful. So nobody could tell me any of the content. Again without saying it out loud, can anybody tell me anything. Did you get anything about the quality of the other thing in the mixture? I absolutely made you concentrate that's the point. When you are successful at concentrating, you will suppress things. So just for fun, if anyone wants to get in contact with my very dear colleague Steve Coburn he's a wonderful man, very very knowledgeable about speech and hearing and binaural hearing -- just 617 before those digits you're set. The other thing in the mixture, though, didn't get processed. Now if I had asked you more right after I played the mixture, even if you hadn't successfully pulled out the content, some of you would have been able to say it with a woman. Because I've done this since my kids were tiny. I've used this example because it's engineered to be a good example. I'm going to play the same mixture, but now in exactly the same mixture just listen for me. It's a little quieter so I'll play it because some of you may have hearing impairment.
It's hard to understand two things at once
So that's the thing, when you focus when you concentrate, when it's easy to segregate things, focusing on one suppresses the other. It's a competition. And we've done some studies where we literally try to get people to listen to two things at once, and really there's not evidence for you listening simultaneously attentively to two things at once. There's evidence that you multiplex, that you switch between them rapidly in order to make sense of things. Which can work in normal conversation because there is so much predictability, especially when you've been married 29 years and you're trying to watch the TV and your husband is talking. I know what he's going to say before he even says it. I don't have to hear every word. I just know from the way he moves what he's about to say.
So anyhow, predictability is a huge feature that we use, and it's when things get hard that predictability becomes less helpful, because it's not as redundant because we're slower and we're really trying to pull this information out. But one of the things that's fun is, when you focus successfully -- which isn't always -- but when you do, you will suppress the other thing if it's segregated. And you you can see it in the brain. This is where I think it gets really fun.
Cortical responses are strongly modulated by attention
So there's been a lot of studies looking at the responses of the brain to sound. And one of the things you can do is put a cap on the head and measure voltages, and Elyse I'm sure will tell us much more about this, but you can put a cap on the head and measure voltages on the scalp. So this is Inyong Choi and this is Scott Bressler. This is Inyong when he was a postdoc now he's a professor at University of Iowa, but he did some studies with us when he was a postdoc looking at exactly this, and got me in the lab too, which is fun. And I'm you can actually see, whenever there's an onset of a sound it actually causes synchronous activity in a lot of neurons in the brain. And when that happens it will end up -- those the currents that are being generated by the brain are aligned in the right way so it aligns and produces a strong total current so that they don't cancel. You'll end up seeing energy on the scalp when you measure it on the scalp. Any onset of a sound can lead to a very strong, very stereotypical response. And you can see it on the scalp simply by measuring voltages. So we played a game where we created an attentive task. We played really simple melodies, there are three melodies. They were different in timbre like different musical instruments. One came from the front, one came from the left, one came from the right. But the game we played is that the each of these melodies had a different number of notes. Which meant that the onset of each of these melodies was isolated in time, so you could actually see from the the different time points which brain response you were looking at. It was either a response to this set of tones or this set of tones or this set of tones. Which is really nice because EEG, electroencephalography this measurement of voltages on the scalp, is actually very precise in time so you can actually by looking at time know which signal you're seeing. So the melody on the left was four notes. And each of the melodies either went from a low note to a high note and back down which we called zigzagging. It could have been high to low to high and still be zigzagging. Or it started high and went low. Or it started low and went high. But to make sure that we made people listen to the end if it was a zigzag melody, the last zig or I guess it's the last zag happened on the last pair of notes. So you had to pay attention through the whole thing to know whether it was zigzagging up or down. So just a way of forcing people to pay attention. And what I'm going to do on this here is just plot all of the voltages from all the places on the scalp for exactly the same mixture of sounds -- that mixture of sounds from the left right and center -- but up on the top, I'm going to plot it when you're attending to the left stream, which has four notes kind of shown here. And on the bottom I'm going to plot it when you're attending to the right stream. But it's exactly the same mixture of sounds and they look pretty similar. Until you look at the details because after every onset that you're attending, there's this stereotypical negative positive negative response. The differences in the magnitude come from differences in place on the scalp and reflect the fact that sound evokes -- that sensory response tends to evoke strongest responses at the top of the head for reasons I can explain or I'm sure Elyse could explain better. But if you look at the right, it's similar, but the timing of the events is different because they're aligned to the onset of the attended stream, not to be ignored stream. So literally you can look at the scalp and tell what a person is listening to. Inyong actually took this a step farther and was able to predict for a single three-second trial like this, which of the streams the person had attended based on knowledge of what their brain did on all the other trials at better than chance performance, for every single listener. So one single trial, if you know what the structure of the signals are, you can tell which one the brain is responding to. Which is pretty neat. And the other thing is there's a large individual variation -- in this kind of test you can get big variability in how well people do, and it turns out the variability is related to how well you modulate the response in the brain. So this is to the same sound -- on average there's a big response when you're listening here at this time point to the blue, but not when you're listening to the red. So if I look at the change in the brain response, the same signals but it's big here and small here that's a a modulation of the brain response. I can measure how big that is on the individual listener, and that turns out to predict how well they do on the task which suggests that there's actually differences in how effectively you can focus attention that you can see in the brain signal.
So this is the data. This is the change, the modulation of that brain response, versus how well people did on the attention task. For 11 listeners, and you can see there's a pretty strong relationship. It turns out to relate to other features in the brain signals but this is pretty neat. That how effective you are at suppressing the unattended sound relates to how well you can do the analytic task of answering what the sound was that you're attending. It's also interesting because Scott Bressler just published a study, and I think I didn't even update the slides because that was published last week it finally came out -- looking at a bunch of veterans who've been through blast. And it turns out they aren't very good at these kinds of tasks at all. And we studied these veterans who had been blast exposed because there's lots of evidence that they have lots of cognitive problems. One thing that's kind of hard to tease out is, they may have some hearing damage too because they're veterans, they've been exposed to lots of noise. The blast exposure itself could cause damage to the cochlea. We did some control experiments to convince ourselves we don't think that there's damage to the cochlea or to the sensory apparatus that explains the differences that I'm about to show you. We think it really is a central deficit having to do with the blast wave that goes through the brain. When the blast wave goes through the brain it's this physical wavefront that passes through the brain. The brain is not one single monolithic thing, it's got different material it's got skull and air and and soft tissue, so that blast wave travels at different speeds depending on the medium, which means that as you're going through the brain you're to get faster and slower parts of the wave which is going to cause shearing forces on the on the brain and that may damage long-range connections in the brain which could lead to the kind of cognitive executive function deficits that people often have. I know that I missed him because I was not here during it, but Eric Galen was here earlier in the week talking he has a whole session on this. So we had this example that we did with a bunch of veterans who are blast exposed to see if there were cortical deficits in this population. We did exactly the same kind of task, it was three different streams. They actually had pitches that were very separate so this is pitch and this is time, and again they were doing exactly what I just showed you attending left or right to these different melodies. This is a a plot of the control subjects it's the same data I just showed you. But now what I've done is I plotted on top of each other the responses in the brain on average across this group for exactly the same stimuli. But when they were attending to the red onsets which are kind of vertical lines and red, when they're attending to the blue onsets or when they were told not to respond. So a third of the trials we actually -- the screen would come up and say listen left, or it would say listen right, or it would say don't do anything. And every time anybody got a right answer we give them a penny. Which on the trials that worked. The passive trials meant they couldn't answer anything or they didn't get their penny. It's amazing what a penny will do because these are really kind of boring experiments, but a penny per trial is enough to keep people on task. And they really focus, they want that penny. So these are exactly the same stimuli it's exactly the same inputs to the brain. The brain is changing what it's doing, and so after a red onset you see a strong negative response when they're attending to the red but not when they're ignoring everything. Or not when they're paying attention to the blue stream. Conversely blue is bigger than red, red is bigger than blue, the attempt that there's actually that third stream from the center they never attended actually has onsets that never elicit big responses, blue bigger than red, red bigger than blue. So they're doing exactly what I showed you before, it's just a different way of plotting the data. Here's the same data in a group of mild traumatic brain injury patients. They don't show the same modulation. In fact the ERPs are noisier to begin, with but there's no modulation to speak of. Some of the listeners got a little bit, most didn't get much. Here's the behavior -- again this is the data that I didn't show you, but it's the data from that beginning, that first set of control subjects with Inyong's study. This is how often they got the right answer for this particular version of the task. This was an easy, easy version of the task for normal subjects. They were around ninety five percent correct on average. The worst one was at about 85%, you can see the individual subjects in circles. This is the percentage of the time on those third of the trial -- this is the percentage of the time that they failed to respond in the little time window that they were allowed to respond in, on the two-thirds of the trials where they had to say what the sounds were. And they had very low likelihood of not responding. And this is that the percentage of the time when they were told don't respond or you're not getting your penny, and they incorrectly made some responses regardless of what it was. And most of the subjects were quite low. There's one dude who -- I think he's ADHD -- I know there's one guy who just couldn't control himself. I don't know. But in general he's clearly an outlier. Here are the data behaviorally for the blast exposed veterans. The best subject is still barely at the 50th percent -- I mean the 25th quartile the worst subject is actually below chance level which is one-third, although chance performance here because of the number of trials we have, this is statistically not different than chance. But you can see most of them, the mean is sitting around 62 percent. These people cannot do this task. Now it's not that they couldn't do is the melody going up down or zigzag. Before we even take this data we test them on just that task by itself. Here's a melody, is it going up, down or zigzagging, no problem. You put that in a case where there's another melody at a completely different frequency range from a different direction, and they fail. They fall apart, but they don't just follow apart there. They're actually also unlikely to answer -- of course that could be because they're not sure, they're trying to work it out and then time's up. But they even are worse at withholding responses. That's an executive function problem. They cannot withhold the response when they're supposed to withhold a response. So they clearly have executive function problems here. And like I said we did some tests, it's in the paper we convinced ourselves there are some that there they don't have -- as a group -- statistically they're hearing isn't really different than the normal group. That's not because they're hearing is great, it's because the normal control group of young adults who are college age have huge variation in their hearing ability, even when they have normal thresholds, and again we'll come back to this at the end. So our blast-exposed veterans look a lot like the worst of our subjects who come in every day, statistically the two groups are not different because it's not a large number in either group, but the vets -- the hearing difference doesn't explain this because even the worst of our normal controls is far, far better than this group. So that's kind of the first part.
Interim summary
So what did we talk about? The big picture, the big take-homes: auditory attention allows us to communicate if you can segregate the sources -- if you can separate them. And if you can do that the cortical responses in your brain are actually modulated, as long as the processing that allows you to do that executive control is working.
So how many of you -- we are going to change gears a little bit -- how many of you know what a neuron is most. people know what a neuron is. How many of you, would you say that a neuron spiking, firing of neurons is that where information lies in the brain? Can we kind of agree on that? Most people if you ask them -- I mean I can believe that the information is in the spike patterns of the neurons, right, there's information going on and there's electrical activity. But one of the things about it is in order for those neurons to work well they need to talk to each other. The traumatic brain injury, that blast exposure is causing damage not necessarily even to the neurons, it's causing damage perhaps to the connections in the brain. This is an image -- I love this image -- of all of the white matter. The connections between parts of the brain colored to make it clearer, and you can see these big sheaths, it's like wires truly like wires connecting different regions of the brain that have different function. And the thing is, a neuron maybe spiking and doing the right thing, but if its message can't connect to the other parts of the brain that need to know what its computing, it doesn't matter so the interconnected networks in the brain are really, really critical for you to hear in a complex setting where you need executive function, where you need the front part of your brain to help you out. And it's those connections from parts of the brain that are making decisions about what's important right now that are sending modulation back to other parts of the brain to suppress the things that are unimportant.
Focusing attention causes preparatory brain activity
So we did some studies using MEG. MEG is a lot like EEG, it's measuring electrical activity, but it's doing it by measuring the magnetic fields generated by those currents in the brain. And it it gives you a slightly better look at some parts of the brain, but it has this property of having very fine time resolution just like EEG. So this is a study that KC Lee who was a postdoc at the time, and Hari Bharadwaj who was a postdoc at the time --also they're both former students, both were still working with me as postdocs, now they're both professors. Hari just started at Purdue. But we did an MEG study and it was a really simple study to try and understand what is it in the brain that allows you to focus on different aspects of sound. So we had a cue that was a visual dot, and it came on one second before the sound that you're about to hear. And you're supposed to fixate on that and hold your eyes there. And then at time zero a pair of digits came on, just one pair of digits. One was high-pitched one was low-pitched. One was from the left and was one was from the right. So you could listen to those digits, you were supposed to listen to one of the digits, and then respond. But the cue you told you how to listen. You could listen for the source on the left. Or you could listen for the source on the right. You could listen for the source that was low pitched, or you could listen to the source that was high-pitched. So there are two ways of doing exactly the same thing -- one paying attention to a spatial feature and one paying attention to a more acoustic feature, the pitch.
What I'm going to show you is a movie of the brain, and parts of the brain that were more active statistically than average on a pitch trial. And the first movie -- so this is a blown-up side cross section of the brain this is facing actually for you it's like this -- this is the back of the brain, the front of the brain. The back of the brain is where visual inputs come in and the first thing that happens is we give a visual cue that tells people what to do. And that happens at minus 1000 milliseconds so this is a little time stamp back here. So the first little snippet of movie is just that. Just you'll see activity in the visual cortex. Boom little bit of activity -- this is statistically thresholded to throw out a whole ton of noise. But statistically there's a little bit of activity because the visual cue just came on and told you what to listen for. The next piece of movie gets a little more interesting. Most of the movie is at negative time before the sound you're about to pay attention to is on. But it's starting after that little blip of activity -- you now know what to listen for. And what's cool is you'll see activity in the brain that's significantly more than average activity in different regions in preparation for the sound that's about to happen. First you'll see an area which is the frontal eye field area, it's an area associated with moving the eyes, which is really closely associated with attending to things at different places in space and is thought to be part of the visuospatial attention network. But guess what it's not just a visual spatial attention network -- because this is a sound, and you're not moving your eyes, you have fixate your eyes, yet you'll see activity in the frontal eye fields part of the visuospatial attention network, because you're listening spatially. After that happens, you'll start to see a little bit of activity in the more frontal regions of the brain. This is a latest evolutionarily speaking part of the brain, and it's the part that makes decisions, and it's the part that controls what you're going to do. Around just after that little bit of activity starts, suddenly you're finally up to the time when the sound comes on, and then you'll see areas in auditory cortex come on, and when the sound actually starts you'll see all of these areas talking to each other. So here we go. So you see the end of the visual cue the frontal eye fields are engaged, sound still not on, there's this activity boom, sound comes on and the whole brain talks to each other this whole network is talking to it each other. This is what happens in a spatial trial. Tou engage this spatial network. If I just kind of look at that spatial region and look at the activity over time, this is what happens to that region -- just the energy in that region when I'm paying attention to space. And this is for the exact same trials -- or not the same trials, exact same statistically stimulus, but when you're paying attention to pitch there's more activity in this area, when you're paying attention to space then when you're paying attention to pitch.
If I look, however, at a different region of the brain, an area that's associated with auditory processing, that activity is actually bigger when I'm paying attention to pitch than when I'm attending to space. And in fact in both sides this is when the sound turns on, this is the evoked response from the sound both areas so a little bit of activity that you can't localize with MEG perfectly but that's much bigger in the auditory areas. But all of this stuff is happening before the sound happens. Because the brain knows what it wants to do, and it's preparing to shut down the stuff that's inconsistent that's about to come in.
Auditory and visual attention engage different brain networks
So I've shown you EEG, I've shown you MEG, I also collaborate with David Sommers a professor who does vision work. And he's an fMRI guy. Now fMRI is a way of looking at brain activity that doesn't have good temporal resolution, but it allows you to see things with great spatial precision, what parts of the brain are engaged. And this was work that we did with Sam Michalka, who just took on a job as a professor, and Abby Noyce a current postdoc with us. And we decided to compare what -- we have this suspicion that the visuospatial attention network is not really a visuospatial attention network. So we did tasks where we compared attending to sound versus vision.
So in the first experiment what we did is we had sound and we had visual inputs, and I won't go into the details but it was kind of comparable tasks in the two domains but both sound and vision are on all the time and you're either attending to the sound or either attending to the visual inputs. This is data for three different subjects and what we've done is color-coded areas of the brain that were more active when you're paying attention to the sound, there in warm colors, or more attention to -- more on when you're attending to vision, in the cold colors. So the cold colors -- visual areas, unsurprisingly in all the subjects in the back of the brain. But if you look carefully in the frontal regions of the brain there's a blue region and then another blue region. There's a blue region and another blue region, across all subjects across both sides of the brain. Now people know that these areas are involved in attention, but one of the things that's kind of funny is most of the studies that have looked at multi-sensory attention have kind of said, well all of these areas up in here are kind of engaged whether you're paying attention to sound or vision. Part of the problem is if you look at how these places align across subjects, it's pretty poor. It's very hard to get them to align because different people's brains anatomically are different, so it's hard to get them to align. What we did is actually said, okay, we've got four individual subjects this really clear pattern of blue and blue, and if you look inter-digitated sound gets the auditory areas and yellow and yellow. And everybody shows a similar pattern, but their individual areas are different from one to another. If we use those areas defined for that person, can we get interesting results out? And the answer is obviously yes or I wouldn't be telling you.
So one thing you can do is say, ok I've got these regions of interest, these ROIs. I'm going to define them for the individual subject based on whether they were more turned on for it for vision or audition. And I can also see sensory areas that are visual and auditory. If I look at those and define those I can actually plot they energy as a function of time in fMRI, and it's smeared out it's nothing like the precision that I just showed you for EEG or MEG, but you can still get this time course. And then you can ask a very silly game or ask a silly question -- if I just put you in the magnet and don't ask you to do anything special, what parts of the brain tend to turn on together? If this time course is on at the same time as some other part of the brain statistically at the same time, that's called being connected. It kind of suggests those parts of the brain tends to work together. When I was an engineer when I first kind of got into this field I thought it was the silliest thing in the world you're putting people in a magnet and measuring this brain activity, not telling them what to do. What a waste of money and time. It's a lot of money. It's five hundred dollars for me per hour to get this person to sit there. Why are we doing this? But the answer really kinda cute: It's like a big data problem you're saying I don't know what's going to happen, its statistical and on average, if I do this enough, I'm going to get actually interesting answers out. So what you do is you look and see what parts of the brain are correlated more than chance.
And we had a hypothesis -- we hypothesized that if I look at those visual areas in the frontal regions they should be connected to visual regions, and not connected to auditory regions. And if I look at the visual sensory areas they should be strongly connected to the visual frontal control areas, but not to the auditory. Conversely auditory sensory regions where the sound is first represented should be talking to those little orange dots that are auditory, but not to the visual ones.
So that's our hypothesis and here are the data. If you look at the little blue areas and ask are they connected to this blue region the answer is yes, bilaterally, both sides of the brain. But if I look at the orange areas they're not. If I look at the auditory regions they're connected, they are strongly correlated with activity in the auditory sensory regions, but these two areas are not strongly connected to the visual regions. So it really looks like these areas are kind of specialized, sort of.
But there's this one little twist. And that one little twist is actually kind of interesting. This is the frontal eye field, this network that includes these two things is really what people have called the visuospatial attention network, and also includes areas in the back of the brain up here in the parietal cortex. This is the area that the MEG study I just showed you said was involved in a spatial auditory attention task, yet if I just emit resting and just doing random stuff it doesn't connect to the auditory regions.
But it turns out you can see connectivity to that region in a task that has spatial demands that's auditory. So if I do an auditory task and try and make you do a spatial --an auditory input and try to make you do a spatial task we predict, we thought going in that that might recruit auditory regions might recruit the visuospatial attention network. If I did an auditory task that was temporal, it shouldn't recruit the visual network. We also wondered if the converse might be true too. Sound is really -- auditory inputs are really processed with great temporal precision, but not visual inputs. If I do a visual task and it has high temporal demands it might recruit the auditory network even though a visuospatial task doesn't need to recruit that network. And again I'm showing you the data, it's published it's out there, we got what we expected. If I do a visual task whether its spatial or temporal, if I look at those two regions in the brain at the front of the brain that are kind of visual, they're on, they're turned on. But if I look at the auditory regions in this control area of the brain that are kind of auditory, if I do a visuospatial task the visuospatial task doesn't recruit those areas, but the visual temporal task does. So sound is encoded in a network that's kind of processing temporal information, and if I do a visual task that's got high temporal demands that grabs that network.
Let's look at the auditory areas. The auditory areas are on when I do an auditory task whether it's spatial or temporal. If I look at the visual areas they're on also in the spatial tasks and a little bit in the temporal task it turns out, that may be that's something that we're still pursuing. But what's interesting is they are more recruited, they're more strongly recruited, when the task is an auditory spatial task. It's grabbing this visuospatial network. It's not just a visuospatial network it's a network that is spatial, and vision is always spatial -- it's kind of the way we think about it.
Interim summary
So second quick summary: focusing attention cause preparatory activity. Before you know the sounds you are going to attend to, you know what they're likely to be, and you're going to process things differently based on what you want to hear. And that starts to happen before the sound turns on. Once you know what to listen for. If you see your husband's coming from over here and there's a lot of sound over here, you prepare to shut that off and pay attention over here. No, I didn't say that.
The brain networks are biased to represent different kinds of information, temporal versus spatial, but that actually means that the temporal tasks -- the temporal network is always kind of addressing auditory information and similarly the spatial network always encoding visual information. That's kind of the inherent kinds of information that those two senses encode to begin with. But you can show recruitment of the other network even if the input modality is auditory it will recruit the spatial network if the task demands are right.
Object formation and object selection both require exquisite spectro-temporal resolution
So, the last little piece I want to talk about, I kind of hinted at, which may be most relevant to some of you who do hearing stuff for people with hearing impairment, which is looking at how it is that things can fail if the sensory part of the apparatus is damaged. One of the things I like to think about is when I go to get my hearing screened, they ask me: do you detect that something is there? And if I say yes, they say great. Which is not really what it takes to make sense of a scene. If I went to a vision specialist and he said I was fine just because I can detect that blob, I would not be particularly happy to go out and drive.
So one of the things that's kind of interesting is the most overt forms of hearing loss really elevate hearing thresholds, but there's more and more evidence that you can have normal hearing thresholds, yet still have differences in sensation that are really physiological. And the first hit we had of this -- Dorea Ruggles was a student in my lab and wanted to study the effects of reverberation on hearing. And I was at the time about 45, and I was starting to have trouble in settings where there was reverberation and crowded settings. So we started to look at normal hearing listeners, and we thought we were going to study an age effect. And what I now call them is listeners with normal hearing thresholds because I don't think normal hearing is a really good descriptor for people who happen to all have normal hearing thresholds.
So the test is a lot like the kinds of things I just showed you. People were under headphones. We simulated sources from three different directions, each of these three directions had a stream of numbers and your job was to listen to the ones in the center, this is kind of time. So the right answer would be 9 2 6 1. The separation in space that we simulated is quite small -- it was 15 degrees. Which isn't very much at all, it's about what the limits of a good listener can do. Which was on purpose.
And when we tested people, we tested a bunch of people with different ages in the initial study. What you'll see is really we're kind of disappointed because there wasn't much effect of age to see. What we did see -- which was shocking -- were huge variance in ability. On this particular task we're doing, chance turned out to be thirty-three percent. That's because when people got the numbers wrong, it's not that they just reported a random number one out of nine -- they reported one of the three numbers, just the wrong one. They could hear the words. They all had normal hearing, we expected them to be able to hear the words. When they failed it's because they failed at using these really fine spatial cues to choose the right one. They failed in selecting the right one. Not failing to hear, but failing to select the correct word. So chance performance given the variation, we expect statistically anything above this grey stipple area is above chance.
Everybody's above chance. Some people are terrible. Some of our worst listeners are pretty young. Some of our best listeners are pretty -- I said that backward -- some of worst listeners are pretty young -- this guy. Some of our best listeners are pretty old. So age didn't turn out to be a factor. We actually got some extra older listeners in. Still not a factor. It turns out they were better than our initial older listeners. And notice these aren't really aged listeners, these are middle-aged listeners. Our oldest was 54, I believe.
What we then did is simulated reverberant energy, and every single person took a hit. Every single person, their ability went down. Now some people are in chance levels.
And then we simulated -- this is not a classroom. We made a classroom whose walls were like a bathroom wall so very very echoic. Every single person got worse again. Now the only listeners who are above chance, it turns out two much younger listeners, and two of the listeners are kind of the 30 age range.
It didn't turn out that age was a factor. if I just looked at age, I couldn't tell you whether it was a good listener or a bad listener just based on age. But if I told you which of the older listeners it was. Or if I told you the age of a listener and I know how well they did in the echoic case, the older you were the worst the reverberation made you. The hit that you took was bigger. So the younger listeners were more robust. They didn't go down as much as the older listeners. So age was not a factor, but the older listeners were hurt more by reverberation suggesting that there is something going on with aging.
But the individual variation even among the young listeners was what really shocked us. These are all young listeners with normal hearing yet they vary enormously.
It turns out there's some interesting work going on down the street from us looking at animal models, and looking at noise exposure. And it turns out you can get a loss of nerve fibers without an effect of threshold detection. This is now known as hidden hearing loss which is a cute term. More correctly it's auditory -- or cochlear snynaptopathy. What really happens is the synapses in the ear, due to noise exposure, are damaged. When they die off the nerves they enervate die and then you have fewer nerves coming out of the ear to convey information to the brain. So it's a really very peripheral, in my mind, sensory loss. But it's not peripheral in the sense that the cochlea actually is functioning perfectly. The amplification in the cochlea is fine, and you can show this.
It also turns out that neuropathy occurs with aging even if you don't have noise exposure. If you have noise it ends up speeding up what looks like a normal aging process. Which kind of starts to make sense if you think about the data I just showed you -- older listeners are more hurt by reverberation. They have a little less robustness. They have fewer nerve fibers. Some of our younger listeners probably were exposed more to noise, etc.
So Hari Bharadwaj again came in and looked at what happened for this supra-threshold hearing. So we're not testing things at threshold and seeing whether you can tell that there's something there or not. we're testing things that are clearly there and audible and asking can you make sense of them and analyze them and use selective attention well.
So we got a whole bunch of listeners in -- all younger because we knew already we would see great variation even younger listeners. They had normal thresholds -- within 15 dB of normal hearing up to 8 kilohertz. We didn't test higher, they might have some high frequency loss above that. But they also had normal cochlear function. We did a whole bunch of tests, and it's in the paper that's now out, looking at the cochlear function. Their cochleas were intact, the amplifiers intact. It has all the right properties. What small differences were there in their cochlear function didn't correlate with any of the data I'm about to show you. Which suggests there is something that correlates.
And what we did is measured again using just voltages on the scalp how well the sensory apparatus is encoding fine timing information. You do this exactly like you do when you're going to measure the brain, as you normally think of it that kind of cortex neocortex. But if if I look at high frequency sounds and put in a sound like this.
The DA syllable. Courtesy Nina Kraus who has done a ton of work with these sorts of experiments. You play a DA syllable. For our purposes, the nice thing about the DA syllable is it has a nice pitch. This is a hundred hertz pitch. That hundred hertz is so rapid that the cortex doesn't fire that fast. It can't follow that fast a rate. But lower parts of the brain, the subcortical portions of the brain can fire at a hundred hertz can entrain to the hundred hertz stimulus. So if I just filter out the low-frequency stuff and look at just the high-frequency stuff, instead of looking at cortical responses like everything I've shown you so far, you actually look at subcortical early parts of the brain. So you can look and, say, play the same sound over and over and over and just average it and you can get out that. Which is a noisy thing, but you can hear the pitch in it, right?
Well you can hear that pitch. It's in there. It's noisy/ This is a track just made by one listener sitting for many many trials and we average the heck out of a bunch of trials in response to this DA syllable and you see this response.
You can then come up with a metric for how strong is the subcortical portion of the brain encoding the hundred hertz information. And that metric turns out to be very predictive and very variable across subjects. Again for the sake of time I'm not going to go into the details but you can look at a measure of that of how strong is that response and up in this graph is more fragile, so better listeners are down here. And then you can look at how well do they do on an attention task, and there's a negative correlation. It's quite strong. The people who are weaker in their brain response do poorer on this attention task. The people who have the strongest brain response are the best on this attention task -- tests just like I've shown you already.
But it turns out almost anything you do, it seems, that's really probing really fine timing information in the brain. These listeners all -- these measures are correlate. The envelope following responses -- that measure I just talked about in vague terms -- that measure of the subcortical ability to encode this hundred hertz sound. That correlates with selective attention ability I just showed you that. It also correlates with how well you can pull out fine amplitude modulation in terms of a sound. It also correlates with how well you can detect a little frequency modulation of a sound. It also correlates with how well you can -- what your discrimination threshold is for interaural time differences of sound. So all of these things are features that really rely on fine spectra temporal timing. And there's something that you might think might go wrong if you had fewer nerves coding things. Every single auditory nerve is a little bit random. The normal ear has many different nerve fibers encoding almost the same information, but with different randomness on them. And the brain takes them and puts them together to get rid of some of that noise, to get better temporal coding. If I have fewer auditory nerves I can't get rid of as much noise. I can't average it out the same way. So it's these kinds of things where these problems show up and interestingly that turns out to really affect selective attention ability in a crowded setting. That's where you need to be able to pull apart sounds you need to have a really fine representation. When it gets muddy it's almost like that picture of all blue. I can no longer segregate the sound so I can no longer effectively select the right sound. I can't segregate them or select them. And in fact the very features I might want to use to select them may not be as clear, like the location cues may not be as clear.
So these are two examples. This is the modulation threshold and an amplitude modulation threshold. And here it is I don't even read -- this is versus the brainstem response strength, and you can you can see a nice correlation people who have a stronger in this case -- excuse me, a weaker brainstem response on the top. This is how big does it have to be for you to detect it. And if it's big, it's bad. And similarly how big is the modulation have to be for you to detect it, and big is bad. And so again there's this nice correlation. Here's that same behavioral AM detection threshold versus interval time difference threshold. And again there's this nice correlation.
So listeners who are all young adults, who all have normal cochlear function -- we would say our normal young adults -- differ in their physiology and that correlates with their behavioral ability. So this is, I think quite real, and it's always the things that it shows up in are tasks that require coding of fine spectrotemporal features. It turns out also it was sort of a post-hoc thing after we started this study we added a little questionnaire and ask people to self-report their noise exposure. There was actually a relationship. The people who reported by their own self-report more noise exposure tended to be the worst subjects, and it was statistically related.
Now there's been some other studies since then including one very large scale study out of England that doesn't find a relationship between self-report and these sorts of measures, but in talking to the guy who did that Chris Plack, he's a great researcher. The one thing he hasn't looked at -- so he doesn't see a relationship between self-reported noise exposure and the measures I just talked about -- but what he hasn't looked at is whether there are self consistent differences. So self-report is pretty unreliable, I'm surprised that we found it in fact. And it also may not capture everything that matters like genetic predisposition, and you know all sorts of other things. So self-report may not be really a good reliable measure, and even noise exposure may not be the only thing that matters here. But what we see and what Chris hasn't yet looked for, is we see individual differences that are very, very consistent that show up in the physiology objectively, and that show up in behavior in all sorts of ways.
Summary
So I think I will end here and the last piece of the talk, the thing you should take home, is that there are big differences amongst hearers who we think of as normal, with normal hearing thresholds. and normal cochlear function that are different in their sensory abilities. That also has an enormous impact when it comes to hearing in noisy environments because you're feeding that central system a representation that's not rich enough to make sense of. And we think that this is a form of cochlear synaptopathy, although showing that that's what it is in humans is basically impossible directly because we cannot do a basic kind of study to look at that. But with that I'll stop and take questions if there's time. Thank you.
Questions and discussion

Audience Question:

My name is Greer Bailey. I'm from West Virginia University. So instead of just asking for a noise exposure history, if you also asked about history of exposure to ototoxins, for example opiates or solvents, will you get more robust relationships?

That's a great question. So the question about ototoxicity. Almost anything like that would probably add power to looking at these relationships. I mean one of the things that's kind of funny is that these effects seem to be robust enough that even though we did these things, you know kind of post hoc that there was any relationship at all. So ototoxicity, genetic predisposition, all these sorts of things. The other thing is self-report is such a weak variable to look at. right. Chris Plack, who I mentioned earlier, is actually in the midst of doing a disometry study where he actually has people carry around a device to test their exposure. And I can't remember the details of how he did it, I think he's using the cloud and he's gonna ping some device at random times and sample what the noise exposure is at different times and stuff. But something like that is really kind of the only way to really get a clear picture of what noise exposure is happening. And he's going to be doing this longitudinally over a four year kind of study. More power to him. I mean that's that's the only way you're going to show cause and effect here, and like I said everything, with respect to this neuropathy, everything you can do in the human is very indirect. You can't use invasive methods, so you're trying to intuit what's going on with these really crude methods that can't look under the hood, right. So that's really the limit, but you're right: ototoxicity, genetics I think is probably another really big important factor. Great thing to look at.

Audience Question:

I'm Adam Bosen on a postdoc at Boys Town working with Monita Chatterjee. My question is, I was really struck by the results you got from the traumatic brain injury subjects, the stuff you did with Choi. I was curious, do you believe that the effect you're seeing is more of a global attentional mechanism -- so if you did this in a cross modal competing attention task, would you see the same results? Or do you believe this is something that is specific to an intermodality competition?

I think that's a great idea. In fact my interest in traumatic brain injury first grew out of the collaboration I had with David Sommers, the vision scientist. We have these different networks. You can imagine, in a beautiful world, you can imagine funding agencies that gave us money to look at the following. Imagine that this these waves are coming in from different directions. There's different networks controlling temporal processing and spatial processing. We had this big goal of getting subjects in and seeing if the pattern of variation -- whether there might be variation in the deficits. We think it's probably, a lot of the kinds of things were probing, probably have to do with the long-range connections -- that kind of rainbow picture that I showed you with all the connections. We think those long-range connections are the ones that are most susceptible. Those are exactly the ones that are involved in those kinds of long-range networks, and for sensory processing it seems that there really is, there's tons of evidence for sort of two main networks, a more dorsal and a more ventral network which turned out to kind of map nicely to our spatial attention network, which is cross-modal, and a more content processing one, which always engaged by the auditory system. So you can imagine differential damage to those two different networks depending on the person. And David has pursued this by getting studies just from -- there's a large-scale project called the Connectome project which has a ton of fMRI data and looking at just people at rest. And he shows that you can find those four little areas that we defined in our study together, you can get those four areas just by looking at random data from this database, you can find those four areas on an individual subject from resting state data. So you can define the areas they're really strongly there. So there's all this evidence that these networks are real, the damage to the network's might be differential, and you could then maybe tease them apart by doing tasks that are more temporal or spatial. So we think probably thinking in terms of those networks is the thing that's probably going to pay off. Some subjects -- what gets really funny is these blasts guys, most of them if they've been in Afghanistan or Iraq, they've been blast exposed seven, eight, nine times. So they've probably got damage to all of them. And they, you know, any executive function that requires a -- the motor action, they can't even not push the button kind of problem. That probably isn't a sensory problem. That's probably the motor response is something they can't even inhibit, right. So it may be hard to find people where the damage is specific, because many of them probably have a lot of damage to lots of networks in the brain. But for for the purposes of of the intellectual exercise, we think that really was something we were interested in. Because you could you could look at the resting state data and get differential measures of the strength of one network versus another and predict, well this guy's got an intact spatial processing network he's gonna though maybe have trouble with temporal processing and attention to temporal features. So yeah I think along the same lines as you, but that's really where our heads are at when all of this started. Scott Bressler who was involved in that study and veterans is actually now working with folks at Walter Reed Medical looking at current service members who are coming into the audiology clinic and looking at some of the same metrics in them to see if if we can get more data on this. So thanks.

Audience Question:

Gabrielle Merchant. I'm at UMass right now with Karen Helfer. I too like kind of smaller maybe questions. One kind of follow up Adam. The TBI data was really interesting. Did you look at some of the subcortical measures in them.

So, I hadn't spoken about the subcortical measures when I talked about the TBI data. The measures that we did in these guys, we have their thresholds they were within normal hearing, we threw out people that didn't have normal hearing that was one of our criteria. But what we did is the subcortical measures. And on these EFR, these envelope following response measures of the subcortical responses in the brain, they were in the bottom quartile. But normal, compared to normal young adults. And now that gets kind of interesting because one of the weaknesses of that study that I didn't go into is our TBI patients in that study were veterans and most of them were in their thirties. Most of the people we're comparing them to were people from our studies who are grad students and undergrads, so it's not a good age-match control. There's all sorts of not great control for each other. But that said, their EFRs were in the normal range and the worst of our control subjects was worse than the ones than the TBI patients. So that's exactly what we did.

Audience Question:

And then for that zigzag task. I know you said they could do it, like tell the modulation independently of having to selectively attend when multiple sources are playing. I'm assuming that the SNR was like 0 -- everything was playing at the same level when they had SN. Did you look at the effect of SNR, like if the sound they are attending to is, louder do they do better?

Yeah that's a great question too. I didn't do that, we didn't do that. We had pretty limited time and so didn't get to that. That's a that's a really interesting question, but everything I know about attention in general is that there's a competition for attention. And if things are about equal then that's where attention can have the biggest sway. There are some things that are so salient, they'll grab attention even if you're trying not to pay attention to them. If I were to suddenly slap the table -- I won't, some people are sleeping and I don't want to disturb them. But if I were to make a sudden sound, that would wake people up. That would grab attention even through sleep. So a louder sound is more salient inherently. Tt takes more effort to suppress it. So I would imagine actually as I say this it's kind of interesting -- maybe they'd actually -- they would definitely do better if you got the SNR higher. It gets kind of interesting because I would expect that they would drop off even faster with SNR. That there might you know -- that when normal people, normal people can actually attend using the quieter thing as a feature. So people Doug Rumgard has done some really nice studies of competing talkers. And if I make one talker 4 or 5 dB quieter, I'm actually better at hearing them then if they're the same level as the other talker, because that itself is a feature to listen to. I'm sure that wouldn't happen in somebody who has no attentional control, right. So SNR is a variable we didn't play with. There's some really interesting things one could think about though. That could be a neat future study.

Audience Question:

I'm Holina Shiminsky, I'm a PhD student CUNY in New York. I have a question which is probably not -- there's no easy answer to that. But I'm just wondering what are your thoughts on reversibility of all those deficits. You know thinking along the lines of auditory training.

So the question is Nina Kraus has a bunch of really nice studies showing effects of training on that very frequency following response that I talked about, right. And I'm here to find a lay claim to the fact that it's a sensory representation, which suggests that you can't train it. So it is a complicated thing. I think both things are true, and I think the fact that -- despite the fact that plasticity can affect things you still see a correlation with basic sensory responses and other things -- suggest that you can train it, but only so much. There's some behavior -- there's some animal data rather in Dan Polley's lab, who has done some stuff on where he oblates the ear pretty severely, and you see gains, you see increases in gain when you produce less input into the midsections of the midbrain. The mid brain compensates by amplifying what's there more. Which can compensate to some degree. It can't really get back all of the timing information. And so I think it may be -- I think both are true, and I think that interplay is something we have to still work through, but what it is -- it's an interesting mystery that can be both reflecting sensory processing, yet also somewhat plastic.

Audience Question:

My name is Yi-Hsin a PhD student at University of Illinois at Urbana-Champaign. I have a question regarding the attention piece of information you have. So I know in realistic for clinical or experimental design we always want to kind of direct our patients or subject to focus on one sentence, especially for speech in noise test. You also demonstrated that when we focus on the seven digit number we kind of ignore others signals. So how would you evaluate like current clinical speech in noise testing or will have any other recommendations on like what kind of study design will be better evaluating like speak in noise abilities?

That's a great question that I haven't really thought of in those terms. And I'm going to pause because I can opine all I want, but one of the things I've learned is -- I don't have, I've not been in the clinic and the clinicians know a whole lot more than I do about what's appropriate and what works in the clinic. The speech in noise tests that are done are actually quite good. I mean they get at something pretty fundamental about everyday listening. I guess to me the problem is there are multiple ways that speech in noise can go wrong,. And a speech in noise test can identify people who have trouble in everyday settings, but it doesn't tell us whether it's an attentional problems in the network that's controlling attention, or whether it's a sensory representation problem. Because if I make things muddier in the sensory representation, I'm going to fail at the speech in noise task. But I could also fail in the way that the TBI patients do, which is they seem to have okay representation that's enough if the central cortex was intact, but they still fall apart. They could also fail that speech in noise task. And so I think one of the things that one of the great pressures when I talk to clinicians one of the great pressures is time: time time time. You have so little time with the patient. a speech in noise task is getting at all of that in a mix, and that's probably the probably the best you're going to be able to do. And the other feature of this that I think is as -- theoretically just killed me but, we had some ideas for how some of these EFR tests might become useful as sensory diagnoses for is it a neuropathy or not. And either using an objective EFR task or using some of these behavioral tasks that seem to be sensitive to neuropathy. And we talked to the Patent Office at BU, and they said, "Do you have a treatment? DO you have a way to make the hearing of these people better now that you know, once you diagnose them?" And I said we're working on that with great enthusiasm. He said, "Come back when you do, because otherwise there's no point." So a better diagnosis is not really helpful unless you know what to do with that diagnosis. And that's a really hard pill to swallow for someone like me who is a very theoretical, likes it in my lab kind of person. But that's the truth, right. And so perhaps the measures you guys in the clinic already have are as good as we should have until we know how to treat different forms of impairment.

Audience Question:

Rich Tyler from Iowa. You mentioned that different people have different brains. I wonder if you could elaborate that and tell us how it influences your research.

Different people have different brains. All I meant by that somewhat provocative statement taken out of context. For my purposes what I meant was the anatomy is so different that if I'm trying to align a functional region that's the auditory region with the functional reason that's an auditory region in prefrontal cortex for two listeners, I can't do it using some automated thing based on anatomical landmarks. The anatomies of people's brains is different. It's like an earlobe or a fingerprint in that they're statistically kind of similar. There's some features that everybody has that are functionally kind of common. But the prefrontal cortex is an area where it's kind of the Wild West in terms of that anatomy. That's where the individual variation in anatomy are greatest and that's where trying to align things functionally based on anatomy doesn't work well. And that's all I meant, but I could again opine a long time about differences in people's brains. I haven't really studied that per se, other than to look at differences and effectiveness in a very, very limited task like the ones I talked about.

Audience Question:

So it makes it more difficult to average across listeners for brain regions?

Yes absolutely. And in fact, and I didn't really emphasize this as much as I might have, one of the things that people do in fMRI is to use anatomical landmarks. So there's different folds in the brain that everyone has that divide things like the temporal lobe from other major sections of the brain. Those anatomical landmarks you can identify automatically, and that's what people do they'll identify key features of the brain automatically. And then they have mathematical algorithms to take your brain and kind of morph it in the smoothest smallest distortion way possible onto an average brain. So they take the activity that they measured in this configuration and they kind of squeeze it onto the average brain aligning all those key features. And then they average activity across the people. Which, if the anatomical landmarks are the places that are functionally the same that you're trying to look at, it works great. It doesn't work well at all for the very region that we that we were looking at. And when people have done studies of visual versus auditory attention and done those kinds of techniques, what they end up concluding from those studies is auditory and visual attention get this broad set of regions up here, and it's all mixed together. But really if you pull that apart and look at the individual level it's because every person has this pattern, but they don't align well, and when you go through that warping process to average they smear out ,and the auditory with this person overlaps the visual of that person.

Audience Question:

I'm Claudine Vielo of New York, I work at Montefiore Medical Center. And my question was, I couldn't help it, I'm an audiologist. Why use DPOAEs instead of TEOAEs? Why use distortion products as opposed to transient evoked?

I will talk to you afterwards. Here's why: because I am ignorant about these things. I mean literally about five years ago I wasn't doing brain imaging. About three years ago I just started doing DPOAEs, and I'm happy to learn more about the different advantages of different ones. It worked. We did it, we tried it, it worked. Chris Shara who's an expert on the autoacoustic emissions is down the street and good friend -- he was down the street and moved to the other coast. But we talked to him, we got his opinion. He told us what to do. We did it, and it worked. But come find me because you have information that I would like to have.

Audience Question:

Caitlin Masterson University of Nebraska-Lincoln. I kind of have a follow-up question on the veteran TBI sort of thing. So you said that what the result looked like was that the they couldn't perform the task because it was like an executive functioning problem and attention which are typically associated with later latency ERP components around three and four hundred milliseconds. Has your lab ever looked at like the early sensory component such as p50 and auditory sensory gating to see if the problem is actually occurring earlier?

It's funny. So a lot of these attention studies you actually see modulation of pretty early responses. In fact Inyong Choi's stuff which uses those different melodies which is the task we used the vets, it's the n100 that shows the biggest modulation due to attention. Which is kind of interesting if you think about the fact that the brain is preparing before the sounds start to listen left or listen right. Based on some of the other data I have shown you it makes sense that the n100 would already be modulated when the sound first hits kind of auditory cortical areas. We don't really analyze the p50 which is an even earlier response. We could it's a little noisier, We tend to focus on the n100 and we'll receive whopping large effects but most of the analysis I showed you or talked about today using EEG really was looking at modulation of n100 as the attentional modulation index, as an index of how strong the attentional modulation is. And you know it is normally thought of as an early sensory -- relatively early sensory response. It you know is thought to arise from auditory primary and secondary cortices. Yet it's modulated very very strongly in these kinds of tasks where people know ahead of time what to listen for by attention. And so that gets pretty interesting. I would also say though that one of the reasons you see it here that you might not in other studies is you only are going to see that kind of attentional modulation when there's competition. And we've been following this up a little bit -- if there's nothing, if there's only one thing going on in the soundfield it will get represented strongly regardless of attention. And so it's only once the competing sounds turn on that the attentional effects show up, in response to competing sounds. So that preparatory activity even though it's going on, it doesn't seem to modulate for instance the onset of the first distractor. So in a lot of the tasks we have we have a distractor that comes on first time in front -- you know you're gonna listen left or right. It doesn't matter if you're going to listen left or right it turned out. But that might have been just because it's in front, right. Correct, right. But we've since done studies where the distractor that starts first could come from left or center or right and you know you're going to listen to the right or to the left -- that doesn't modulate the first distractor which is all alone. You get a really whopping large response because it's the only thing going on. It's only after than that other sounds start that the modulation due to attention reveals itself. So that's a long-winded way to say, p50 was a great idea to look at -- I mean that's a good question. We don't see it there, but we haven't looked for it there, but that's because we've been focused on the in 100 which is also an early sensory response and we do see stuff there.

Audience Question:

My name is Calli Fodor and I'm from the University of Maryland. And my question was, if you could give a definition of what you're using for normal for EFR. You saying the people have normal EFRs?

Yeah I probably shouldn't use the word normal EFR that's -- you caught me. What I probably meant when I use that phrase if I'm being more precise is to say EFRs that are within the range of typically developing college-age students who have no known hearing deficits and are coming in to get $15 an hour in our studies. Which is different than picking up somebody off the street and paying them $15 an hour for their studies also by the way.

Audience Question:

My name is Samantha Gustafson I'm at Vanderbilt University, and I'm wondering if you can talk a little bit about what you think the role of experience is in the development of those attentional network between auditory and visual areas. I'm a pediatric person so that's where my brain went.

That is a great question. You know I have nothing to say other than saying the thought of doing a developmental study just awes me because it would, the logistics involved are just so daunting to me. We did one study that I talked about today with older listeners. It took us two years to get that very small number of older listeners in. I mean we're just not equipped to deal with different special populations, it's not what I've done. And the idea of looking developmentally it's fascinating and would be great, and some of the the kinds of experiments we've done I think are really nicely tuned to show some of the effects of these attentional networks. I don't know anybody that been doing exactly what we've been doing an in a developmental kind of study, but it would be fascinating. I don't follow that literature though, and more power to you for worrying about that somebody needs to do it and it's not going to be me.

Audience Question:

I'm Erica Lescht I'm an SLP student at University of Maryland. So this is a little bit outside my scope. I was wondering since these are adults who have already processed language what would happen if there were children born with neuropathy such as on specific language impairment. How do you think those types of children as they grow up would do on these tasks?

Yeah again it's a really great question. I mean I don't even know how to speculate about that. There are all sorts of things that that could be happening and yeah, I don't even know. I really don't even know. I don't study development, I don't know enough about development. It's a great question about development, and again somebody other than me is going to have to address it.

Audience Question:

Boji from UT Austin. I'm a very new person to hearing. I see that there are many special analogies when we try to understand listening. So to me that is fascinating that's very interesting by the same time for the domain there's obviously a lot of difference from the auditory domains like transitory signals for noise or how the stimuli or the signals interact with the background. They differ from social to auditory. So to what extent doing different vision tasks at the same time when you're listening, to what extent that could tell us more about hearing. Let's say ask a person to do a listening task at the same time ask him to do visual segregation or doing mental rotation. To what extent that could help us to know more about hearing. Because to me that would help us to know if there can be at our interventions to heal not just focus on not hearing. The second question is -- you have shown correlation between extent of attention and listening abilities in a way, but at the same time if I push the extent of attention to an extreme like someone can be very focused in attention at the same time they may not be flexible in picking out different information. So let's say when I'm doing that number task, putting down the phone numbers, but the same time someone else they say here's five so you have to go -- So maybe someone who is not having a very good focus attention they are more flexible in a way to pick up other information? So maybe would it be actually be some blessing for someone or someone who has less attention for that domain may actually do better in listening for someone else?

Those are both great questions. The first question was -- I'll paraphrase -- why don't you just do some really beautifully designed experiments where you're dividing attention and look at the cost of dividing attention and divided tasks kind of tasks, and pit vision versus audition, and you can vary the visual task to interfere with spatial or content processing. Great idea in fact I agree so much that I wrote it into a proposal that I submitted two days ago. Great idea, love that. So literally, that' exactly along the lines that we've been thinking and again you can hear it today, there's so many different directions one can follow up, choosing what the best one is and that's the most productive for me, that's why I don't want to go after the developmental piece. For me to gear up to developmental studies didn't make sense. But to do something like you're suggesting, that's exactly where we're thinking.
The second question, oh gosh, remind the second question was -- oh no I got it. I remember, because it's also something we're looking at actually. I have a student Jasmine Plaza. The question is: so focused attention you've shown how people are better or worse at focused attention. But focused attention isn't necessarily a measure of everything that matters. And I am especially inspired to believe that the same philosophy you were kind of saying, because I have a son who's a little bit scattered and a little bit ADHD and he's really good at something that don't require focused attention because he's processing everything in a more holistic way. And that got us thinking and Jasmine my student is very interested in that kind of question. And so we're about to undertake a study where we're looking at a test -- bear with me it's kind of cute, but you'll see why it might be relevant here. People start listening to a sound from in front, And their job is either just listen to that sound in front regardless of what happens. That's the fixed condition, and we contrast that with listen to the sound that's in front. Sometimes and only sometimes there's going to be another sound that starts from the left, not from the right but from the left. There's always something from the right. But if the sound from left comes on, switch attention to it report it instead. And we're doing it with ADHD patients. We've done this task and have seen some really interesting patterns of activity that come up that seem pretty robust. So focusing attention versus switching attention is it actually engages different parts of the brain, and our hypothesis is that people who truly have ADHD may differ in how well they do on the fixed task, how much they're hurt by having to be able to switch. One of the things that's kind of fun is on that fixed task, sometimes that other source comes on, sometimes it doesn't. On the on the cases where you're ready to switch attention, there are cases where you stay fixed in front because this thing never happened. But even though you're doing exactly the same task of reporting within front and nothing else happened that you had to pay attention to -- you take a hit for having to be ready to switch attention. But how big that hit is varies across people. And the idea is some people may be kind of listening, not really focusing well to begin with so it doesn't really matter if they're gonna have to switch attention because they're already kind of listening in that way. And we're going to be doing this one when these ADHD patients are on their ADHD drugs and compared to when they're not. So exactly. We think that this might be a way to tease apart some of these -- what looks like a deficit is only a deficit in the kind of weird situations we put you in and maybe it's an asset in another situations.

Audience Question:

My name is Molly Brown I'm from the University of Pittsburgh I'm an AuD student. I went to the talk on Thursday from NCRAR are where they were reporting that the mild TBI patients were -- they purported a lot of things but particularly that the prevalence of subjective tinnitus was higher in those TBI patients. Is there anyone that's looking into or suggesting that this tinnitus could be a result of this lack of modulation that you found?

That's a great question. Not that I'm aware of directly. And again to this is an area I've only started to get interested in as I've seen how it intersects with stuff I've been doing. So I'm not an expert. I will say they're a bunch of people at Mass Eye and Ear Infirmary -- Dan Polley he's a person I mentioned he does animal work. Sharon Kujawa and Charlie Liverman have been doing a lot of the studies in animals and humans of cochlear neuropathy -- are also very interested in tinnitus. There's lots of their suggestions that cochlear neuropathy -- a loss of the peripheral input -- can lead to tinnitus. And the idea is you're turning up this gain to try and compensate for a weak input and when you turn it up too much what you're really amplifying is nothing, and you're amplifying noise and you get tinnitus. That's kind of the cartoon idea of what might be going on. Jennifer Melcher who is at the Mass Eye and Ear Infirmary has for years talked to me at conferences about how attention could lead to -- a failure of being able to suppress tennis -- could be related to who suffers from tinnitus and who doesn't. So those ideas are out there I haven't really studied them. But I think they're probably all interrelated. The tinnitus may be the result of this adaptation and plasticity that's trying to pull up, you know amplify the signal that's no longer there. And you end up with tinnitus. When you have that kind of over amplification higher stages made still be able to compensate and turn that off, and so both maybe going on and that may be why it's so hard to tease apart why we get tinnitus and what it represents. So again the problem with tinnitus it's sort of like neuropathy you can try and come up with animal models but you're never sure if what you're spending in the animal model is exactly what the humans are experiencing which makes it a challenge.

Audience Question:

My name's Sarah Kennett and I'm a PhD candidate at University of Arkansas at Little Rock and for the RMPTAs in the room, you shared an article with us about object-based auditory and visual attention, and in the article you extended a framework from vision to audition and attention, and then you also mentioned today vision may factor advances in hearing. I have two questions. First is that I would like to know if you have a reason why you think that that happened, and secondly are there things that we can even still now learn from other sensory research out there that we can apply in audition?

That's a fun question. So why did vision take over the field of attention and advance farther than us? Sheer numbers. Auditory neuroscience is an order of magnitude fewer researchers than vision neuroscience. Like vision is just dominant. If you go to the society for neuroscience, vision is the single most represented sense and probably if you sum every other sense audition would be the huge chunk of what remains, but the people who do rodent models and olfaction and put us all together we're still probably less than half of all vision. I mean it's really quite warped. The good news is, or the bad news for vision is, the really mean and nasty to each other and competitive as a result. I think the auditory field is incredibly warm and nice and perhaps because there's less competition. So really I think that's why. There's actually a secondary reason why did vision take over in physiology and that has to do with just techniques and where things are in the brain. If you're looking at where the auditory cortices are, they're buried in the brain. Visual cortex is just right there, it's easy to get to and you don't, it doesn't traumatize the animal. So as techniques have gotten better, we've gotten better and better at doing good physiology in the auditory sense, but vision was way ahead because it was easier logistically to get to, and it took over and it's dominant. As to whether there's more we can learn from other senses, probably. I think the fun thing for me has always been seeing, to take these paradigms and ideas from one sense to the other, because you are encoding different kinds of information you can't just do the same thing, it doesn't even make sense. So my vision scientist friends often have: There are 17 objects and they're all identical but moving and you have to track one. And you can't do something like that the auditory domain because 17 sounds that are identical is noise that adds to other noise and you get noise, right. It doesn't make sense. So to figure out what's the underlying idea that might be common and how could you act on that idea through auditory stimuli that we can process in sensible ways is where the fun comes in and kind of the design and creativity comes in, and that's what I've really had fun doing. So I'm sure there's more one could do there and you should come do that, it's fun.

Audience Question:

Thank you. I'm thinking of the case of asymmetric cochlear synopathy and how if you were to try to look at the brainstem response and related to behavioral task, if you could possibly underestimate dichotic or binaural tasks.

That's a great question. So you know I started out my life as a binaural hearing person and spatial hearing was my bread and butter. And when we started doing these measures of the brainstem, we have always done them with bilateral stimulation, like identical diotic stimulation. And one day in the lab I kind of looked at everybody in the lab, and I said, we're not we're not even sure if this is symmetric. We've been doing it, stimulating both ears simultaneously with identical signals. And everybody kind of nodded and we were like, why did we do that? Like wouldn't it make sense, and we've never done it. Like we haven't even started. So again yes, it's a great question, what happens it's asymmetrical? What could you learn from that? You know absolutely, great question. Haven't done it all, and it's kind of ironic that we hadn't even thought about it until after we were two years or three years into the study. So great question, don't know the answer.

Audience Question:

My name's Jean Shin, I'm and from Northwestern University. And thank you so much for a great talk I really enjoyed it. I guess I have a question regarding the zigzag task. So in that task you use tone sequences, so that's a great starting point to look at stream segregation and the results really cool. I'm I guess I'm going a step further into speech on speech difficulty. And because I think speech has multiple cues, things like pitch, timbre, duration, those kind of cues sort of all play out in this kind of how people can use these cues and pay attention to one stream and try to filter out the other. Do you have any comments on maybe the attention that -- like logical basis for using these cues and how they play out in this kind of complex scenario.

Again great question. One of the things that we do in the lab to be able to isolate these effects is to make things hard in ways that are easy to manipulate, and parametrically manipulable, which is how you end up with these kinds of tasks. But you know we really are interested in speech, and like you said speeches multi-dimensional. There's multiple features that allow me to segregate my voice from a male voice for instance. And even things like content. The fact that I played nonsense sentences and that was confusing more than if those sentences had been meaningful, that's actually true to. Like even the meaning of the words can help you string the right words together through time. It's not even a sensory representation it is a much more abstract representation that can help you stream. So absolutely that's the right way to think about it. Absolutely we make these very simplified stimuli removing features to make it hard enough. The way I think about that though is very much inspired by some of the work that [inaudible] have done thinking about when you're streaming things there's some internal representation which is highly multi-dimensional. Highly dimensional space. It's got pitch, it's got timbre, got all these features things have to be separable in that space, and when they're not that's when you start to fail. Now I can create simple ones that are very clean but still hard to separate by making them vary only at one dimension or by small amounts. I think hearing loss actually even in that multi-dimensional space, everything gets blurred out, and so even though they're far apart in there means they're so blurry that they blur together despite the fact that there are multi variables and high dimensional and should be segregable. And I think reverberant energy does a lot of that too. I think that's why when we blurred the speech by adding reverberation it looks a lot like the kinds of problems people report that it blurs out these features, things become less distinct in that multi dimensional representation, whatever it is. So absolutely speech is hugely, hugely rich. And that's why we're so good at it. And you can see the effects by making it less rich, by taking away some of those features, so that you can show the mechanisms. And that's the game you play as an experimentalist.

Audience Question:

Tina Grieco-Calub from Northwestern. Great talk. So I'm, full disclosure, the motivation for my question is from a development perspective, but I'm going to try to ask you in a way that doesn't -- so you started the talk off by challenging us to go into the poster session and plug on ear. So this idea of being able to use spatial hearing to help us segregate. And so when did the spatial hearing system is impaired whether through development or even following development, but because of [inaudible] problem do the attentional networks take a hit? Or can people compensate like, is spatial hearing sort of a fundamental requirement?

It's not a fundamental requirement. i mean a lot of the games I played with you guys were over speakers where the two talkers are mixed. And that comes back at times very beautifully with the question we just had, that space is just one of the features that helps you. It turns out that the harder things get, the more any-- let me back off. Space is just one of the features that can help you. In a normal speech there's so many of them I can take almost all of them away, and you still can do the task. And so space is important especially when it's really, really hard because any one feature is just giving you just a little increment because they're all muddy and bad and you know there's some threshold at which I can segregate and space might in a muddy setting gives me this, and pitch gives me this, and talk her identity gives me that, and maybe I can just get enough that I can do the task. As opposed to, in quiet, in the lab, over headphones if I have a male talker and a female talker, it's trivial for most people with nothing else going on. Nonsense sentences spoken together by a male and a female talker are perfectly segregable. And the only way you kind of mess up is if you start listening to the wrong person because you let your attention glide to the other person. So I don't know how it develops and that's a great question, but one of the things about it is space is important in the really hard settings because it's really hard and any one thing is important in that setting. And the other thing that -- and I'll just leave this is the teaser because we really should let you guys have a break -- but it's pretty clear from -- I started as a spatial hearing person. I love spatial hearing, it's my it's truly my deepest early memories as a scientists are spatial hearing -- space is actually weak in the auditory domain. it's really easy to direct people to attend to direction., and it's something that we think is really important. But all the other features, in terms of gluing things together through time, are stronger. Space is the baby brother, but what's interesting is because it's probably is because it's co-opting this other network, but its not the natural way of hearing. It's not encoded at the periphery you're computing it. And I can talk to you offline about that. But it really is the case that even though space is important, it's weak.