Sociolinguistic research shows that listeners' expectations of speakers influence their interpretation of the speech, yet this is often ignored in cognitive models of language comprehension. Here, we focus on the case of interactions between native and non-native speakers. Previous literature shows that listeners process the language of non-native speakers in less detail, because they expect them to have lower linguistic competence. We show that processing the language of non-native speakers increases lexical competition and access in general, not only of the non-native speaker's speech, and that this leads to poorer memory of one's own speech during the interaction. We further find that the degree to which people adjust their processing to non-native speakers is related to the degree to which they adjust their speech to them. We discuss implications for cognitive models of language processing and sociolinguistic research on attitudes.