An interesting and brief video from John Piper on that question. His answer is to ask two more questions in response. First, does the person actually know any Christians? I like the disdainful way he separates Christians from "churchgoers". The second is to ask what it means if we get more joy from people who in turn get their joy from something other than what Christians ought to get their joy from, i.e. God. You have to watch the last bit more than once but it is an interesting question.
I think there might be a lot of reasons why many Christians prefer to hang around with non-Christians. First, a lot of Christians are just not pleasant to be around. I am probably Exhibit A for this as I am generally more argumentative than I need to be and am one of those people that in a conversation isn't listening so much as I am waiting for the other person to stop talking so I can. Another more troubling reason is something my wife said after we watched the video. She said that she feels more relaxed around non-Christians because it often seems like other Christians are constantly judging their brothers and, just being honest, their sisters. Is your house neat enough, are your kids well behaved enough? The subculture around Christianity in America is often more of a suburban, upper middle-class country club atmosphere than one of genuine care and acceptance of one another. That is not true everywhere or with everyone but it happens often enough that a lot of Christians feel very on the defensive when around other "Christians". I generally don't mind because I have enough religious trivia and can whip up decent verses to deal with most issues so I can impress people. For my wife and a lot of other women in the church this seems to be less common. Don't get me wrong, people love my wife. When someone I know meets her for the first time they always say later how nice she is although it is rather irritating that they sound so surprised that a kind and decent woman would marry me and bear me 8 children. There is also the very real way that we have cordoned off "church" into a once or twice weekly set aside time to get our fix and check the "went to church" box on our religious checklist. I spend time with Christians on Sunday and Wednesday night because I am supposed to, the rest of the week I spent time with people I actually like being around. This is not true for everyone but it is true for a lot of people.
We better start to check ourselves and ask why what Piper is talking about is so common. I think it bodes ill for the church if we can't start to take more pleasure in other people who share with us a common love and joy for our God.