Imagine living in a house full of passive-aggressive teenagers as your roommates. They are both ignorant and condescending. Every time you turn around, they’re right there, staring at you and whispering to each other. They make the odd snide comment here and there. You try everything just to get along. You try being polite. You try ignoring them. You curse them out.

“What’s that supposed to mean?” you ask repeatedly.

“Nothing,” they say.

But you know it’s not “nothing.”

Then it gets super weird because they ask to touch your hair. You’re about to lose your mind, but what can you do? These are your roommates — and your lease isn’t up for the rest of your life.

That is usually what it’s like to deal with racism as a person of color in America. Most of the time, it’s not very dramatic. We’re not usually running from skinheads or the Klan or cops, even though we still have plenty of reasons to fear those groups.

The racism we deal with comes in the form of vocal tones, comments and judgments that amount to: “Well, you know how they are.”

Day to day it usually doesn’t go much further than that, because racists are still people, and people mostly behave themselves, at least to a certain degree. Most racists aren’t going to call me a nigger to my face. They’re going to be more subtle than that — though not by much.

[MORE: A New approach to combating racism in Maine]

But the long-term effects of subtle racism on black people have arguably been more harmful than the other types of direct, or even physical, racism. The Klan and other white power groups have unsuccessfully tried to kill us off since abolition. But the subtler racism permeates our institutions, some of which are allowed to torment and kill us.

In America, the story goes, you can be successful if you’re smart and work hard. This has generally been true for white men. It has generally not been true for minorities.

That’s not only because of personal prejudice involved in hiring, but also because of institutional bigotry. The U.S. government has created program after program (such as the Social Security Act and the housing components of the GI Bill) that excluded minorities and helped white men get education and jobs, which lead to property ownership. That property then gets passed down generation to generation, eventually helping to create a racial wealth gap, which is further widened by acts like redlining (declining services and home loans to people who live in black neighborhoods) and voter suppression.

Our inheritance is passive-aggression, insults and disproportionate unemployment, poverty and incarceration rates. And since it’s mostly the poor who are policed, we are also left with a difficult and often deadly relationship with law enforcement. That’s not even getting into the personal prejudices of some law enforcement officials.

And yet, some still regard destitute black men with, “well, you know how they are.”

This attitude can be even more dangerous when it’s coming from our leaders.

When integration began in the 1960s, a lot of white southerners viewed equality as their loss. The white voters in Alabama in particular had a very difficult time with this. So much so that, in 1963, former judge and newly elected governor George Wallace famously said in his inaugural speech:

It is very appropriate that from this cradle of the Confederacy, this very heart of the great Anglo-Saxon Southland, that today we sound the drum for freedom as have our generations of forebears before us time and again down through history. Let us rise to the call for freedom-loving blood that is in us and send our answer to the tyranny that clanks its chains upon the South. In the name of the greatest people that have ever trod this earth, I draw the line in the dust and toss the gauntlet before the feet of tyranny, and I say segregation now, segregation tomorrow, segregation forever.”

That’s not much more subtle than calling someone a nigger. But it was subtle enough so that later in life, when asked about that speech, Wallace would say, “I don’t hate blacks. The day I said ‘segregation forever,’ I never said a thing that would upset a black person unless it was segregation. I never made fun of ‘em about inequality and all that kind of stuff. But my vehemence was against the federal government folks. I didn’t make people get mad against black people. I made ‘em get mad against the courts.”

Wallace brings up race out of absolutely nowhere, but maintains his subtlety by only mentioning his own color. He then invents an adversarial situation in which he is part of a victimized group (white southerners, as victims of the north). Then he presents himself as a defender of his group when he is actually only persecuting another group (black people) and, by extension, his own. (If a black child from Alabama has, for example, the potential to cure cancer, but is denied a proper education, then everyone loses.) Finally, he later denied his obvious implications and intentions.

Recently, right here in Maine, Gov. Paul LePage displayed his subtler racism as well:

“The traffickers — these aren’t people who take drugs. These are guys by the name D-Money, Smoothie, Shifty. These type of guys that come from Connecticut and New York. They come up here, they sell their heroin, then they go back home.

“Incidentally, half the time they impregnate a young, white girl before they leave. Which is the real sad thing, because then we have another issue that we have to deal with down the road.”

His spokesman later claimed that LePage wasn’t “making comments about race.”

Much like Wallace, LePage brings up race out of absolutely nowhere but maintains his subtlety by only mentioning his own color. He then invents an adversarial situation in which he is part of a victimized group (Mainers, victims of out-of-state villains, as opposed to actual problems of addiction, poverty, etc.). He then shows himself as a defender of his group when he is actually only persecuting another group (again, black people) and, by extension, his own. (Let me make the same point as before: By shirking our state’s responsibility for itself, he only widens the distance between addiction, poverty, etc., and a solution).

[MORE: LePage’s ‘white girls’ apology was definitely not an apology]

Subtle racism furthers race-based inequality. It is a big problem, but it is a solvable one.

In the end, we obviously need to condemn racism. This is easy when it’s blatant. It’s easy to condemn the Klan. But that can be much more difficult when racism is subtle. In the cases of elected officials and public policy, we can fight with our vote. It doesn’t work every time, but it’s something most of us can do.

Condemning the racism of our weird uncles, coworkers and roommates to their faces can be tricky. Some people avoid confrontation, but we can ask them, “What’s that supposed to mean?”

And if they answer, “Nothing,” we can tell them, “I know it’s not nothing.”

Samuel James is a musician and writer living in Portland.