On a faraway world, all the matter is formed of crystals that have a very curious property. There are only a few dozen types of material found on the surface, and each one is so structured that the shape that the crystals will take corresponds to the color. Put simply, anything you find that's red will always be spherical; anything that's blue will always be cubical; anything that's green will always be pyramid-shaped. And the other way around: all spherical objects are red, all cubical objects are blue, all pyramidal objects are green.
(Okay, so this is a bit ridiculous, but it's a though experiment to demonstrate a point, bear with me.)
The creatures who live there don't have separate words for "red" and "spherical" because, after all, they tell you the same exact thing. There aren't separate concepts for them. In their form of kindergarten, someone holds up a redsphere and says "redsphere" and everyone nods, and from then on, they know that that's what that means.
Color sight is of no evolutionary value to these people, but by the same token, it's no great detriment either. Unbenownst to them, a handful of them have the ability to see color, but they don't realize that they're seeing anything different from anyone else.
At least until one day they build their first spacecraft and a handful of them travel to a nearby world in their solar system. Here, there are a lot of materials with which these people are unfamiliar, and which have the weird property that sometimes a red object isn't spherical but pyramidal. But this was of no particular interest to most of the new arrivals. Only a few of them would stare enraptured at these pyramids and say, "it's redspherical, but it's also greenpyramidal!" and the others would stare at them like they were insane. All because they could see something that everyone else couldn't.
One could probably use this kind of argument to support the idea that mystics who are derided by skeptics are really the victims of a lack of understanding, not hoaxsters. But it's also a good way to make a very strong point. Finding a way to demonstrate that the color-sighted amongst these people are really seeing something is trivially easy. Put a few of the people in locked rooms isolated from one another. Bring objects to them one at a time and ask them to describe them. See how much correlation there is. If they're telling the truth, the correlation will immediately emerge and be immediately empirically verifiable. If they're not, it won't. Easy-peasy. Let's hope these crystalline folk have enough sense of scientific method to realize that, for the sake of those blessed with color-sight!