i believe after the years, of playing c major, what happens is that some people's ears and their theory minds, become "atuned" to C, an arbitrary blip on the circle, that everything else is judged in relation to it.
in this way, it is very easy to see flat keys (such as both f (1 flat) and fminor (4 flats) ) as sadder, or more somber, as they are both moving downwards in key relative to cmajor (you can think of it as -1 for f major, and -4 for fminor). for the same reason, keys like e major tend to be bright (+4).
people such as beethoven (eb being his heroic key) and scriabin (key colour) tended to use keys in this way, probably for their own flow of ideas, as each key would be a touch stone, like a smell can be, while Bach, would openly transcribe pieces back and forth among key indistinguishably..
anycase, i really think it has something to do with the layout of the keyboard, and the fact that many musicians are made to learn c major for the early part of our lives.
*of the minor (sadness) and major (happiness), the answer is actually quite a bit simpler (and more solid), in that a major key has a harmony where the 3rd of the chord is more "in phase" (lower in harmonic series) than a minor 3rd which is used in a minor key. (** the harmonic series is like a ladder of consonance to dissonance). this means that the major 3rd clashes less against the other chord notes. and just like how families work, less clashing equals more happiness.
getting ahead of myself, but i hope that helps in some way to your question :)