although it would seem that variability within a (separated) family is about as high as variability between strangers.
Yep.

Unlike some other polygenic traits (i.e. height, intelligence, etc), personality traits appear to primarily be the product of non-additive genetic influences (thus, they don't 'breed true'- only monozygotic twins can be expected to show significant similarity). Of course, non-shared environmental effects are important contributors as well, although they too accentuate within-family differences.