On Psychometrics and Individual Differences
Psychometrics and individual differences are the parts of psychology that are devoted to measuring people. This measurement is most commonly concerned with gauging either intelligence or personality through some form of test, but it can also be applied to a host of other attributes (e.g., social skills, emotional intelligence, clinical practice, etc.). However, this measurement should not be carried out recklessly.
Before a test can be used, there are three conditions that must be satisfied:
Be certain that what you want to measure is actually measurable
Be certain that you can measure it accurately and consistently
Be certain that measuring it is useful
If you cannot meet all three of these conditions, then you may end up doing more harm than good when using a particular test.
These three conditions are related to three very important concepts in psychology:
Measurability
Reliability
Validity
The order in which they are given here is not a coincidence. For a test to be valid, it must first be reliable. And for a test to be reliable, what it is testing must first be measurable. Of course, at the moment, I’m writing mostly in jargon. Let’s simplify things.
Measurability
It is not uncommon for people to assume that everything can be measured. Certainly, that assumption fits well with our everyday experiences. If we look around us, we see many attributes like height or weight that are obviously measurable. It is also clear from our everyday experience that people vary in terms of their intelligence and personality in much the same way that they vary in terms of their height and weight. But does that mean we can measure intelligence and personality like we measure height and weight?
Not necessarily.
Consider a simple statement: John weighs twice as much as Jill. The meaning of this statement is clear. If John weighs 100 kilograms, then we know that Jill must weight 50 kilograms. Likewise, if we have John and Jill sit on opposite sides of a seesaw, we know that Jill will have to sit twice as far from the centre as John to balance the seesaw.
But what about intelligence? If John has an IQ of 100 does that mean he is twice as smart as someone with an IQ of 50? Well, not exactly. An IQ of 100 is considered average but someone with an IQ of 50 would be considered severely intellectual impaired. Certainly, many courts would be reluctant to hold a person with an IQ of 50 responsible for their own actions since such a person would be in the bottom 0.15% or so of the population.
Clearly then, two people with an IQ of 50 are not going to be able to do the same amount of intellectual work as one person with an IQ of 100. Contrast this with the weight example. Two Jills (each 50 kilograms) really do equal Jim’s weight (100 kg). Weight is additive but intelligence may not be, at least, not in the same linear fashion.
We can make a similar case for personality. If someone receives a score of 100 on a test of extroversion, are they really twice as extroverted as someone who receives a score of 50? Even a cursory examination of most modern tests of extroversion will show that such a belief is simply not tenable. It doesn’t make sense.
So what does this mean? Tests are instruments that assign numbers to people. However, that does not mean we can treat these numbers like we treat other numbers. Weight assigns a number. Weight is also quantitative (a very special property), which means we can make a range of very important statements, like :
5 kg + 10 kg = 15 kg
10 kg divided by 2 = 5 kg
10 kg multiplied by 2 = 20 kg
0 kg means that something has no (zero) weight
Those statements might seem obvious, but let’s try to apply them to IQ:
50 IQ + 100 IQ = 150 IQ
100 IQ divided by 2 = 50 IQ
100 IQ multiplied by 2 = 200 IQ
0 IQ means someone has no (zero) intelligence
Do these statements make sense?
If you take someone who is severely intellectually impaired and have them work with an average person, is their intellectual output similar to a genius? No.
If you take two severely intellectual impaired people and have them work together, is their intellectual output similar to a normal person? No.
If you take two average people and have them work together, is their intellectual output comparable to a genius amongst geniuses? No.
If you take someone with a score of zero, do they have absolutely no intelligence? No. Well, in fairness, they’re likely not very bright, but they would still possess some level of intelligence. Just being able to attempt the questions (even if they get them wrong) is a sign of intelligence.
Clearly, then, weight and intelligence are not the same, or at least, scores on them do not have the same meaning. Weight is clearly measurable. Intelligence may well be measurable too, but not to the same extent. This is where scales of measurement become important:
Nominal
Ordinal
Interval
Rational
When we measure things in a nominal way, all we can do is tell them apart. Think of names. We know that Tom, Sam, and Bill are different people. How are they different? We don’t know. We just know that they are different.
An ordinal scale can rank things. So, for example, if Tom came first in a race, Sam came second, and Bill came third, then we can have an ordinal measure of their speed. Not only do we know they have different speeds from each other, but we also know how they rank. What we do not know is how big the differences are (e.g., Tom might have only just beaten Sam while both of them were way ahead of Bill). If you are measuring things in an ordinal way, all you can do is rank them. You cannot comment on the size of the differences between them.
An interval scale is more advanced. It can accurately reflect the differences between things. The best example is the Celsius scale for temperature. The difference in temperature between 5 degrees celsius and 10 degrees celsius is the same as the difference between 10 degrees celsius and 15 degrees celsius. What is missing, however, is a meaningful zero. 0 degrees celsius does not reflect an absence of temperature. It is simply the freezing point of water. Without a meaningful zero, we cannot multiply and divide meaningfully. For instance, 10 degrees celsius is not twice as hot as 5 degrees celsius.
A ratio scale has all the properties of an interval scale but with a meaningful zero. The Kelvin temperature scale is a ratio scale since its zero point does represent an absence of heat. Weight and length are also ratio scales since 0 kg means an absence of mass and 0 metres means an absence of length.
It should be clear by now that when we say that we are measuring something, we like to imagine that we are measuring it on a ratio scale. That lets us do all of the fancy maths and statistics that we like, and even more importantly, it means that all of those maths and statistics actually mean something. But if we don’t have a ratio scale, what then? Well, we can still do the math and statistics… they just won’t mean anything.
And this is where things can get ugly. On what scale do you think we can measure intelligence, personality, and other psychological attributes? I think we can be confident in measuring them to at least an ordinal scale, but are we measuring them on an interval or ratio scale?
More worrying is another question: is it even possible to measure intelligence, personality, and other psychological attributes on an interval or ratio scale? If it isn’t, then tests of intelligence and personality are built upon foundations of sand.
Reliability
The question of measurability is highly theoretical and conceptual in nature. In contrast, the question of reliability is much more straightforward. For a test to be reliable, it must measure something consistently and accurately.
Consistently and accurately means that if the attribute being measured has not changed, then a test should give the same result when used on multiple occasions. Consider weight. If my scale is reliable, then if I weigh myself ten times in a row, I should get the same result each time. Likewise, if someone’s personality or intelligence has not changed, then a test of those attributes should give the same result each time.
Unfortunately, this is easier said than done since psychological tests are generally vulnerable to practice and demand effects. If I gave you the same intelligence test five times in a row, one after the other, you would undoubtedly get better at it because you have seen the questions before. Similarly, if you are applying for a job, and I give you the same personality test five times in a row, you might start to figure out what sort of personality is desirable for the job and change your answers accordingly.
This is why measures of test-retest reliability are seldom used in personality and intelligence. People learn when they are asked to do a test multiple times, and this in turn changes their scores. So how can this problem be solved? The simplest method is to design alternate versions of the test that are matched in difficulty and overall content (e.g., you might have two versions of a maths test that have the same problems, just with different numbers involved). You could then look at how closely scores on the two version corresponded. This is called alternate forms reliability.
Finally, there is something called internal consistency. If your test has multiple questions that are each attempting to measure the same thing (e.g., multiple questions about someone’s extroversion or multiple questions about problem solving), then someone’s performance across all of these questions should be fairly similar.
The reliability of a test can thus be measured in several ways:
Test-retest reliability
Alternate forms reliability
Internal consistency
A test that does well in these categories is consistently and accurately measuring something. Now, that doesn’t mean it’s measuring the right thing (that’s what we’re about to get to), but it is measuring something.
Validity
In simple terms, validity is all about whether a test measures what it is supposed to. More practically, a valid test is also a useful test since it can either predict some important outcome or corresponds to something important. In other words, if you want to know if a test is valid, the question you should be asking is: does this test measure what it is supposed to measure.
The simplest way to check validity is also the one that is most prized in human resources and organisational psychology: predictive validity. A test has predictive validity if scores on it can predict some important outcome (e.g., an intelligence test predicting job performance or a personality test predicting managerial ability).
If a test of something is good, then the scores it gives an individual should be similar to the scores given by other good tests of that same thing. For example, if you design a new intelligence test, then the scores it gives people should correspond to the scores given by pre-existing good tests of intelligence. Another way of thinking about it is to imagine that you’ve made a new ruler to measure length. If you already have a good ruler, then the two rulers should produce similar results when you measure the lengths of different objects. If they don’t, then one of them is wrong. This form of validity is called concurrent validity.
Content validity is about whether or not a test samples adequately and representatively from the domain of interest. For instance, if you are constructing a test of mathematical ability, including only questions about addition while neglecting subtraction and other operations would not be a good idea. Likewise, if you were designing a test to measure reasoning ability, having only problems of one type is unlikely to give you a valid measure of reasoning.
Perhaps the most complex form of validity is construct validity. In simple (but complex) terms, construct validity is whether or not your test measures what it claims to measure. The two mains ways of determining this are to see whether it relates to the things it is supposed to be related to (e.g., virtually all measures of intelligence are correlated with each other) and whether it does not relate to things it shouldn’t (e.g., measures of intelligence generally do not show strong correlations with measures of personality).
Conclusions
Before we use a psychological test (or any kind of test), we need to be certain that we are doing the right thing. This is particularly important when we move away from research into areas where the test serves as the basis for an important decision, such as in human resources or organisational psychology. But how do we know we’re doing the right thing?
A good test must be able to satisfy three criteria:
Measurability
Reliability
Validity
These criteria are not independent. A reliable test is one that measures something consistently and accurately. A valid test is one that measures what it is supposed to measure. Therefore a test must be reliable before it can be valid. Yet the issue of measurability is even more important. An attribute (e.g., intelligence) must be measurable before we can measure it with a test. Moreover, it must be measurable in certain ways, such as being quantifiable and measurable on a rational scale, if we are to apply many of the advanced statistical techniques that are so popular. It is not uncommon for people to simply assume that something is measurable, yet measurability, much like reliability and validity, is something that must be demonstrated, not simply assumed.
Tests and assessment instruments can have a powerful impact on people. It is the responsibility of the people using those tests and assessment instruments to make sure that they are using the right tests in the right ways.
If you want to read more about my thoughts on writing, education, and other subjects, you can find those here.
I also write original fiction, which you can find here.
