Tag: personality inventory

Posted on

9 signs you might be using the wrong personality test

Stamp that reads Test Failed says you're using the wrong personality test

Personality testing is a big part of the way organizations make hiring decisions — it has been for a some time now (it wasn’t popular before about 1980). With advances in technology there has been a great proliferation of personality assessments. They’re not all good. A personality test is much easier to generate than it is to validate. This quiz, below, can help you to know if you’re using the wrong personality test. (Have some fun with it.)

Directions: The following list of paired statements(questions) reflects things I occasionally hear when folks are evaluating personality tests. For each pair, one response is more problematic when it comes to evaluating personality tests. Reflecting on your current situation, which of the two statements would I be most likely to hear from you or others if I were a fly on the wall when you were getting the pitch from your vendor?

Quiz to raise the question "Am I using the right personality test?"

Response Key: For all odd numbered pairs the problematic statement is in column A, for even numbered items the more problematic one is in column B.

Some of the statements do require more assumption than others, don’t get too caught up in the scoring. These are my answers and rationale:

  1. “It sure worked for me” — Frequently a personality test is sold by having the decision maker complete the assessment. This isn’t a bad thing — I encourage users to complete a personality test for themselves. The potential problem is that this is frequently the primary (or sole) evaluation criterion for a decision maker. Vendors know this and some hawk an instrument that produces unrealistically favorable results. “It says I’m good, therefore it must be right.” As for column B, the 300 page manual, good ones are typically lengthy. It takes some pulp to present all the evidence supporting a properly constructed inventory.
  2. “A type’s a type” – The most popular personality assessment of all, the MBTI, presents results for an individual as one of 16 types. Scores, to the extent that they are reported, only reflect the likelihood that the respondent is a given type or style – not that they are more or less extraverted, for example. But research and common sense say that personality traits do vary in degree, someone can be “really neurotic.” Two individuals with the same type can be quite different behaviorally based on how much of a trait they possess. A very extraverted person is different from someone who is only slightly extraverted — same type, different people. (No, I don’t condone mocking or calling out anyone’s score, as it would appear I’m suggesting in column A, but with a good test such a statement is potentially valid.)
  3. “That’s a clever twist” – Few personality tests are fully transparent to the respondent – this helps control the issue of social desirability. But some go too far with “tricky” scoring or scales. This is a problem in two ways: 1) if the trick gets out (google that) the assessment loses its value, and 2) respondents don’t like being tricked. It’s better to be fairly obvious with an item than to deal with {very} frustrated respondents who may just take you to court.
  4. “It was built using retina imaging” – Here’s another statement that needs a little help to see what’s going on (no pun intended). I’m not against new technology, it’s driving ever better assessment. But sometimes the technology is misused or inadequately supported with research. There’s a reason that some personality assessments have been around for more than 50 years. Validity isn’t always sexy.
  5. “That’s what I heard in a TED talk” — My intent here was to implicate “faddish” assessments. They may say they’re measuring the hot topic of the day, but more often than not, what’s hot in personality assessment, at least as far as traits are concerned, is not new. Research has concluded that many traits are not meaningfully different from ones that have been around a while. Don’t fall for an assessment just because you like the vocabulary, check the manual to see if it’s legitimately derived. There’s a reason that scientists prefer instruments based on the Big 5 traits (not the big 50).
  6. “Now that’s what I call an algorithm” — More complicated isn’t necessarily better. Some very good — typically public domain — assessments can be scored by hand. Tests that use Item Response Theory (IRT) for scoring, do have more complicated algorithms than tests scored via Classical Test Theory (i.e., more like your 3rd grade teacher scored your spelling test). Still, a three parameter IRT scoring method isn’t necessarily better than a one parameter model and it isn’t three times more complicated anyway. Proprietary assessments typically protect their copyright with nontransparent scoring, but for the most part what’s obfuscated or obscure is what items go into a calculation, not that the calculation is necessarily complex. Good assessments should employ fairly straightforward scoring to render both raw scores and percentile, or normed scores.
  7. “It really has big correlations” — As with some prior items a bit more context is needed to get the point I’m trying to make. Here the issue is sufficiency. Yes, a good instrument will show some relatively high correlations, but they need to be the right correlations. (And they need to be truthful. Unfortunately, I know of cases where misleading statistics have been presented. It helps to know about research design and to have a realistic expectation for that validity correlation. If the vendor tells you that their assessment correlates with performance above .40, make them prove it. (And a .40 correlation equates to a 16% reduction in uncertainty, not a 40% reduction. Sometimes vendors get this confused.)
  8. “It’s too long, let’s cut some items” – It’s tempting to simply eliminate irrelevant scales or items for your specific need. After all, you’re not touching the items that comprise the traits you want to know. The problem is that the assessment is validated “as is.” Both the length of an assessment and its contents can influence scores. Priming biases are one example of how items interact with each other. Anytime you modify an assessment it needs to be validated. This is typically the case for short forms of assessments (i.e., they’ve been specifically validated), so it’s fair to ask about this alternate form.
  9. “That’s amazing” — By now you should see that a common factor in my problem statements has to do with how much goes on “out of view” (less is better) and how thorough the test manual is. “That’s amazing” is for magic shows, not science (I realize I’m parsing semantics here – you get my point).

A personality test can be — and most often, is — a legitimate assessment for many (most) jobs. (This even applies to machines. Researchers are using a variation of personality inventories to manipulate the perceived personality of robots.) Without exception, it’s critical to ensure that any assessment be validated for specific use, but you want to start with something that has been thoroughly researched. If everything has been done right, you can expect local results to be in line with the manual (assuming your tested population isn’t that different from the test manual sample(s)).

A lot goes into validating a personality test and test manuals are lengthy. Although this is good and necessary for adequately evaluating the test, it can be used in intimidating or misleading ways. It’s easy for claims to be made out of context even if the manual is true, especially when decisions are made that affect one’s job. It’s important to review that test manual, not just the marketing brochure. (The good news is these manuals are boringly redundant. For example, the same figure is used for each scale, or trait, when repeating testing for gender bias.) Although I’m sure your vendor is a “stand up” person, you can’t rely on this fact if your process gets challenged in court. It pays to review the manual thoroughly.

I hope your personality inventory passed the test.

Psychways is owned and produced by Talentlift, LLC.

Posted on

Why Personality Inventories Don’t Tell the Whole Story

Self Check - Personality Inventories

The vast majority of personality inventories rely on “self report” for their input. Quite simply, individuals assess themselves on what I’ll call the “first level.” Since I refer to a “first level,” there obviously must be at least one more level. There is, and it’s a level of assessment that individuals can NOT provide by themselves no matter how good the inventory nor how “truthfully” the individuals respond to it. Therefore, personality assessments don’t tell the whole story.

You don't know yourself as well as you think you do. How can we assume that even the best personality inventory completed by oneself would know you any better?

This doesn’t mean personality assessments aren’t useful (or ‘valid’ in scientific terms), it simply means that there’s more to a person’s story than they can reveal via any series of questions in a personality inventory. This goes for ALL personality inventories, some, more than others, but none can overcome the limitations of self-assessment. In short, you don’t know yourself as well as you think you do. How can we assume that even the best personality inventory completed by oneself would know you any better? This is where an expert in psychology comes in handy. To get the best understanding of an individual, an expert in psychology and psychological assessment can help to ‘fill in the gaps’ that we ALL leave in our own account of our personality.

Although many psychologists would agree and offer varying degrees of scientific proof, Sigmund Freud developed a theory of personality that serves my point. Freud’s theory is grounded in the way he described the structure of the human psyche. This structure includes three components; the Id, Ego and Superego. Without going into the details of each of these components, Freud also developed the concepts of Consciousness and Unconsciousness (although he wasn’t the first to describe them). Almost everyone has some familiarity with these terms – even if not exactly in the way that Freud defined them.

Consciousness has to do with one’s direct awareness of their thoughts, feelings and behaviors. We can fairly accurately describe things that we experience while we are in a conscious state. Unconsciousness is the other ‘side’ of ourselves; the side to which we do not have direct access and therefore do not readily understand nor recognize. As such, we are unable to describe things that exist in our unconscious mind – even though it is constantly at work.

I could stop here and have a pretty good case for why self-report assessments don’t tell the whole story. They don’t include our unconscious self and our unconscious self has a big impact on who we are.

But there’s more.

Freud also described how the conscious and unconscious aspects of our personality work together. I’m not going to go into great detail here except to say that the unconscious mind significantly influences our thinking, feeling and behavior. And it's far more influential than most think.

Here’s a simple example of how unconscious behavior reveals itself in our daily lives: Tying your shoes. This is an activity that we perform virtually every day – but odds are you can’t tell me how you do it. We’ve done it so much that it’s become “automatic.” Basically, we do it without thinking. There are many other examples. Sticking with the shoe example, behaviors that we do repetitively oftentimes become “automatic.” Automatic behaviors require very little (if any) thought, and true to unconscious behavior, we have a hard time recalling or describing them. (A nice benefit to automatic behavior is that it uses almost no mental resources. This means that we have plenty of resources to attend to other matters – aka, multi-tasking.)

Automatic behavior is just one way in which unconsciousness affects who we are. Unconsciousness also affects our thinking and feeling. In short, we are very significantly influenced by psychological processes that we aren’t even aware of. Others may note these influences (or outcomes in our behavior) but we don’t. Things we say may be very apparent to others, but pass completely unnoticed by ourselves. For example, some individuals have a habit of repeating various phrases (usually “filler” words) without any awareness. You may know someone who repeatedly says, “at the end of the day”, or “you know what I mean?”, “um”, “actually”, or any of a cast of phrases that are “thrown in” to the conversation but add no value. Even if they are partially aware that they say these things, they have no idea how frequently they do it – unless you record them and show it back to them. In addition, people are very poor judges of how much they talk (vs. listen). You can test this with a friend, but I must warn that you this is almost never appreciated. Test at your own risk.

These are some simple ways in which our unconscious mind affects our behavior without our awareness. But that’s not all. There are even more “active” ways that our unconscious mind affects us that can be very confusing, or even misleading to an accurate assessment of ourselves (as actors) AND others (as observers).

Freud also developed the concept of “defense mechanisms.” In short, these are ways of thinking and behaving that counteract a thought or memory that is bothering us at an unconscious level. One such example is called, “reaction formation.”

Reaction formation is the term Freud used to describe the unconscious -- and extreme -- change of thought and behavior resulting from one’s unconscious need to (over)compensate for previous behavior that the individual now considers offensive. By way of “reaction formation” the individual unconsciously undergoes a radical transformation wherein the behavior or attitude they once held, suddenly becomes hyper offensive and disgustingly deplorable -- in others! Smoking is often given as an example. Former smokers sometimes become the loudest and most assertive critics of those who smoke. Freud’s theorizing is that by engaging in overcompensating behavior, one is clearing up or avoiding the unconscious tension they experience by virtue of having been a former transgressor.

Other forms of defense mechanisms include denial (unwilling or unable to accept the truth because of the psychological harm it causes), projection (attributing one’s own intolerable thoughts or problems to another so as to ‘shift blame’), repression (a less extreme variant of denial that involves pushing one’s hurtful thoughts or feelings into the unconscious self so as not to deal with them directly). And there are others.

Scores on personality assessments may be radically different from what an objective assessment would reveal.

The point is, not only are we largely unaware of our most frequent behaviors (automatic behavior), but our psyche is constantly at work trying to protect ourselves from threatening thoughts, feelings or behaviors (defense mechanisms). As a result, scores on personality assessments may be radically different from what an objective assessment would reveal. And this isn’t because the respondent is lying, they really believe that they are accurately describing themselves. There are many other factors that distort our valid understanding of ourselves, these are just two of the most common.

An expert in psychology and psychological assessment can identify these, and other unconscious influences on behavior, and consequently, scores on a self-report personality inventory. Sometimes this can be done merely by noting unusual or telling patterns in the individual’s responses to a reputable personality assessment, but frequently it requires the collection of data beyond the single assessment. Psychological interviews are among the best ways to spot potentially misleading information as taken straight from the personality inventory. The content of these interviews can be designed specifically to test questions raised by the instrument.

It’s very important to stress that these types of advanced interpretation of any psychometric assessment are complex. They need to be left to experts who have a thorough understanding of psychology as well as tests and measures used as tools to predict behavior.

In sum: Solid psychological assessments offer great value over less scientifically constructed measures (e.g., typical unstructured interviews). But, as with any other tool, it’s important to know the true strengths and limits of what they offer in the complex task of psychological assessment. As anyone who’s made a regrettable hire can agree, what you see in the interview isn’t always what you get on the job.

Psychology at work: It really makes a difference.

Psychways is owned and produced by Talentlift, LLC.