Separating the foolproof from the flimsy: what makes a good personality assessment tool?
by Dr Curt Friedel
A recent report shows that there are over 2,500 different personality and behavioral instruments in the USA alone. With figures like this, it has become increasingly difficult to separate the foolproof from the flimsy.
So, how do you know if a particular psychometric inventory has scientific merit, or is just based on claims of organizations trying to make a quick profit?
In this article, I have outlined the key criteria a reputable psychometric inventory should provide evidence to support its use.
Theoretical Support
Many consultants and practitioners scoff at theory as useless, because it is associated with the ramblings of academics in ivory towers who have no “real-world” experience. However, a good theory offers practical explanation towards how findings may be applied, as well as predict future outcomes.
The large majority of psychometric inventories maintain theoretical support underpinning the premise that better understanding of myself may make me a better person or a better leader. Many of these theories also align with the notion that if I understand someone else better, I can appreciate them more. Then there are some theories and conceptual models which offer more sophistication through calculations to explain relationships between individuals, or predict how people may work together. Using a psychometric inventory beyond the guidelines of its corresponding theory is performing malpractice as a consultant or practitioner.
- Explanatory and Predictive Power: A sound theory allows one to make concise explanations and/or predictions regarding a situation, and determine if the explanation and/or prediction is correct through empirical observation. In short, the inventory aligns with the foundational aspects of the theory.
- Falsifiable: A theory does not explain all things in all situations. Data may be collected to refute the theory, in which case the theory does not apply. Theories that cannot be refuted are laws, such as the law of gravity.
- Parsimonious: Theory should provide a clear, simple, and logical way of explaining findings.
- Internally Consistent: Principles associated with the theory do not contradict each other.
- Heuristic Provocativeness: A sound theory makes room for new hypothesis to be developed and provides and provides a framework for organizing knowledge gained.
Theory criteria may seem too academic for many, but one should not place trust in using a psychometric inventory if there is not a sound theory to underpin its application.
Indicators of Reliability
One of the most important criteria of a sound psychometric inventory is that findings must be repeatable. This is achieved when the tool measures one variable accurately and consistently. If it is trying to measure too many variables at the same time, then the inventory is unlikely to be academically robust.
For example, a person completing a psychometric inventory should receive equivalent results after multiple completions. Furthermore, indicators should exist within the instrument to provide evidence of an individual’s results being unreliable. Often, an unreliable result is caused by the practitioner’s (or researcher’s) subjectivity. This is why many inventories involve users completing a data sheet or online questionnaire, rather than relying on the observation of another (observations are subjective, and therefore inconsistent).
The reliability of the inventory may also depend on the specificity of the questions. For example, an ambiguous item or question may result in users answering in different ways or giving a response that does not answer what the instrument was designed to measure. Once again, this would result in an inaccurate finding.
The best psychometric inventories not only make allowances for these inconsistencies, but preferably find ways to eradicate them. Having a quantitative inventory, rather than qualitative, helps to achieve this – and so too does having clear items and/or questions, which cannot be misinterpreted by the user.
Indicators of Validity
Psychometric inventories should measure what they are designed to measure. Published research should provide evidence that the inventory is correlated with other instruments believed to be theoretically similar, and not correlate with instruments believed to be theoretically different.
In this sense, psychometric assessments are typically the most reputable when they have been compared to other personality inventories.
Now, of course, a group of profit-based organizations could group together to develop evidence to support the validity of one another’s’ instruments. This would result in the instrument appearing to be valid, but in reality, it may not be true. This is why many practitioners and consultants look for assessments that have been approved by industry-leading socieities and organizations tasked with professionally reviewing psychometric inventories.
Finally, there are also statistical procedures used to provide additional evidence of validity of an inventory, such as factor analysis.
Ensuring that Items Discriminate
Item discrimination refers to the degree to which an item placed on the psychometric inventory differentiates correctly among completers in the behavior that the inventory is designed to measure.
The best personality assessment tools have items that discriminate. This also means that each item or question in the instrument contributes to the results, so that there is a normal distribution (bell curve) associated with the general population.
Results along a continuum in general should center at the midpoint. Items should correlate moderately with each other and within part scores of the inventory.
Interpretable Results
Many psychometric inventories provide results in the form of categories (or letters) instead of scores. This may present a challenge in scoring the inventory when an individual’s results indicate being close to the line and possibly being within one category or another. If this situation is further exaggerated with an unreliable inventory, the person may believe they are associated with one category, but actually should be in another. If the participant begins to believe that they are something that they are not, psychological harm may be a result.
Inventories should provide easily interpretable results to the participant with little error so that individuals receive meaningful and correct feedback.
A Usable Manual
The inventory should be supported by a manual providing the consultant or practitioner information regarding the supporting theory, evidence reliability and validity (through published studies), and how to provide proper feedback to the individual completing the psychometric inventory.
Training to certify an individual to administer and score a psychometric inventory should include teaching of these criteria. If the training does not include these items, the psychometric inventory is likely to be trendier and less reputable.
The KAI Inventory
The Kirton Adaption-Innovation Inventory is academically robust, and has decades’ worth of research supporting its measurement of problem-solving style in connection with Adaption-Innovation (A-I) theory. A-I theory has explanatory and predictive power, and is falsifiable, parsimonious and internally consistent.
We continue to offer strong evidence of validity and estimates of reliability through published research in different countries, social groups, and businesses for more than 45 years.
The KAI is the world-leading inventory for measuring problem-solving style. It is supported by A-I theory, known for connecting an aspect of one’s personality to one’s engagement in change.
Meet The Practitioner:
Dr Curt Friedel is Associate Professor and Director of the Centre for Co-operative Problem Solving at Virginia Tech in the USA, where he is also the Director of the KAI Certification Course. Click here to visit his LinkedIn profile and click here to see his profile on the Virginia Tech website.
