Understanding Self-Assessments: Part Two

Validity and Reliability of Self-Assessments

This is part two of Understanding Self-Assessments. Find Part One here.

JUST THE FACTS MA’AM

Given a choice, most organizations would likely prefer to obtain a D&I profile based on a self-assessment that was statistically validated and reliable, rather than a questionnaire that developed with a small sample size and a lack of rigor in the questioning. It’s important to understand how validity and reliability are determined.

Several factors contribute to the validity of any survey instrument. In some cases, the validity is a result of all three measures. In other instances, one of these measures will suffice.

• Time. How long has the survey been used? The National Institute of Health has been conducting surveys since the late 1950s. The Census Bureau has been surveying the U.S. population since 1902.
• Size. How large is the sample size? There are a few online calculators to determine sample size. As time progresses, the aggregate sample size can become quite large. The Implicit Association Test (IAT) (a reaction time test) measures the strength of automatic associations people have in their minds, thereby testing bias. Since it was launched in 1998, the test has been taken more than 20 million times.1
• Tested. Has the survey undergone statistical analysis and validity testing? Computer programs now do the work of spreadsheets, calculators and statisticians to provide statistical rigor for new surveys.

I have been struck again and again by how important measurement is to improving the human condition. ~  Bill Gates

SURVEY SAYS!

Validity testing: SPSS is a widely-used program for statistical analysis in social science used by market researchers, health researchers, survey companies, government, education researchers, marketing organizations and others. The original SPSS manual has been described as one of “sociology’s most influential books” for allowing ordinary researchers to do their own statistical analysis.2 The “S” language is often the vehicle of choice for research in statistical methodology, and “R” provides an Open Source route to participation in that activity. “R” was used to validate the Spectra Diversity and Inclusion Profile™.

Types of statistical analyses include:

• Mann-Whitney test
• Wilcoxon rank sum
• Kruskal-Wallis test
• Wilcoxon signed rank test
• Friedman test
• Spearman’s Correlation Coefficient
• Chi-squared test

Here’s the good news: for any company considering using a self-assessment to gather information about employees and their attitudes about the company, it should be reassuring to know that…

SELF-ASSESSMENTS PROVIDE RELIABLE AND ACTIONABLE DATA.

Recent studies, summarized below, support the concept that self-assessments are a valid and reliable means of measurement.3

• Several studies conducted on ESL students (column one in the chart below) rated their English-speaking competency using a self-assessment and the tutor or teacher scores found a correlation (positive connection) between the self-assessed scores and the teacher/tutor assessed competency score.
• A study conducted in 2010 of ESL students at the University of Michigan, showed that self-assessments (column two below) were proven to be a valid form of measurement, for any age and either gender.4
• However, among some population samples (column three below), self-assessment doesn’t work as well. The evidence suggests that physicians have a limited ability to accurately self-assess.5 The processes currently used to undertake professional development and evaluate competence among physicians may need to focus more on external assessment.

Strength lies in differences, not in similarities. ~ Steven Covey

If you measure, it will improve.

The validity of self-assessments does not negate the fact that something called the Dunning-Kruger Effect exists. Coined in 1999 by Cornell psychologists David Dunning and Justin Kruger, it is described as a cognitive bias whereby people who are incompetent at something are unable to recognize their own incompetence.

Where the Dunning-Kruger Effect may come into play for D&I self-assessments, is for those who are on the lower end of the D&I scale. “Poor performers often show little insight into the shortcomings in their performance, presumably because they suffer a double curse. Deficits in their knowledge prevent them from both producing correct responses and recognizing that the responses they produce are inferior to those produced by others.” 6

Like the Dunning-Kruger Effect (people who believe they are better than they are) is the Optimistic Bias. Optimistic bias is a psychological phenomenon in which people believe they are less likely to experience adverse events than other people. It is a sunny outlook when gazing into the future. It is unrealistic optimism. An optimistic bias is a predictive concept. We believe this is not a factor for D&I self-assessments which gaze into the present, or the past, and not into the future.

Some may use the words “self-rater bias” to refer to either the Dunning-Kruger Effect or the Optimistic Bias. In terms of a D&I self-assessment, we do not believe that either of these biases significantly impact an individual’s self-assessment of diversity and inclusion skills or beliefs.

On balance, research suggests that self-assessments contribute to higher achievement and improved behavior.7 The University of Toronto reported that, “The psychometric properties of self-assessment suggest that it is a reliable assessment technique producing consistent results across items, tasks and contexts and over short time periods.”

Students with greater confidence in their ability to accomplish a targeted task were more likely to visualize success than failure. This caused them to set higher standards of performance for themselves. Student expectations about future performance could thereby influence effort.

A study of college students showed similar results. The study found that self- assessment increased the interest and motivation level of students, leading to enhanced learning and better academic performance, as they developed critical skills to analyze their work.8

So, by the simple fact that “something” is measured, the “something” shows improvement.

In terms of diversity and inclusion, self-assessment measurement can raise consciousness about diversity and inclusion (and self) as an issue to be addressed.

TIP: If you’d like to make diversity and inclusion improvements in your organization, don’t wait to measure. The simple fact that you measure may increase/improve the interpersonal skills and beliefs of employees.

Employee Engagement Survey v. Self Assessment Profile

Organizations, government entities and educational institutions may be tempted to modify an existing employee survey to include diversity and inclusion by adding a few questions. When budgets are tight, this is an understandable impulse. Be aware however of three factors which may dissuade an organization of pursuing this method:

• A few questions are unlikely to provide meaningful data (see “Just the facts ma’am” section above);
• An employee engagement survey and a self-assessment profile are two entirely different animals (see “Profiles in action” section below);
• A survey informs the organization whereas a profile informs the individual.

Self-assessment profiles can shed light on an individual’s skills, abilities and talents. The difference between an employee survey and a self-assessment profile is illustrated by the following example.

EXAMPLE: MYERS-BRIGGS V. DISC

Carl Gustav Jung (basis for the Myers-Briggs Type Indicator or MBTI) sought to explain why people differ from one another by identifying fundamental personality traits or preferences rooted in biology. You don’t change personality, but it can help you change behavior if you understand where it’s coming from.

William Moulton Marston. (creator of the theory of behavior behind DiSC) sought to explain how people adjust to varying environments, by starting with their emotional response to it and relating this response to behavior. DiSC provides information for individuals to make a conscious choice to change behavior to improve results.

While the MBTI can be helpful in recognizing your personality style, a DiSC assessment provides actionable data and information for individuals. The underlying philosophy of the DiSC assessment and approach is that understanding yourself is the first step to understanding others.

Both self-assessments measure personality dimensions, although a little differently.

The primary difference to individuals is the report which accompanies the self-assessment.

DiSC describes behavior based on how people view themselves in their various environments. The Myers-Briggs Type Indicator (MBTI) describes how people approach the environment intellectually and attitudinally and how they process information.

The profiles obtained from each instrument cannot be expected to be correlated. They were designed for different purposes and based on different views of human behavior. Because the two instruments provide different kinds of information, they might very well augment each other as separate views of the same individual.

I see what you did there: The importance of anonymity

 Whether we are based on carbon or on silicon makes no fundamental difference; we should each be treated with appropriate respect. ~ Arthur C. Clarke, 2010: Odyssey Two

“Making privacy assurances to participants is one of the requirements of informed consent in psychological research, where participants are told that their information is anonymous and/or confidential. Survey researchers often make distinctions between these two concepts, and many past studies have been designed around the premise that anonymous and confidential (de-identified) experimental conditions will yield different outcomes. However, are participants making the same conceptual distinction, or are they interpreting these two concepts as a single construct? One study demonstrated that respondents to a Web-based survey were able to distinguish appropriately between anonymity and confidentiality in different experimental conditions.”9

• Perceived anonymity: “the degree to which the identity of a message source is unknown and unspecified; thus, the less knowledge one has about the source and the harder it is to specify who the source is among possible options, the more anonymity exists.”10 When a political pollster calls to ask how you’re intending to vote, that is perceived anonymity – even though the pollster has your name and phone number.
• Perceived confidentiality: “this often means that information about individuals is protected from inadvertent disclosure to others by physical means, such as a locked cabinet, legal means, such as signed confidentiality statements, or methodological means, such as the use of coded files.”11 When you give your medical history to your physician or clinic, that is perceived confidentiality– even though your doctor knows you by name.

For example, the Spectra Diversity and Inclusion Profile™ is confidential. Spectra Diversity knows who the respondents are, and the respondent has a reasonable degree of certainty that data shared with the organization are only shared in the aggregate (non-identifiable) form. To this end, any demographic data with nine or fewer respondents will not be shared.

A 2-step verification process is one means to provide a higher level of perceived confidentiality. It is the process that a bank or other financial institution uses to have you create and then sign in to an account.

“It is the responsibility of the researchers collecting data to ensure that steps have been taken to prevent the accidental disclosure of data, and accordingly, researchers are obligated to make assurances to respondents that adequate security measures have been taken.”12

Evaluation apprehension, originally defined by M. J. Rosenberg in 1965 is “an active, anxiety-toned concern that [the subject] win a positive evaluation from the experimenter, or at least that [the subject] provide no grounds for a negative one.” 13

When an individual perceives a survey as confidential, he or she will have less concern for social evaluation.

A study of 410 college students revealed that in a survey condition for which respondents’ identities were known, respondents understood that their information was not anonymous, but confidentiality was still guaranteed to some extent by the privacy assurances made to them.14

“Evaluation apprehension was shown to be significantly and negatively related to confidentiality perceptions, but not significantly related to anonymity perceptions.”15 So, when people are completing a survey or self-assessment, not only can they tell the difference between anonymity and confidentiality, but their ratings do not differ by any significant amount.

Just do it!” Linking profiles to action/engagement

Profiles in Action

A self-assessment profile may raise the consciousness of the individual taking the survey, and increase awareness, thereby improving behavior by some small amount. More importantly, a D&I profile can provide action steps for either an induvial or an organization, or both.

A Diversity and Inclusion self-assessment profile can:

• Reveal an organization’s perceived commitment to diversity and inclusion
• Indicate action items that those heading the effort internally would like to address because of the organization’s results
• Determine Individual development opportunities to focus training efforts based on the findings
• Help individuals by increasing awareness of their personal cultural lens and how that lens impacts others around diversity issues

The nitty gritty: Measurement in the D&I world

KEY FINDINGS FROM PEOPLE FLUENT 2015

• D&I challenges are consistent among organizations. If your organization is facing lack of budget, difficulty working across functional areas and a lack of accepted D&I benchmarks, you’re not alone.
• HR is often responsible for D&I initiatives. Other areas that typically integrate with D&I efforts are employee engagement, talent acquisition and learning and development.
• Vendor usage depends on D&I maturity. One in three companies uses D&I vendors to accomplish stated objectives.
• Measurement is key to successful diversity strategy. More than half of all organizations have been attempting to do something D&I related for at least four years. One of the areas which has been tracked consistently is workforce demographics. This tells an organization about diversity, but does nothing to measure inclusion.
• Effective organizations track more, measure more and reward more. “Advanced organizations have successfully used D&I metrics to make the business case for culture change.”16

TOP BARRIERS TO D&I SUCCESS

• Lack of reliable benchmarks (42%)
• Lack of budget for D&I resources or initiatives (41%)
• Difficulty working cross functionally (40%)
• Lack of resources (39%)
• Difficulty demonstrating impact of previous efforts (39%)17

Measurement in the D&I world

INDIVIDUAL

To understand others, it is important to begin with understanding yourself. This self-awareness can help you become more sensitive about how your behavior might affect others. The Spectra Diversity and Inclusion Profile™ also assesses respect for others. When you display openness and non-judgmental behavior it demonstrates that you value the diversity of others in the workplace. In turn, behavior is directly influenced by your self-awareness and interpersonal skills. By being self-aware, you can identify your belief system that drives your behavior. By understanding that, you can further choose to adapt behavior to be more inclusive of others in the workplace.

The profile measures the following:

• Beliefs (self-awareness and personal beliefs related to diversity and inclusion)
• Interpersonal skills (putting inclusion into action)

ORGANIZATION

Knowing where the organization’s commitment lies on diversity and inclusion matters greatly. An organization that recognizes how diversity and inclusion directly impacts employee engagement and innovation is a forward-thinking, inclusive organization. When employees believe their organization is invested in diversity and inclusion efforts and that all stakeholders (employees, customers, partners) feel welcomed, valued, respected and heard, the result is a more engaged workforce.

Organizations benefit by seeing an increase in productivity, innovation and morale. The profile measures the following:

• Management (how management sees itself and how management is seen by
• employees in terms of diversity and inclusion)
• Culture (the organizational norms and beliefs)
• Policies, Practices and Procedures (3Ps) (how diversity and inclusion efforts are put into practice on an organizational level)

Every line is the perfect length if you don’t measure  it. ~  Marty Rubin

Next steps: Moving the needle 

A Spectra Diversity client asked one of our partners for a program to deal with their racial bias issues. After surveys, focus groups and study of their employee population, it turned out that the organization’s problem was not racial in nature – it was generational.

If you don’t know where you are – it’s hard to develop a path forward. The purpose of this paper is to provide clarity and understanding regarding self-assessments used in the Diversity and Inclusion (D&I) space, and we hope we’ve done so.

We agree with PeopleFluent when they say, “To drive long-term, sustained change, [organizations] need to continually analyze the diversity of their workforce, identify gaps across people processes, and develop and execute D&I strategies that evolve their workforce toward their diversity goals.”

When an organization gathers a baseline measurement of diversity and inclusion perceptions, it can then develop a plan and execute on that plan to increase D&I throughout the entire organization, rather than as a stand-alone effort. To develop a diverse and inclusive organization you must:

•  Align D&I goals to organization’s strategic goals
• Invest in an assessment process and training that will give you a quick and measurable impact
• Create training time frames that works for your organization
• Inspire behavior change in participants, by engaging them in learning with experiential learning

Measurement is key. It’s never too late to start.

Sources

1. Joseph, Nancy, “Implicit Bias and the Presidential Election”, University of Washington, College of Arts and Sciences, November, 2016, retrieved on June 22, 2017 from https://artsci.washington.edu/news/2016-11/implicit-bias
2. Nie, Norman H., Dale H. Bent, Hadlai Hull, “SPSS: Statistical Package for the Social Sciences”, McGraw-Hill, 1970.
3. Mistar, TEFLIN Journal, Volume 22, Number 1, February 2011.
4. Mistar, ibid.
5. “Accuracy of Physician Self-Assessment Compared With Observed Measures of Competence A Systematic Review”, JAMA, 2006.
6. Schlössera, Thomas, David Dunning, Kerri L. Johnson, Justin Kruger, Journal of Economic Psychology, Volume 39, December 2013, Pages 85–100.
7. Ross, John A., “The Reliability, Validity, and Utility of Self-Assessment”, Practical Assessment, Research & Evaluation, Volume 11, Number 10, November, 2006.
8. Thawabieh, Ahmad M., “A Comparison between Students’ Self-Assessment and Teachers’ Assessment”. Journal of Curriculum and Teaching, Volume 6, Number 1, 2017.
9. Whelan, T.J., “Anonymity and confidentiality: Do survey respondents know the difference?”, Society of Southeastern Social Psychologists, Durham, NC, October, 2007.
10. Whelan, ibid.
11. Whelan, ibid.
12. Rosenberg, M.J., “When dissonance fails: On eliminating evaluation apprehension from attitude measurement”, Journal of Personality and Social Psychology, Volume 1, pages 28-42.
13. Whelan, ibid.
14. Whelan, ibid.
15. PeopleFluent, “Measuring Diversity for Success”, 2015.
16. PeopleFluent, ibid.
17. PeopleFluent, “Diversity & Inclusion in High Tech: Three Key Strategies for Promoting an Inclusive Workplace”, 2017.