10.05.2017

Are the Questions Validated? Does it even matter?

People who are new to culture surveys invariably ask the textbook question: “Are these questions validated?”

 

 

It is a fair question, of course. People want to know that surveys have been well constructed and the questions or statements have been vetted and stress tested. And I can assure you that, yes, we use questions that are validated in the classic sense. They have been drawn from research, tested against large and diverse populations, and have been shown to measure the thing that they're meant to measure.

The question always makes me smile, though, because people sometimes ask it without really thinking through what it means. If I’m feeling cheeky, I’ll respond: “Valid for what?” Validation is about ensuring from a scientific point of view that a question can be used in a certain context, so I’m curious to know if the client has thought through how they will use the responses in the context of their organization.

 I was working with a client who was frustrated because the consultant she had been collaborating with was suggesting one question and we were suggesting another. We both had a PhD on our side to back up the validity of our questions and the client didn’t know which to choose.

I said to her “at the end of the day, what matters is which question makes sense to you? This is your business, if you don't understand the question don't let anybody tell you that you need to ask it. You're the one is going to have to do something with the results, you pick the question that you think makes sense to you.”

For best results, start with the validated questions, and then take them and make them your own. Learn from how and why they were written the way they were, then mould them to fit your organization.

Fundamentally, if you're not going to do something with the answer don't ask it.

There is such a thing as a valid but useless question

Some questions have been validated, but serve little purpose. My favourite useless validated question is one of the Gallup Q12: “I have a good friend at work.”

I know Gallup has done lots of research to show that if people have a good friend at work then they're more likely to be engaged, and that's nice. But what are you meant to do with that? Should you go and buy someone a dog? It's weird and crosses a lot of different boundaries. I get the intent, but I think it’s more designed to get commentary than to create a useful outcome.

(We wrote about this question in detail in our previous post ‘Do you really need a “best friend” at work?’)

There are also unvalidated yet useful questions

Conversely, unvalidated questions can be useful if phrased correctly. Where there are no validated questions that correspond with what you're interested in, there will often be a validated form to assist you to phrase a question.

For example, every company's values are different, so not every question about every value has been asked and validated. But there are models you can use like “I believe at our company value X is used on a day-to-day basis” or “Here is the experience we want you to have. Is it occurring?”

One question we ask in our internal surveys that other organizations don't is, “I am motivated by our desire to be a culture first company”. That very much goes to the heart of the company we are.

 Should I ask that?

Checking that there is scientific validity behind survey questions is savvy, but you have to go beyond that. Survey design is not about a psychologist in an ivory tower dispensing validated questions; it's about thinking about the experience you want your people to have and then measuring if it's occurring. You should use the questions that help you answer that.

If there is one take-away from this article, it’s this: if you don't understand the question and what you’re going to do with the answer, don't ask it.

 

Didier Elzinga is a People Geek and CEO/Co-Founder of Culture Amp. You can follow Didier on MediumTwitter or LinkedIn.

About Culture Amp

We've brought together experts in technology, data science and organizational psychology to ensure you can get the clarity you need, when you need it. Measuring employee engagement, experience and 360° feedback is simple using Culture Amp.

Learn more