Surveys and Reports on the Internet: They must be true!

It’s 2009. The media and blogosphere are abuzz with posts reflecting on the previous year and predicting the upcoming one.

Why is a Raven like a Writing Desk?

But it’s 2009. This is a different marketing landscape. This year we are data-hungry. (Ironically, our hunger for data cannot be quantified.) And to fulfill our desire for more data we also have plenty of surveys and reports being laid on the table.

However, before we dive ravenously into these tasty morsels it would benefit us to keep in mind a few common sense rules to determine which are the meat and which are the empty calories.

1. Just because it’s published on the Internet doesn’t mean it’s true.

(I hope no further explanation of this point is necessary.)

2. Check the source of the data to understand its biases.

It’s very difficult to find truly independent research these days. Analyst firms are gold standard for industry research. You should still look at their client list and check if a company sponsored a particular report.

For other surveys and reports, look into the company that performed the study. Are they a part of the industry they are reporting on? Do they present positive and negative findings from their research?

When you see a company’s survey results or report cited in media outlets check for any connection the publisher might have with that company. This does not invalidate a report but it would make me want a second source to confirm the findings.

3. Understand the methodology that they used.

Don’t worry; I’m not talking about a PhD-level understanding of statistics. One of the first things that I look for in any study is the sample size. The second is how random their sample is – if they’ve only surveyed their customers it’s not truly random. And finally, if they present the original questions alongside the findings.

This basic level of transparency lets you determine if you believe the survey was crafted well enough to be significant or if it might be biased towards a particular finding.

4. Play Devil’s Advocate with the analysis.

One of the benefits of being data-driven is that you get to use the insights of others to help guide your business. Rather than re-inventing the wheel, you can read a report describing the uses of the wheel to make business profitable.

One of the dangers is that you can use data as a shortcut to insight without going through the critical thinking process. When you read the take-aways from a survey, don’t believe them. Jot them down and keep them next to you. Then as you go through the data, see if those insights are proven by what you see in the data.

It’s 2009. And there’s a whole delicious world out there waiting for you. By eating smart and staying nimble, you’ll stay in good shape throughout the year.

Image Credit: Lisa Williams