By Ralph E. Jordan, Founder & CEO
TRIDENT HEALTH RESOURCES, INC.
A mentor of mine years ago once told me, “Always be wary of survey results and ask a lot of questions.” That’s what we did when we came across new results of a salary survey conducted by Researcher Alicia Sievert, a certified perfusionist who works as an assistant professor and admissions coordinator at Medical University of South Carolina (MUSC). The results of this national survey of perfusionists were recently reported at a perfusionist only message board and reprinted on an industry blog, where permission for the survey’s public viewing was given (
LINK). A copy of the questions was found in the public domain and is presented below this editorial.
In short summary, the survey results report provided “average” salaries for perfusionists and an “average” length of time in their careers. Data was summarized for perfusionists working on adult cases versus pediatric or both, and if the perfusionist was full-time, part-time, or working as a chief. Pretty much, that’s it… no regional analysis, no discussion of findings or their limitations, just the numbers. Interpretation is left to the reader.
First, surveys can be a very good thing. Informal surveys conducted by companies or self-interest groups are understood to be snapshots of a condition (non-scientific) and are quite useful in better understanding situations. In fact, Trident has conducted its own surveys in the past to capture the self-reported attitudes and experiences of perfusionists. We have reported our findings in past issues of our newsletter and on our blog. I’m in favor of surveys, generally speaking.
There’s a responsibility and a caveat often necessary when reporting survey results, however. In the case of a salary survey – this is an entity of uniqueness and the integrity of the results must be scrutinized. To be valid and reliable, a survey instrument must measure what it is intended to measure and consistently measure over time, in repeated efforts. That said, when we first saw the 2011 salary results, we had many questions about how data was collected, the methodology, and the analysis of findings. We didn’t want to throw a reputable organization’s research under the bus – we wanted to communicate and learn more.
We made a list of our questions and contacted Ms. Sievert by both email and phone to tell her we were writing this article about the pitfalls of self-reported surveys. We acknowledge that her past work in research was published in a peer-reviewed journal and that she has credentials for performing research with integrity. She told us however, that unlike her published 2006 survey which was overseen by others in a class project, the 2011 survey was a shortened version she created, and the results were not analyzed (yet) with precision. In an email, she wrote, “This was just a survey for perfusionists that was meant to be taken as-is…”
But at Trident, we feel a responsibility to provide our staff with clarification, especially since salary surveys are leveraged in negotiations for new hires and pay increases of existing perfusionists. Employees historically bring salary surveys to their bosses as evidence they are underpaid. The 2011 salary survey utilized self-reported numbers, without qualification. So in a telephone conversation with Ms. Sievert, we asked the hard questions about how salary data was collected.
We asked how confident she was that perfusionists answered questions honestly. Consider: To ask someone “what is your total annual salary?” (without qualification) is akin to asking an athlete among his competitive peers how many bench presses is his average number. You might get an inflated number. One must ask if the data is measured statistically, or “jussayin’.” Is there a motive for inflating a response? You betcha! Ms. Sievert recognized that the data collection could, indeed, produce unreliable numbers, but she believes the majority of perfusionists surveyed were honest.
A valid salary survey would be based on data scientifically collected. For example, salaries could be determined with W-2s or pay stubs analyzed, or reported by disinterested parties (HR departments). Also, without criteria, how do you interpret the question, “What is your total annual salary?” Does it mean total compensation? I say this because at Trident Health Resources, Inc. we pay for our perfusionists’ continuing education, travel, and malpractice insurance, 401k contributions on the employee’s behalf, annual and merit bonuses based on performance, and contributions toward their health and life insurances – all this is part of a total compensation package. A total compensation package is significantly higher than a net or even a gross salary minus multiple benefits package. The researcher acknowledged that she received some comments from survey takers that said they received quarterly bonuses, etc. and she realized that some data (additional income) was not captured consistently based on the wording of the question – a limitation of her study. She said, “There was a lot of screening and cleaning of the data.”
We talked about the geographic representation of the sample population. It cannot be denied: regional differences in salary exist. California salaries are higher than Missouri, for example, ‘nuff said. In a future report that will hopefully be published, Ms. Sievert hopes to analyze her data and provide clarification. We simply point out that the numbers presented “as-is” are not meaningful without geographic consideration and interpretation.
And how was an “average salary” actually determined in this study? Statistical equation of mathematics? In any statistics analysis it is prudent to remove “outliers” – that means, of all participating surveys, cases of extreme that are outside a plot analysis would be thrown out of the mix. Keeping outliers into analysis skews the results. To skew means to distort or bias. That’s bad. Ms. Sievert indicated that extreme “highs” and “lows” were excluded from the survey. Unfortunately, these extremes were removed “by hand” and not with precision through statistical calculations using software. We point out that it is a pitfall in survey analysis to manipulate the data without more rigidity.
We seek interpretation of results. Specifically, in any analysis of “averages” one must ask what the standard deviation is. That is, what is the spread between the numbers? For example, you could have a salary of one person at $50,000 and one at $120,000. The average is $85,000. But in this example the standard deviation is huge! For these and other reasons it makes sense to use means and standard deviations, with some assumption that there will be a "normal" range for a position's data. An “average” salary amount is a flawed calculation method based on assumption. Standard deviation for a perfect normal distribution (bell shaped curve) means 68% of samples drawn from it fall within +/- one standard deviation of the mean. If salaries followed a normal distribution perfectly, we would expect 68 of 100 perfusionists to have a salary within 1 standard deviation of the mean.
Ms. Sievert addressed this, providing us the standard deviations in dollar terms. In the case of this salary survey, the mean (average salary) for all perfusionists is $109,773 (SD = 28046), meaning salaries in the survey were spread by about 3.9 RSD (relative standard deviation, represented as a percentage of salary). It would be helpful to readers of the survey to be provided with text language that clarifies what is “average” and how the spread affects the average so it is understood. Ms. Sievert, being closest to the data, is the authority on the data collected and her reflections about the results are key for full appreciation of the work.
We hope to see a future article from MUSC that answers many questions. I’ve presented herein only a few of the pitfalls of conducting a self-reported survey of salary. Bottom line, “self-reporting” of salaries is a weak methodology of data collection, as most researchers and statisticians would agree. It’s not valid. It’s not reliable. More importantly, in an industry full of clinicians who appreciate precision and exactamondo hard numbers, we appreciate getting the finer details of such a report. Without all the facts, conclusions are drawn irresponsibly. To use a metaphor, like a pebble thrown across a lake, there is a ripple effect when “word” gets out – a bit of information taken out of context – and the consequences are an uproar of reaction. We, at Trident, would like to impede that ripple effect.
Ms. Sievert gives us hope that we will be hearing more about her findings and we are thankful for the time she gave us to answer our questions.
Your comments are always welcomed!
Ralph
Copy of Survey found
HERE & copied below:
1. Demographic Data
City/Town
State
2. Age
3. Gender
Male
Female
4. How many years have you been a practicing perfusionist?
5. Define your position as a perfusionist:
Part-time staff/faculty
Full-time staff/faculty
Chief perfusionist
Other
6. YOUR specific perfusion practice is best described as:
Adult only
Pediatric Only
Pediatric & Adult
7. How many cases per year are YOU the primary perfusionist?
8. What is your total annual salary? (sample answer: 100,000)
