NSS 2021 - It's ok, the VC reads all the comments
NSS free text responses - to analyse or not to analyse?
We often contact universities directly to offer our NSS free text analysis service. One such approach resulted in a response of, "Thanks for letting us know about this, but we won't be needing your service as our Vice Chancellor reads all of the comments from the NSS".
On the plus side, this is great, the VC cares enough about the student experience to read all comments that were made in response to the NSS question: Looking back on the experience, are there any particularly positive or negative aspects you would like to highlight?
On the negative side, while this 'read-through' approach will give the VC a flavour of the issues raised, it is also in danger of resulting in highly skewed perceptions of the issues since there is a complete absence of any systematic and impartial analysis of the topics raised.
To illustrate how this could go wrong, here's a lesson from a project we undertook several years ago:
We conducted a student satisfaction survey for a university department. It included a free text response question similar to the NSS question. The department's head of student experience was also a lecturer. When he did a 'read-through' of all comments, he focussed in on two very specific comments which he thought were direct criticisms of his own teaching. They may not have been, but this is how he perceived it, and bear in mind that this was two comments out of just over 500 comments made in total (0.4% of comments made), so it was not a major issue.
But when I spoke to him all he could recall and all he wanted to talk about was this apparent criticism. I later presented him with a systematic analysis of the free text comments, and it emerged that across the 500 or so comments, teaching quality was mentioned five times more frequently in a positive than in a negative light, and that there were other far more prevalent issues evident among the negative comments which he, in his role as head of student experience, had the power to address.
So while a read-through of the comments is an admirable task for university leaders to take on, it is no substitute for a systematic analysis of the entire set of comments, which will not only identify the issues being raised by students (both positive and negatives), but will also contextualise them by revealing how prevalent the topics are. This in turn can help you in prioritising the further development of the student experience.
With the publication of the 2021 National Student Survey results and data sets due shortly, why not contact us today to find out more about how we can help by undertaking a systematic analysis of the comments and providing you with a summary of the topics mentioned and how often they were raised.