In 2003, Fred Reichheld, a marketing consultant at Bain Consulting, published an article in the Harvard Business Review titled:  ‘The One Number You Need to Grow,’ with a tantalizing claim that the response to this one question, converted to a simple metric called Net Promotor Score or NPS, can predict your organization’s revenue growth rate. It then made even a bolder claim that no other metric is better than NPS in doing so.

By the time research by independent observers showed that these claims were either overreaching or false, the NPS has already gained a serious foothold in many organizations including many Fortune 500 companies. Lured by the simplicity of one question and Harvard Business Review’s cache (followed by two books on the subject by Reichheld), the simplicity of a single metric that anyone can compute with a simple calculator, and the golden promise of linkage to revenue growth (never in the history of business before has there been a single metric that could predict revenue growth), NPS, despite its many flaws has grown like wildfire. As a result of high usage, it was possible to create a set of benchmarking data that other organizations could use, thus increasing the appeal of the NPS metric.

At the same time, many researchers started checking on Reichheld’s claims and found many issues with these claims as well as other aspects of NPS formulation.  This article is meant to provide a perspective on the validity and utility of NPS.

What is NPS?

Although NPS is widely known, it is useful to list it here as a reference for our discussion.

“How likely is it that you would recommend [Organization X] to a friend or colleague?”

Customers are asked to rate their answers on a 0-10 scale, which is divided up into three categories: “Detractors (0 – 6),” “Passives (7 – 8),” and “Promoters (9 – 10)”

NPS = % of Promotors – % of Detractors.

Once organizations started using NPS, they realized that they had the NPS as a number, but didn’t know what to do with it. If the number was low, they had no idea why it was low, and what actions they could take to improve it. Based on this rising criticism, Fred expanded the one question to two questions, the second question being an open-ended question asking why the respondent answered the NPS question the way she did in order to explore the reasons. Now you still had to read, code, text analyze and collate these observations to come up with some insights and action avenues.

(As a personal note, PeriscopeIQ was the first to introduce the concept of metrics taxonomy (using right metrics for right strategic objectives) and introduced its own ‘NPS’ (called Net Recommendation Score or NRS) long before there was NPS. NRS avoided many of the issues attributed to NPS.)

Issues with NPS

Some researchers question the formulation of the question itself as a speculative (How likely is that you would recommend____, rather than a behavior formulation, In the last 6 weeks, did you recommend___). Some feel that the addressed audience (__ a friend or colleague) is not comprehensive enough and maybe a family member (_ a friend, colleague or family member) should be added who is usually the first recipient of such recommendations. In our view, such criticism is minor, and the question formulation is fine, but it should be refined slightly based on the audience.

As indicated earlier, many serious researchers validly challenge the expansive claims of NPS as being the best metric to correlate to revenue growth. In fact, some researchers have shown using the same data (to the best extent possible) as used by Reinbach that a well-known metric, American Customer Satisfaction Index (ACSI) reported now for over 25 years, is as good as the NPS in making such predictions. Further, correlation is not causation. There is no doubt that high customer loyalty is one of the key factors in driving revenue growth but companies that do one thing right also do many things right to help drive the growth.

The most important issue about NPS is how NPS is calculated. The concept, although somewhat intuitive, is totally random. On a highly dispersive scale of 0 to 10, it is hard to distinguish between giving a rating of 8 to 9. But the outcome is stark: Give a rating of 9, and your individual NPS is 100. Give a rating of 8, and your individual NPS score is 0. If someone gives you a rating of 0 (Not at all likely, usually given by people who are disgusted with their experience) or a rating of 6 which is a kind of neutral rating, your individual score is still -100. These random cut-offs and large reversals, without any true scientific justification, give a fit to serious researchers.

NPS’s claims that it works for nearly every business or organization are overreaching. NPS is a metric based on voluntary recommendations that work better for some businesses than others. Businesses that are based on standardized products and direct-to-consumer sales benefit more from the use of NPS. B-to-B businesses, businesses offering complex products such as computer hardware and software require multidimensional insights (rather than only one offered by NPS) from the customers to drive success. Understanding these limitations is important.

NPS Merits

The biggest thing going for NPS is its ubiquity. NPS is now used widely, across nearly all industries and its use is spreading globally. This wide use allows access to a wide range of benchmarking data (although the data quality and poor application of such data remain problematic) that can be very beneficial for process improvements. Further, NPS has become an industry in itself, spawning consultants, trainers, conferences and seminars that allow opportunities for learning and interaction with other professionals facing similar challenges.

Despite all the challenges and concerns whether it is the best metric as claimed or not, there is a general agreement that it is a useful metric when applied properly. Considering its popularity and availability of related resources including benchmarks, it is a metric that should be considered.

NPS is simple to compute and easy to understand.

Key recommendations in using NPS

First, if your organization has been using other metrics such as customer satisfaction or likelihood to purchase in the past and you have found those to be useful, there is no need to get rid of those metrics and replace those with NPS. As indicated earlier, researchers have found that some of these metrics are as good or better for predicting desired outcomes. You may want to add NPS as an additional question to the survey if your management insists its addition for the sake of benchmarking. If you are conducting your study properly, you will see a high correlation between NPS and other metrics anyhow.

One of the biggest and highly valid criticism of NPS is that it is a lone question, it tells ‘what’ but not ‘why’. The addition of the open-ended question is useful, but the actual experience shows that many respondents don’t leave any comment and those who do require detailed review and categorization to make much sense of the data. Text analytics software is useful but not highly reliable despite all the advances in text analytics and is likely to remain so considering the wide variance in style and content of the comments. We recommend a 4 to 6 question survey (less than 2 minutes of expected completion time) which includes a few structured questions focused on the underlying reasons for the response. It would also be a desirable to include a satisfaction question, and an open-ended question for general comments. We like the recent trend in NPS surveys where the NPS question is included in the invitation email message, and a response to the question leads to an expanded survey which respondents may choose to respond to it or not.

NPS benchmarks are useful for knowing where your organization ranks relative to others, but an obsession with benchmarking can be more harmful than useful. First, one does not know how valid and reliable the reported benchmark data is as it is rarely accompanied by reliable statistics to assess its validity. Second, such benchmarks vary substantially by the type of industry, and whether the organization’s product is a physical product or a service. For example, NPS of a consulting company or a membership association should not be benchmarked against NPS for retailers. So, if you are benchmarking yourself, make sure that you do so with reliable benchmarks for comparable industries.

Much more important than external benchmarking is internal benchmarking by setting your own goals every year and then trying to meet them. Amazon really set the stage for obsession with customer excellence but never looked for and still does not look for external benchmarks. External benchmarking can be helpful if your scores are very low and then deciding what your internal target should be for the next year. With external benchmarking, there can also be smug satisfaction if your scores are comparable to the averages, but it could hide some serious pitfalls.

– One of the key scientific arguments against NPS, as indicated earlier, is illogical scoring (where the value of response of 6 gets the same score as a response of 0) to compute NPS. Some organizations separate out the % count of those who select a response value from 0 to 2 and call them extreme detractors or some other similar nomenclature. These organizations report both NPS and % extreme detractors (who are highly dissatisfied and are much more likely to bad mouth you). If only one metric is to be used, we recommend a separate benchmark (call it Modified NPS) where extreme detractors are scored as -2 (instead of -1). So, the Modified NPS would be = NPS – % extreme detractors. Obviously, only NPS should be used for benchmarking but the focus on extreme detractors is important which under the normal NPS schema gets washed out along with other detractors.

– NPS is a loyalty metric and it is not the same as a transactional satisfaction metric. Loyalty is built based on experience with multiple transactions and experiences, and similarly, it typically takes more than one bad experience to create a serious dent in loyalty. For example, one bad experience with returning an item to Amazon is not going to change how likely are you to recommend Amazon to your friends. However, if asked about the satisfaction with a particular transaction, the response may be highly negative and a drastic shift from the experience with a previous transaction. That’s why it is important to understand the relationship between loyalty and satisfaction. Normally there is a direct correlation between loyalty and satisfaction but there are instances when it is not. Understanding and properly dealing with those instances is very important to build a culture of customer excellence which will yield both higher satisfaction and NPS metrics. Unfortunately, we have seen that the NPS is now widely used for measuring transactional satisfaction which can lead to many invalid responses.

– And, finally, NPS, despite its popularity, is not really the best metric for all industries. This is specifically true for B2B organizations with infrequent transactions of moderate or high value. The buyers here can’t reliably answer the NPS question as they are not going out and recommending an organization to their colleagues and friends. It is much better to ask a question about their satisfaction with a particular transaction and all its attributes (quality, timeliness, price, etc.) than ask the NPS question (this is also the reason that a properly designed satisfaction metric measured over a long period has a much wider applicability and is a more reliable metric for desirable outcomes). We have seen that the NPS question is blindly used for nearly every occasion. If the NPS question must be used for a certain reason, we suggest modifying question to: If asked, how likely are you to recommend …

Organizational success depends as much on how organizations react to external environment and market forces as on internal excellence. This has become even more important in the last decade as the forces of rapid technology shifts and globalization have upended normal business practices.

About Unilever

Unilever, a leading multinational consumer company, with 250,000 employees in 134 countries, is highly regarded for its product quality and sales effectiveness. The company runs an internal global training center, providing training in scores of key skills and competencies, for its salesforce.

The Challenge

Unilever needed to optimize the effectiveness of their company-wide, global training program. Although effective, the systems relied on voluntary initiatives or a supervisorʼs recommendations, for staff to request and receive training. The company did not have a system to actively identify gaps between an employeeʼs skills and required key skills for their position. Filling such gaps and equipping each person with all the skills required for their position will help the individual to achieve their full potential while helping the organization achieve higher productivity and greater revenue.

The PeriscopeIQ Solution

PeriscopeIQ and the company worked closely to develop a Skills Optimization System that:

• Helps individuals assess their own skills relative to skills required for their position
• Helps the individualʼs supervisor assess their skills relative to skills required
• Identifies gaps between the individualʼs and the supervisorʼs perceptions, leading to a dialogue
• Identifies the right training courses according to skill gaps, and allow individuals to sign up for these courses
• Automatically builds an executive reporting system allowing managers to look at skills need and gaps on an aggregate and individual basis for all the employees reporting to them
• Allows senior executives to drill down and identify skills needs and gaps by managers, departments, positions, or key skills
• Allows training director, for the first time, to have a predictive look into training needs and tailoring the program to meet such needs

Based on the PeriscopeIQ’s Employee Insights solution’s 180-degree feedback system and leveraging PeriscopeIQʼs powerful analytical engine and assessment design capabilities, the system was developed in less than two months. Key attributes of the systems included:

• Completely online system with results available in real-time
• Immediate feedback on skills gaps, targeted training courses and access to training signups upon feed back submission
• Immediate, real-time updating of the Executive Reporting System
• Automated workflow system allowing an employee and their supervisor to know the status of the assessment submission
• A permission-based portal allowing managers and executive to login and have access to only the right information
• Turnkey project requiring zero IT or other resources from the company

The system was first used in the United States to assess skills needs and gaps of nearly 500 professionals, and then expanded to Caribbean countries where it was deployed in Spanish and English. The system will next be deployed in scores of countries in Asia and Europe in multiple languages.

Outcome and Impact

For the first time, Unilever was able to develop a profile of skill needs and gaps across multiple groups, departments, positions and individuals.  At the executive level, this knowledge is transforming the way the company manages its training needs and development, and allowing managers to map skills and competencies in their departments as never before. At the individual level, the professionals are now able to see the competency gaps, apply for targeted training and have a basis for meaningful dialogue with their supervisors for professional growth and development. For these reasons, the company is launching the system globally.

About PMI

The Project Management Institute (PMI) is the world’s leading not-for-profit membership association for the project management profession, with more than a million members and credential holders in more than 180 countries.  Their worldwide advocacy for project management is supported by globally recognized standards and credentials, extensive research programs, and professional development opportunities. PMI is widely regarded as an expert in employment and compensation issues for project managers worldwide, publishing countless research reports and studies annually around the world

The Challenge

Many years ago, PMI management determined that the organization needed to assess the PM industry on a more comprehensive, global basis and needed to understand how the recession was affecting project manager salaries, which couldn’t be determined. “At the time, PMI used outside research firms on a sporadic basis and lacked a formalized approach when I came on board,” explains Jason Dolfi, manager organization products. “In addition, PMI needed more reliable data and better reporting capabilities, as well as, a higher member rate of response.”

Researching A Better Solution

PeriscopeIQ was engaged to provide two major research platforms, one for quarterly surveys of members using the PeriscopeIQ’s Employee Insights solution (PeriscopeMX). A second equally important initiative was to compile, analyze and report on industry salaries using PeriscopeCOMP and other tools.

Superior Membership Feedback Using PeriscopeMX

As customer feedback management experts, PeriscopeIQ instituted a more consistent and formal system for measuring member satisfaction. Conducted on a quarterly basis, PMI learned more about member satisfaction levels. They also handled customer support and certification for tracking, on a quarterly basis.

Project Management Salary Perspectives with PeriscopeCOMP

PeriscopeCOMP is a unique, online compensation survey platform ideal for aligning compensation strategies with employee expectations. It provides data on how compensation strategies impact morale, productivity and employee retention. PeriscopeCOMP also includes a process for ensuring the effectiveness of targeted incentives.  It adeptly and economically handles a wide array of compensation and incentive intelligence tasks, including:

• National or regional compensation studies
• Ad-hoc or client compensation studies
• Employee benefit studies
• Sales incentive design studies
• Compensation value optimization studies

PMI used PeriscopeIQ’s compensation platform to survey and produce reports for close to 20 countries (now expanded to 42 countries in 2019) on a country by country basis. This project was one of the world’s largest project manager specific salary surveys and positioned PMI as thought leaders.  On the back end, PeriscopeIQ provided strategic insights, which added a deeper level of understanding of the Project Management profession.  PeriscopeIQ collected self-reported salary information from the membership base and shared the findings with the professionals. “This was a great way to dramatically increase the number of participants, which was approximately 30-40 percent. The data was not only more reliable, but highly useful,” Dolfi says.

PMI was able to leverage the data as well as position itself as a thought leader. The Wall Street Journal quoted PMI’s study, which showed that in the global recession, salaries of project managers were increasing, which validates why project management is an attractive discipline. Dolfi notes, “this is another tangible benefit our members get for being part of the profession.”

PMI’S Perspective on PeriscopeIQ–Trusted Intelligence and Insight

According to Dolfi, “PeriscopeIQ helps companies create programs, not just a survey. They become part of our operations and then they help with strategic thinking. This aspect separates them – they go beyond the data. They have a very clear value proposition and core competency, and that is delivering very high-value and high-quality research. “I see them as more of a high-end vendor in terms of their capabilities and staff. They have a lot of PhDs on staff, and we work with them and through their network. Those people spend a lot of time with the data, and that’s one of the more unique aspects of their organization.”

About PRM

PRM is a professional services firm specializing in providing a full range of human resources and general management consulting services. Their client base includes not-for-profit, higher education, and public sector organizations, along with general industry companies who believe that people are the ultimate drivers of organizational success.

The Challenge

PRM needed to optimize their survey process so that they could turn survey requests around more efficiently and provide a higher level of data quality to their clients and survey subscribers.  Although they currently provided high quality data in a meaningful manner for their clients, they needed to better automate the process and provide more flexibility for client reports and other projects.  The survey process for PRM, as with most comprehensive surveys, is a long and arduous process.  It can take from 3-6 months to collect the data and 1-2 months to analyze it. Initially the data can be very unfriendly and doesn’t allow you to easily look at it in different ways without re-doing it when using a manual process. In addition, to really do this properly you need internal data analysis expertise. If you count all of the hours that people spend, and the expense of the internal expertise, it’s a very expensive and time-consuming process if you want to deliver high quality data as PRM does.

The PeriscopeIQ Solution

PRM worked closely with the PeriscopeIQ team to assess and implement thePeriscopeCOMP and Compensation On-Demand Analytics (CODA) solutions for its not-for-profit compensation survey database, one of the largest and longest-running databases of its kind in the United States.  PeriscopeCOMP is a unique, online compensation survey platform that includes comprehensive pay and benefits information, advanced survey logic, sophisticated data validation, and extensive reports for external analysis. PeriscopeCOMP has a direct integration with CODA, which creates accessible, customizable and accurate compensation reports that contain as much or as little information as desired. In addition, the CODA proprietary algorithms ensure compliance with Safe Harbor provisions.  PeriscopeIQ’s Compensation On-Demand Analytics (CODA™) platform revolutionizes the collection and real-time cleaning of salary and compensation survey data. With CODA, organizations can reduce their data cleaning from days to minutes, create customizable and easy-to-use reports, and greatly reduce their exposure to risk by ensuring they are compliant with Safe Harbor Provisions applicable to compensation surveys. CODA is the only product available that utilizes proprietary algorithms to keep your organization legally compliant without discarding important data.What impressed the PRM team was the fact that not only were they able to collect the data faster and with noticeably fewer errors, but they could deliver a much larger range of tailored reports and analyses for their clients

Outcome and Impact

PRM’s data quality has dramatically improved, the solution is very easy to use compared to the previous manual process, and they are also seeing a much better survey response rate.  “Now that we have the solution set up and running, our surveys next year will be a breeze”, said Lyn McCloskey of PRM.  “We will be able to quickly set it up and send it out in just a few hours.  ”A next step for PRM will be to implement PeriscopePAY, which will allow non-HR users to get the data they need, on-demand, without needing the previously necessary analytical skills.