Why Customer References are a Gigantic Missed Opportunity in Selecting a CMS

Coming up next on Who Want's to make a CMS Vendor a Millionaire

I’ve never counted the number of CMS selection engagements I’ve been a part of, but safe to say it’s a lot. One of the many benefits of being a card-carrying member of the CMS Mythbuster brigade.

We’re always curious how organizations make their selection and what inputs contribute most to the final decision. While all decisions are different, it’s shocking to note how little time gets spent talking to actual customer references.

We just wrapped up our participation in a selection process where a very large organization spent over a year building the business case for a new web content management system and evaluating vendors. We were partnered with one of the two finalists being considered. Our technology partner didn’t end up winning, but it was clearly a very close decision (in their words) between two seemingly evenly matched products and professional services companies.

When asked what impact our customer references had on the decision, the client indicated they had never called them.

We were stunned.

How could such a close decision not be informed by what peer organizations had to say about their experiences with the platform and professional services partners? Especially in the rough and tumble world of web content management where the gap between what’s sold and what’s delivered can be cavernous.

Ok, we weren’t too stunned, because in our experience this happens quite often. And this isn’t a rant about a deal going south. It’s part of a larger trend we’ve seen over the years with companies undervaluing the role of customer references.

A small number of the hundreds of references we’ve provided potential customers over the years ever get called. It’s also what we refer to as the CMS Selection Myth, where the process of evaluating the product consumes a disproportionate amount of time that could be better spent on real strategic planning.

We used to get frustrated being asked to supply references that were never contacted. Now we just see it as a gigantic missed opportunity for organizations that desperately need all the information they can get to make an informed choice about a new CMS — or any enterprise technology product for that matter. This is often a six, or seven figure decision on a platform and a multi-year partnership with an organization.

Regardless of whether or not you feel customer references are necessary to make a decision, the insight you will glean from talking to peers who have gone through the process is invaluable.

Think about it: You’ve been given a list and permission to contact a half a dozen to a dozen people who have recently gone through a large project very similar to the one you are about to embark on. As Kenny Bania would tell Jerry Seinfeld, “That’s Gold Jerry, GOLD!” At one point in the not too distant past, these folks were in the same position you were; kicking the tires on multiple solutions, determining project budgets, organizing internal teams and wrestling with schedules and timelines.

We see four opportunities to improve the way references are used.

Asking the Right Questions

It’s safe to assume the references provided will have favorable things to say about a given vendor. However, dig a bit deeper and they can also shed light onto the process it took to get to the finish line. You’ll gather valuable information about how the vendor and the client both approached the project and the type of teams, budgets and support required for success.

By talking more specifically about the overall approach to content management, you’ll also get a more honest assessment of the end to end process.  This, in turn, will surface additional information about how the vendor performed in specific circumstances.

There’s no shortage of important questions to be asked:

  • How did the vendor staff their teams?
  • What could have been planned better?
  • If the client had it to do all over again, what would they do differently?
  • How are they using complimentary technologies and partners to be successful?
  • What other technology and integration partners did they consider?
  • What was the original budget and biggest areas of unforeseen cost?
  • How are they managing the post launch activities?
  • How did they handle content migration?
  • What are the most common support requests?
  • How is the developer community?
  • Have they been able to train new developers easily on the platform?
  • How well do the developers like working with the platform?

Of course it’s important to respect the references’s time, but if you are upfront with your intent, folks are typically happy to lend advice and share experiences. As we’ve talked about before, selecting a CMS is very much like buying a submarine. The average person doesn’t go through the process very many times, so communicating with those who have is an important opportunity.

Talking to the Right Individuals

A big challenge with references is that the people who purchase the CMS aren’t usually the people who use the CMS, or even oversee the project. The more senior the title, the more further detached they probably were from the actual implementation.

When asking for references, don’t be afraid to ask for multiple people from the same organization. At the very least be clear about who you are talking with and how involved they have been with the project. It’s important to talk with the executive sponsor who had budget authority and made the purchase, but it’s equally important to talk with the team exposed to the day-to-day challenges of designing, building and deploying the CMS.

It’s also important to have the right person on your team contacting the reference, ideally a peer role. It’s fine to have your procurement department ask questions about the overall health of the vendor relationship, but in order to have a real conversation about the project and challenges, you’ll want to have someone on your team that is hands on enough with the process and industry to dig into the details and ask good follow-up questions.

It can also be particularly useful to connect someone on your development team with one of their developers to get more information on what it was really like to work with the system, support end users, and learn the new platform.

Calling References Earlier in the Process

One of the biggest mistakes is waiting until the very end of the selection process to reach references. It’s often a final check point to ensure there are no problems with a vendor already selected. This is missing a big opportunity to gain information early in the process from customers who can help inform your overall approach to the project. Not to mention it’s harder to back out of a decision that’s already been made if you do get some unsettling feedback.

Talking to references earlier can provide the following insight:

  • Guidance on the overall budget and scope of an implementation that would dictate your budget for both software and services
  • Significant red flags that may cause you to drop a vendor from the shortlist early in the process
  • Surfacing additional vendors that may not have been on your radar
  • More self-education on how CMS works to ask better questions from the vendors during the evaluation process

Going Beyond the Provided List

It’s typical to ask for three references, and it’s a safe bet they will be three of the vendor’s most dependable evangelists  But it’s important to do some homework and find other organizations and people that are currently using the CMS platforms under consideration. This isn’t to go behind the back of the vendor to search for dirt, but rather to get a more complete picture of what it’s like to work with that platform. The vendor is often limited in the number of references they can provide as many organizations do not want to be references, or key contacts may have left the company.

Other types of people and organizations that make sense to connect with:

  • Individuals in similar vertical industries and sized companies, as you’ll find more similarities with the challenges and scenarios
  • Companies that have recently switched off of the platform your organization is leaving, or considering
  • Analysts who cover the industry
  • Partners who work with multiple CMS vendors

In summary, don’t underestimate the value of talking to fellow CMS customers in the CMS selection process. Take it seriously, and devote a significant amount of time to it during your evaluation. It’s a unique opportunity to gain valuable insight on what it takes to be successful with CMS and make an educated choice about the right vendors to partner with.

Featured Research: CMS Wisdom Report

CMS Wisdom ReportA free report with 54-inspired tales of web content management trials, tribulations and tips from the men and women who lived to tell their stories. Hear what worked (and what didn’t) from real-life CMS implementations.

Download the CMS Wisdom Report 

About the Author
Jeff Cram

Jeff Cram is Chief Strategy Officer and co-founder of Connective DX (formerly ISITE Design), a digital agency based in Portland, OR and Boston, MA. As the Managing Editor of the CMS Myth, Jeff is passionate about all topics related to content management, digital strategy and experience design.

More articles from Jeff Cram


2 responses… read them below or add one.

  1. Great article.

    I’ve worked with a few organisations on the client side and as a consultant, helping them make these sorts purchasing decisions, and have always included speaking to people in similar organisations, with similar experiences (ie an ‘environmental scan’), as a basic part of the requirements discovery and selection process.

    After all, chances are you’re not the first person in the universe to have done this, so why reinvent the wheel from scratch?

    Now I’m on the vendor side of the fence, and I admit I’m not surprised by your observation that many clients could do this better.

  2. Thanks for the insights. I have never thought about really calling a reference even though it makes sense now. I would guess that our clients never have called any of my references either. That’s kinda sad because most of them would pledge my case I guess.

Leave a Reply
  1. Fields marked with * are required.
  2. We will not publish your email.