When It Comes To Credible Research, The Puck Stops There

A longtime friend has become a better friend to me as of late, because unlike way too many allegedly more loyal ones he occasionally reaches out to me and copies me on some items he’s seen in his mailbox, particularly from outlets behind paywalls because, you know, money’s kinda tight these days.  I had been vaguely familiar with a relatively new media reporting and opinion site called Puck News, founded by a bevy of former print and TV journalists that offers recaps of happenings in Hollywood, the ad sales scene and Silicon Valley, and with a sense of style and snark that I was told I could appreciate.

And, to be fair, the writing and commentary is indeed quite good.  But their lead story earlier this week was, as founder and reporter Matthew Belloni qualified, Puck’s first original research study.  As he explained, the questions being asked were solid and, in wake of news earlier this week about the mixed bag of results and the ensuing executive shifting at Netflix, all the more timely:

Beyond subscriber numbers and churn rates and ARPU, how do consumers actually feel about the various services that consume so much of their time? Do they like Netflix or HBO Max more? Do they think Paramount+ is a good value and can trust its programming? When they watch a show they enjoy, do they even remember which streamer they watched it on?

Those are all questions that contribute to something that doesn’t get talked about enough on earnings calls or at analyst conferences: brand attachment. 

I happen to be a champion of brand research, because when properly executed it reveals the why behind the how many.  Focus groups and in-depth interviews (IDIs), particularly when amplified by exercises stepped in psychological research, can often yield rich and frequently surprising insights, often to the interviewees themselves, when they reveal what they don’t even know what they know.  I’ve particularly loved exercises where we anthropomorphize a brand, in my experience typical a network or platform, and posit “Suppose MTV threw a party.  What kind of food would they serve?  How would the host dress?  What day and time would it likely be held?  Where would it take place?” .  We’d go through those choices and then apply them to networks as disparate as CNN, Showtime, Starz, TNT, etc.  In group settings, we’d break into smaller groups and have the triads or quartets debate those choices among each other.  Once we had our results, we’d ask “So wich party would YOU want an invite to?”  I wish I still had some of those tapes; they could make a fascinatingly blinge-worthy series or even a FAST channel for one of the streamers Belloni was seeking insights on.

But Belloni’s a journalist, not a researcher.  So when they chose the following methodology for this exercise, as a researcher, I was exceptionally disappointed:

To determine what we’re calling Puck’s Official Streaming Service Hierarchy, we engaged The Quorum, an L.A.-based research firm, which contracted with independent pollsters to survey 2,453 people in late December and early January on what they like, dislike, remember and don’t remember about the streamers they actually subscribe to and those they’re just “very familiar” with. The categories of inquiry were Quality, Trust, Admiration, Churn (how likely to cancel), Recall (match the show to the service), and finally a “Net Promoter Score” (we’ll get into what that means in a moment), which gives us our rankings.

And there’s the rub.  The Quorum is effectively a polling service, with an omnibus panel that regularly supplies clients with, as its website explains, distributing long-lead pre-release film tracking for the first time to the general public. For studios and politicians, such moment-in-time research, when conducted in several waves for tracking purposes, is valuable.  But movies and campaigns use these trackings to shape deeper dive research with carefully selected target responders.  Yes, such research takes time and considerably more money than the Streaming Service Hierarchy likely cost.  But anyone who is truly looking to try and understand the motivations and underlying components of what drives consumer behavior cannot credibly reach such conclusions with any quantitative study such as what The Quorum provided Puck, and what Belloni exclusively used for this debut study.

So when Belloni eagerly disseminates these results which were translated into a deliverable involving a Net Promoter Score, which as he further details was developed by the consulting and investment firm Bain and Company, and then offers this sort of summative recap, I kinda get my dander up.  Witness this recap for Hulu:

Hulu is the surprise for me. Despite launching in the first wave of streamers and scoring right behind Netflix in Familiarity, it’s in the middle of the pack for all the attachment metrics. Its scores are actually very similar to those for Prime Video, and I think the Hulu programming is generally more appealing than what’s on Amazon. So it might have a bit of a branding problem.

Great, Matt.  You’ve now given your own opinion-driven filter to the results your woke polling company fed you.  I’m sure you followed the commentary and remarks of Netflix’s outgoing co-CEO Reed Hastings during Thursday’s press conference, where he cited Hulu’s long-term evolution as a credible AVOD platform as the rationale and confidence that Netflix’s recently launched efforts (run by a former head of Hulu ad sales and key members of that team) will eventually bear fruit.  Funny, Hastings didn’t seem surprised that Hulu’s branding problem was an issue.  And, respectfully, what YOU think of Hulu’s programming doesn’t really matter when it comes to The Quorum’s surveys, unless, of course, you were part of the sample.

I’ve often been amused and frustrated by executives and colleagues who attempt to justify their instincts, or their own “focus groups” conducted among their family members, friends, and neighbors by selectively choosing to listen to and draw conclusions from snippets of more extensive research.  I’m even more snarky when people ask me what I think of a brand or a platform.   I remind them that people like us often wouldn’t qualify for most survey research–we’re out of the demo, we work in the industry, some of us make too much.  What WE think as researchers is irrevelant.  We do our best work by REPORTING on the results and insights gleaned from those who DO qualify.

You’d think Matt and his friends, being very successful reporters, might have developed the most basic skill of leaving themselves out of the story?  ‘Fraid not.

So, as a researcher, I’ll offer any one of you who subscribe to Puck dismiss this study, and any followups, with a grain or two of salt.  You can check out this chart for the bottom-line NPS scores for context, but with the caveat they are little more than a starting point.

In the spirit of collegiality and respect, I do think Puck’s reporting is excellent, and would offer that if you can afford it you should consider subscribing.  Perhaps if their next research study were run by a seasoned expert (hand raised) before they spend dime one on it, the site might improve its value proposition.

I believe I write well, but my reporting skills aren’t top-notch.  I’d rely far more on Puck’s team and contacts for that.  I’d offer that when it comes to research, similar self-awareness might be appropriate.

Because based upon what I’ve seen, this research is little more than a bunch of Belloni.

Until next time…

NOTE:  If you’re looking for better-conducted research, and some darn fine people presenting it, please join myself and others at the Media Insights Conference next week in San Diego.  Click on the link below to register; message me for a discount.


The Media Insights & Engagement Conference (informaconnect.com)


Leave a Comment