AI-Driven Content: From Guesswork to Governance

How do we move from guesswork to governance in the AI generated content era? How can we ensure the quality and relevance of content while harnessing the power of AI to scale production?

For this episode, we’re turning the tables on our host, Christopher Willis, who will be the guest and our producer Charlotte Baxter-Read steps in to host (and quiz) Christopher. Be prepared to discuss the ever-evolving landscape of content creation, AI content creation tools, and the pivotal role of governance in shaping the future of high quality content.

Christopher, who holds the unique roles of CMO (Chief Marketing Officer) and CPO (Chief Pipeline Officer) at Acrolinx, shares his insights into the dynamic interplay between AI-driven content and the need for robust governance strategies. And in his “Provocative Statement of the Day” Chris claims that sometimes it doesn’t even matter what the content is. 

Join us as we explore the challenges and opportunities presented by training generative AI on your own content, balancing automation with human oversight, and crafting content that truly engages your target audience. From spurious correlations to the impact on brand reputation, this episode promises a candid discussion on the critical role of content governance in shaping the future of content creation.

So, without further ado, let’s dive into the fascinating world of AI-driven content and the journey from guesswork to governance with Christopher as our guide.

Watch the episode

Listen to the episode

Read the full episode transcript 

Hi Chris, welcome to the show!

Well, hello Charlotte, thank you for having me.

How funny is it being on the other side?

I’m a little uncomfortable and concerned about what’s to come, but let’s see what happens.

Well, in typical WordBirds fashion, we’ll jump straight into the quick-fire questions. So today we want to know what’s your best or has been your best content campaign?

So I think I’m going to say Factors Magazine, and we can come back to talk about that in a bit. But the idea of creating an actionable magazine on a quarterly basis to be able to build an audience and also use that to drive sales.

And your worst content campaign?

I’m sorry because there’s a guest coming up this season and we’re probably going to be talking about this, but the “Don’t Let Bugs Get You Down” campaign.

Very intriguing. We’ll have to unpack that one in a moment. And then out of the two, which do you think you’ve learned the most from?

Both, but that’s not the point, and I hate it when people do that on my show, so I’m going to answer your question. And I think it’s Factors. Let me explain what Factors Magazine was. 

So one company ago, I was in the mobile cloud testing space, and our customers were deciding how they wanted to cover their consumer audience from a mobile device testing standpoint. So if you’re a bank, you’re going to say, “I want to test a number of devices in market.” Say 10 iPhones, 10 Samsung, 10 Google phones, whatever, and that’s going to be your test bed. 

And what we did was we created a magazine. We bought data for this magazine, made it look really nice and official like a magazine and published it on a quarterly basis. That magazine included every device in market at that time on a regional and industry basis. So that if you’re a financial services company or a bank in North America, here are the devices in your region that your audience is most likely to use. Whether you were a customer of ours or not, you wanted access to that data. It was data that we actually had to buy. So it’s something that people wanted. So thousands of people subscribed to Factors Magazine and thousands of people became leads.

I do think content represents the brand where people can't. So if somebody stumbles upon your brand and their one interaction with you is clickbait, they're going to be disappointed. Share on X

It was also something that we used in the sales cycle because a customer would say, “I think we want to test on 10 or 15 devices.” Oh, cool. Flip in the magazine. So based on what you do and where you are and all this data, 10 devices represents about 20% test coverage of your total addressable market. Is that okay with you? No, that’s not okay. We’re a major U.S. Bank. We need like 80 to 90% test coverage. Fantastic. Flip through the book. That’s going to be about 150 devices that you need to test on. And what we just did was we took a deal from like $90,000 a year to over a million dollars a year with a flip of a magazine page. So this content wasn’t just something that drew in thousands of people on a quarterly basis, but also something that helped us increase the value of the product that we were selling.

So what would you say were the qualities of that magazine, the content qualities, that really teed you up for success there?

It was about the actionability of it. Even if you’re never going to use our product, even if you’ll never automate your testing process for your mobile devices, here’s information on what you need to be considering as you bring a mobile application to market. So that idea of inclusive value, you’re not a customer, but you’re still going to benefit from this.

I think that we do that today, I don’t know if you’re familiar with the people in our content group. But we do that with the content we create today. We’re not creating content for Acrolinx customers. We’re creating content for the marketplace. And it goes well beyond people that will ever buy our product. So it’s building awareness, it’s creating goodwill, and in the end, driving leads that turn into pipeline revenue.

I guess education and actually providing value to the reader, regardless of what state they’re in the customer journey, can definitely build trust. And the power within content to do that is probably second to none. 

But I’d like to circle back to your worst content campaign, because I’d love to understand more about that one.

Sure. So again, I’m alienating a guest on this season’s show because this is something that we did together. But again, the same company, mobile cloud testing, selling to DevOps and the idea that they just want to code. They don’t want to spend a lot of time testing. So don’t let these bugs, test bugs, software bugs, get you down. And bugs turned into bugs in two different ways.

So the first one, and this is kind of fun, the team wanted to do, our creative director said, I got this great idea. We’re going to get a sign truck and we’re going to get a whole collection of vintage Volkswagen bugs. And we’re going to drive around at this conference in San Francisco, drive around the Embarcadero area with this parade of sign trucks and Volkswagen’s throwing t-shirts out the window around the theme of don’t let bugs get you down.

I said, “That’s really interesting. Please don’t invite me to your meetings.” I’m going to say no to this. Like if you involve me in this, I’m going to get uncomfortable and say, “no, don’t involve me.” Because I feel like, I’ve always felt that we know our cost per lead and we should be creative. If you can do something and generate value at the existing cost per lead or cost per meeting, do it. If you can rent a sign truck and a bunch of Volkswagens and generate new leads at roughly $90 in MQL. Cool, you should do that. And I shouldn’t step in your way. Have fun at work. But I’m super uncomfortable with this idea. 

So I stepped away and the team put together this program and again, the sign truck is way bigger than you think it’s going to be. I got to this parking lot at a Trader Joe’s in Oakland and the sign truck was waiting for us and it’s huge. And we found a driving club, a Volkswagen bug driving club. So these lovely people brought their whole family. So you got this Volkswagen filled with a family of people and there’s 15 of them. I think we were paying for gas and their lunch, that’s all that they asked for. 

Does a piece of content have to be good or does a headline have to be good? Share on X

We spent the day driving around the Embarcadero with cameras and a drone. We hired a guy with an illegal drone and got thrown out of the conference that we were at. The conference would no longer let us go through the parking lot. And the call to action on the sign truck was, “don’t let bugs get you down.” If you go to that, there was a landing page and a microsite and all the conversion tactics. 

It failed miserably. Nobody followed the link. And it’s hard to measure awareness like that because I don’t really know who saw that and the impact that it had. The silver lining on that campaign was, yeah, it failed. It was miserable from a performance standpoint. But while we were out in San Francisco doing this, we also drove the parade around to all of our local prospects, so Williams Sonoma was there, I think, Levi’s Jeans. There were a number of companies in that area and we went and surrounded the building, paraded around 10, 15 times to the point where employees of these companies came outside to see this. 

We’re filming the whole thing with drones and cameras and set up and like filming the people. And then we use that footage to create individual prospect, personalized video that we then send as an ABM campaign. That was effective, good and bad. A number of customers and prospects just thought it was really cool and they remembered it. Like, “oh, I remember that day. I thought that was really neat.” Some people were like, “we didn’t like that you were filming us.” Okay, fair. Fair enough. Gotcha. But turning that, I’ll say failed campaign, into something valuable and in the end generating the cost per lead that we needed to.

Okay, so being able to take what, from a conversion point of view, might have not been the most successful campaign, but then to recycle that content. I also think, could you argue that it was a successful awareness campaign?

You could argue that. I don’t know that. A lot of people saw it, but how many of those people have anything to do with what we do? Like people at the conference were inside the exhibit hall, so unless you came outside to see what the ruckus was, you might have missed what we were up to. People in the street, yeah, it’s downtown San Francisco in 2017. Odds are we’re seeing prospects, but I don’t know that.

I think the real take away though, from a content program standpoint is, it’s okay to try things. It was super uncomfortable for me. I didn’t think it was a safe idea. I’m not going to say I didn’t think it was a good idea. It was probably a good idea. I just didn’t think it was safe, it didn’t appear to be a home run to me. It was uncomfortable. It was outside of my comfort zone. And it would have been very easy for me to say, “oh yeah, that’s a no, we’re not going to do that.”

But we did it, and it’s a story that we all still tell today. When we did the ABM, we were shipping out boxes, because that was in that window of time between 2015 and 2020 when direct mail worked really well. The boxes were custom made, covered with pictures of cartoon bugs. You open the box, and it’s gummy worms and little coasters made out of amber filled with scorpions. The whole thing was bug-oriented. Again, not a safe home run idea, but it’s okay to try.

And definitely a creative one, a fun one to, like you said, try, give it a go. But I would like to fast forward now to your current role, or should I say roles, as both the CMO and CPO — being Chief Pipeline Officer. Firstly, could you give a bit of context about why you’ve broadened your scope of work, and then secondly, what role content plays for both the CMO and CPO.

I say it’s interesting, I don’t know if it is, but I guess if you say it, it makes it interesting that there aren’t a lot of identifiable pipeline leaders. It’s a thing that sales care about, it’s a thing that marketing cares about, but the ownership of pipeline holistically is different. That’s a thing that we do that I don’t see a lot of companies do. 

So we can see that this perfect piece of content, in our mind, didn't perform. Why didn't it perform? Tie the performance back to the creation. Share on X

We do it because it allows us to better understand the future health of the business than most of the other companies that I talk to on a regular basis. The visibility that we have into the creation progression, Q plus one, Q plus two, help us to understand what we need to do today. Because when you look at marketing and marketing’s relationship with sales, this quarter that we’re in right now, we’re in Q3 of 2023, there’s not a lot I can do for a seller to help them have a good Q3, that ship’s sailed. They’re at this point on their own closing business that has been created long ago based on your sales cycle. 

So our marketing organization has to be focused on that future and for us, we’re looking at somewhere short of a 12 month, a 10 month, new business sales cycle and maybe 120-day expansion sales cycle. So thinking in those two channels, what can I do today to get us to a new opportunity right now that’s going to close in that 10-month period? So that thinking, working hand in hand between the sales organization and the marketing organization is ensuring that meetings are happening today that are turning into early stage opportunities next week that are being dated 10 months out on a constant basis. So that at any given moment, we can look at each individual seller and identify where their gaps are and how to fix those gaps and make sure that they’re going to have successful quarters going forward. 

The reason in my mind that it works that it’s me that owns that function, it’s because I have a lot of those levers. So we go back and we look at that first indicator of what are the meetings that you’re getting from your BDR, from yourself? What’s creating those meetings? What campaigns are we running that are creating those? And does that tie to the product approach that we have at this given moment? 

We’ve just moved into a world of positioning our total platform, which is inclusive of generative AI. So it’s important that the content that we’re using all the way back to the top of the funnel is telling that message that we’re building pipeline for the product that’s going to drop in the coming quarter. I think that because I have the visibility into this creation area and what’s fueling the creation, I have the levers to be able to impact that. So if we’re right now creating opportunities only in the area of inclusive language management, for instance, that’s an indicator that we’re not gonna get any benefit from the product that we’re launching next month. So if that’s the case, how do we go back and create the correct content? 

I mean, transparency here. Charlotte is one of the people that runs our content organization. So I’m back to you saying “what are we doing about including generative AI in our overall platform message and building content that supports that message?” Because that’s the message we need to be telling in the marketplace to generate the leads right now that are gonna create the opportunities for when the product is available. That’s how all of this ties together and gets us to a point where we’re sitting prepared to sell the product that we’re bringing to market.

Absolutely! Using content to build to the demand is often underestimated, but definitely something that we prioritize at Acrolinx. 

I’d really love to touch upon the generative AI piece. Obviously, generative AI is an extremely hot topic, but I’m more keen to find out what’s going on at Acrolinx. You mentioned a product offering, but also internally, how are we approaching generative AI use within our content creation workflow?

Generative AI, it’s a huge topic. Everybody wants to talk about it. It’s very exciting. It’s going to either save us all or kill us all. Take your pick and the news that you read and follow. 

Hopefully it doesn’t kill me!

I don’t think it will. And that’s the thing, is that it has a place and the problems right now, and you know who will tell you the problem’s better than anybody else is generative AI. Go ask ChatGPT what the problems are with the use of ChatGPT, and it’s very open with you. And there are security issues, and there’s hallucination issues, and there’s a large language model poisoning issues, and misinformation and disinformation. All kinds of things that make this an InfoSec and IT nightmare. And we’re being told, the news is telling us, that this is the future, and we need to be using this, and we need to be adopting this. You go on LinkedIn, and every other post is the “500 things that you need to ask ChatGPT.”

You're going to tune your own model? Cool, what are you going to use? High quality, valuable content? How do you know what content? Is it a judgment call? Is it pulling individual documents that you personally like subjectively? And that's where… Share on X

Then I do a survey on my LinkedIn: Has your IT department told you not to use this or block the use of it? It’s interesting, small company people that answered it said, no, of course not. Every large company person that answered it said yes. So what then? That’s where we come in. We’ve been involved in AI and NLP for enterprise customers for 20 years. And that’s what we continue to do. We’re making it possible to create risk-free, on-brand, and impactful content. 

So what does that mean? It means creating language models that are yours, that don’t share your information, that don’t help train a model, that are pulling content that’s your best content and allowing you to build content from that. It’s leveraging all the guidelines that are already in our platform on how your company creates content, so that everything, whether it’s human created or computer created, comes through and looks like your content, follows all the guidelines that you’ve built to be able to deliver content to your audiences. It’s measuring that content, both in production, to see how that content is built and aligns with your guidelines, and post-production, to see how it’s performing so that you can go back and iterate and get closer and closer to your audiences. 

This concept of content governance for a generative AI world. I don’t want to say that we’re the only game in town. But you can. I mean, we are, and we’ve been talking about content governance for a very long time, and it’s a hard road circa 2015 to 20, because people don’t really understand why content governance matters. We’re creating content, we have to create a lot of it, we have to create it in a hurry, and the idea of creating a solid governance model around it seems heavy-handed. 

But now, all of a sudden, just the statement of “you’re going to create new content from your best content.” Cool, so what are you going to do? You’re going to tune your own model, right? Cool, what are you going to use? High quality, valuable content? How do you know? How do you know what content? How do you know? Is it a judgment call? Is it you’re going to pull individual documents that you personally like subjectively? And that’s where Acrolinx comes in and the ability to create objectively good content collections that you can use to tune models to create new content. It’s the only thing like it. It’s the only solution that solves this problem and that’s something that our customers deal with every day.

Yes and Acrolinx’s ability to bring in performance data too, to compare to those quality metrics. I think that’s also something that’s quite exciting when you’re training or tuning your generative AI model.

Well, it is because it takes the guesswork out of it. Even the idea of creating content creation guidelines. It’s a thing that, when I got to Acrolinx, the first thing that I did was build our tone of voice. So I did a bunch of marketing things like we’re witty and wise, but not sarcastic or arrogant or whatever. And it just sounds like marketing bullshit and you’re like, okay, that’s what you do. That’s why they hire people like you to make this garbage up and whatever. 

But we then took that and made it actionable. So what does it mean? What does it mean to do that? It means we’re not going to use technology buzzwords. We’re not going to be super formal. We’re going to be more conversational. We’re going to make the conversation clear and concise. We have a terminology group in the business that discusses the words that we’re going to use, we’re very specific about how we say things. And that becomes our tone of voice. 

Go ask ChatGPT what the problems are with the use of ChatGPT, and it's very open with you. There are security issues, and there's hallucination issues, and there's a large language model poisoning issues, and misinformation and disinformation. Share on X

Here’s the thing, I guessed. We made it up. And if you look at the original slide deck from 2017, if you knew the players at the time, and it was me and Andrew Bredencamp, the original founder, our tone of voice model was a very solid combination of the two of us. So you’d look at it and be like, oh, I get it, those two guys did this. So not only did we guess, we’re also arrogant, and we made it about us. So that’s fine, and that’s what people do. 

That’s what organizations do is they think that they know what their audience wants to hear. They’ve done some audience modeling. They know the personas they’re selling to and they build it and they put it out in the world. And a number of things could happen, but in the case of ours, we put it out in the world with our marketing and sales content and it was well received. Our audience liked the way that we communicated and it helped to create engagement. Cool. 

So, an Acrolinx Score of 80 or better means that we’re going to have engaging content for our audience, the personas that we’re communicating with. What happens when that’s not true anymore? What happens when an Acrolinx Score of 100 means it’s exactly what I wanted. It’s clear, concise, on brand, on terminology. It’s using all of the guidelines that create the content that we want. And it’s perfect. And I put it out in the world and it doesn’t perform. What then? Does that mean Acrolinx is broken? No, it means we guessed wrong. 

So what we’ve been able to do is like you said, tie that post-production analytics to the way that we’re creating content. So we can see that this perfect piece of content in our mind didn’t perform. Why didn’t it perform? Tie the performance back to the creation. What about this content kept it from performing? Show me content that’s performing. No, okay, here you go. Show me why, show me all the components, the guidelines that were used, the level of clarity, the terminology, the tone of voice, the consistency, show me everything. 

Now take that model, overlay it over this thing that I thought would perform, and see if that performs better. And that type of A-B testing, that optimization, does things like more time on page, more reading, more conversion, more lead creation, and what most people are trying to do with this content, get more business. Either more sales, more product adoption, better product usage, better customer engagement, all the things that we’re trying to release content to do.

So say you’ve got your perfect data set of content, you’ve governed your content, you’re inputting it into your new generative AI model. And I know you’ve done that as well for Acrolinx. What challenges are presented there? Because from what I’m reading about generative AI, everybody’s saying train it on your own content. But nobody’s progressed past that point of acknowledging it needs to be trained on its own content. They’re not actually doing it because it’s still so new. It’s still so fresh.

So from your perspective, what’s been challenging about doing that?

Well, so the promise of generative AI, the value, the overriding value is you can create a lot more content. So we’re hearing, and I’ll personalize this, “fire people like Charlotte because computers can write content.” Cool, and not only can they write content, but they can write 20 times more than you can. 100 times more than you can. I can run them 24 hours a day and so much more content. 

The problem is you’ve moved the problem. So now you have all of this content and nobody to run editorial on any of this content. So even the best models can still have spurious correlations of data. They can create things that don’t make sense. I saw something about style over substance. ChatGPT, just based on the style it provides, makes you think that it’s right and real. And so at first glance, you’re like, “wow, this sounds really smart.” It’s wrong, but it sounds really smart. 

So the idea of having that editorial process, whether completely automated or partially automated, to be able to find quality content that comes out of this. Whatever you think you’re generating, 20X, you’re not going to get 20X. You’re going to get some percentage more. Some of it’s going to be garbage and you need to know the difference and there needs to be gatekeepers. Again, both through automation and automated content gates, and actual people that know your business and know your audiences and will allow you to be safe in this. Because as fun as it is, it’s dangerous. Like it’s embarrassing.

Using content to build to the demand is often underestimated. Share on X

And I wouldn’t say that quantity for quantity sake is worth it either. The digital marketplace is crowded, overcrowded with content. Everyone’s trying to grab your attention. So more isn’t always more. It’s about the quality, the impactful, or effective nature of that content. 

That brings me on to my provocative statement, even though I’m not having a provocative statement today. We’re turning the tables on you, your favorite question. What is, Chris, your provocative content statement of the day?

Sometimes it doesn’t even matter what the content is.

Let’s unpack. Doesn’t matter what it is? Do you mean by content type or how it’s distributed? What are we talking about here?

Again, this is a provocative statement. I don’t know if I think this, but I’m just throwing it out to have a provocative conversation. What’s the point? What are you trying to do? I have this conversation about webinars a lot, and it’s how important is the actual webinar? The point of this is that we’re generating leads. We’re trying to get meetings to have conversations. And so does that make the webinar the bait? And is clickbait enough, right? Does a piece of content have to be good or does a headline have to be good? What’s the point? 

I think that you would argue that my provocative statement is garbage because if you actually then start reading the content and you don’t have a good content experience, that’s going to hurt your overall experience with the people that you’re trying to communicate with and build a relationship with. But the jaded side of me says, we’ll fix that on the phone. 

What if they don’t get on the phone? What if they don’t make it to the phone because they’re disenchanted?

Then maybe your clickbait wasn’t good enough. Maybe your 10 reasons why content governance is important in 2023 wasn’t good enough to drive people to click on your ad.

Maybe. But I do think content represents the brand, a company, where people can’t. So if somebody stumbles upon your brand and their one interaction with you is clickbait, they’re going to be disappointed. They’re going to click through and think, I just wasted five minutes, I should have not bothered. I think we should always be providing value at every content step, but maybe that’s a side conversation.

Hey, and if you’re listening to the show and you’re going to be on it shortly, this is what I mean by provocative statement of the day. The host is fighting with me on this. This is exactly what I want. This is what I’m aiming for.

Well, I have to say thanks so much, Chris, for letting us turn the tables on you and interview you for a one-off special episode. Hopefully we can do it again, maybe with some questions from your employees next time. It might be fun. 

Oh my god, I don’t know. Okay, thank you, great job.

Thanks, Chris. Bye bye.

Important links

• Christopher Willis – LinkedIn
• Acrolinx
• WordBirds – LinkedIn
• WordBirds – Home page