Here’s the scene: conference room at The Ken, first product review after launch. We just launched semantic search and recommendations, our AI-powered products. Readers had been telling us for a while that the search bar was broken and was definitely not working the way it should. They were right, it was overdue for a fix. But outside this room, nobody really knew about it. So I said I wanted to write a blog about it. Not about the metrics or a mundane product update, but something about the thinking behind it. 

“Why are we talking about this? Is it that important now?” 

My colleague mentioned this in that meeting, and I couldn’t disagree right off the bat. Functional search bars have been around for a while. Netflix and Amazon have been recommending more of their content since the early 2000s, way before I even started using the internet. 

“Huh, so why are we doing this?” I thought to myself, two weeks into my new role as a product designer at The Ken. 

Here’s what puzzled me. Search and recommendations are arguably among the most important features a publication can invest in. They influence, and even shape, how a reader moves through a website, how they discover more of their own content. Yet across the industry, they’re consistently not as good as they can be.

The technology to do this well has existed for years, so why haven’t news websites fixed it? Why invest everywhere else but here? The answer, it turns out, lies in something larger than the technology or infrastructure itself. It’s about the incentives.

Up until that moment, I had been auditing our website. Exploring it with curiosity, like I was on a small quest. I wanted to understand the essence of the stories at The Ken—what we write a lot about, how we write it, and what makes journalism so… Kenesque, for the lack of a better word.

And so, I began searching.

I typed “2025 stories by Arundhati Ramanathan” into our search bar. The results came back partially accurate and mostly… confused. The results showed stories that weren’t written by her and content in which she was quoted but didn’t author herself. I squinted at the screen and remembered thinking, “That’s odd. I could swear I’d seen a couple of her stories surfacing on Google just the other day.”

So I did what everyone else does. I opened a new tab, went to Google, and searched for the same thing.

Google found them instantly.

Sit with that for a second. Searching for something outside your own website gives you more accurate results than searching for it on your own website. Your own stories, written by your own journalist, published on your own platform, and you need a third party to help you find them. Isn’t that strange?

I poked and prodded around the websites of other media organisations. Most of them had mediocre search bars, and some had infinite scrolls of recommended content at the bottom. Hey, if you can’t get them to click, you can at least get them to scroll. 

Techcrunch's scroll of recommended stories

Techcrunch's scroll of recommended stories

The problem looked solved, or at least solved enough. Technologically speaking, it is. 

I was mistaken, because all this time I was comparing apples and oranges. And it took me a few weeks to understand why. The point was something else entirely, and it had very little to do with technology.

Let’s talk about incentives.

Those other websites weren’t trying to do what we were trying to do, and they aren’t even playing the same game. Their search bars were fine because internal search bars didn’t really matter to them. Their recommendation sections were fine because more recommendations for their own content didn’t really matter to them either. 

Sure, they care in the abstract sense. But the actual incentive structure? It’s built entirely around two moments: getting you in and then, you guessed it, getting you out.

The first moment is search. External search. The question most publishers need answered isn’t “how do we help readers find something great once they’re here”. It’s just “how do we get them here”. The internal search bar on most news sites is almost an afterthought. 

And why wouldn’t it be? The reader’s journey, from the publisher’s perspective, starts and ends with Google.

Picture this, you search for a story you want to read about, Google shows you 10 results with pages as such, and then you click on it. You’re on the page now, reading or more likely scanning, and there’s a display banner or one of those “recommended stories” blocks at the bottom that are ads or links to external articles in a trenchcoat.

Remember Taboola and Outbrain? You click on the ad, and the transaction is basically complete. The publisher gets paid when you click one of those. When you click out. Not when you read more of their stuff. 

This is a perfectly rational response to the incentives. If your revenue comes from ads, you optimise for ad clicks. If your traffic comes from Google, you optimise for Google. And if it comes from a high click-through rate from other ads, then you optimise for that.

Let me tell you why this is interesting to me.

Since The Ken is a subscription platform, we charge people money to read our stories. This means our entire business depends on a thing that most of the publishing industry treats as optional: the reader actually finding value.

This flips the narrative on our incentive structure.

When someone lands on our website, we don’t want them to leave. We want them to discover more and see if they find a story of value that makes them want to subscribe. Every story they don’t discover is a missed chance to convince them that a subscription to The Ken is worth paying for. 

We want them to stay, read another story, and then another story, and eventually think, “Wow, this is pretty good, maybe I should subscribe.” We are, in a sense, in the business of making people want to discover value, not just click in and click out.

Think of it this way: someone types a topic into our search bar, and they’re basically asking: “Do you have any content that’s worth my time?” If the answer is “no” or “here are some vaguely related results that don’t really match what you asked”, then we’ve just lost a potential subscriber. We just couldn’t surface what they wanted as it may have been hidden in our archive. That’s a terrible feeling.

There’s another practical reason here, too. We are a paywalled platform, which means a chunk of what we publish is not accessible to external search engines. On top of that, AI overviews came along in late 2022, summarising answers directly on the results page, eating into the redirection traffic that was previously directed by search engines. 

And recommendations? On an ad-supported site, the “you might also like” section is someone else’s revenue stream. Yes, Taboola, Outbrain, and platforms that serve the same function.

On our site, it’s the single best opportunity we have to show a reader more of what we do. We want every recommendation to make someone think, “Oh, I didn’t know they covered that.“

So we have a massive incentive to make both of these things actually good. Good enough that someone searches for a topic and the results genuinely surprise them with how relevant they are.

For a long time, though, we weren’t great at this. Our search was keyword-based, which is fine if you knew exactly what you were looking for and remembered the exact words we used in the headline. Our personalised recommendations were kind of… non-existent. The ambition was there, but the tools weren’t.

And then AI made it a bit easier.

Semantic Vector Search

We are used to keyword-match searches.

Type in “Rapido”, and if you’re feeling lucky, you get the results containing Rapido.

Keyword search matches words. That’s it. That’s the whole thing. You type words, and it finds documents with those words. If you search for “story on Byju’s”, and we have a brilliant story that was published years ago, you will most likely not find it.

The search engine doesn’t know that, because it doesn’t understand what anything is about. It does not understand intent. It just matches strings.

Semantic vector search is different. What you do is take every story you’ve ever published and convert it into a vector, which is essentially a long list of numbers that represents what the story is about in some high-dimensional space. Humans can’t visualise it but math can navigate through. When someone performs a search, their query is converted into the same kind of vector, and then you look for stories that are closest in that space.

The result is that “show me 10 top stories on AI” now matches with our most read stories on AI because they’re neighbours in meaning-space, even though they share zero words.

The first time we tested it, we saw considerably improved results. 

For a subscription-based business, this matters a lot because, in essence, every good search result is a story that a potential subscriber might read, and every bad search result is a door that stays closed.

 We did not want that door to have locks on it.

Hybrid approach to recommendations

The first method is content-based filtering. We all have that one friend who knows exactly what kind of restaurants, movies, and even stories you like and keeps handing you more of the same. You just read a deep dive into D2C brands? Here are three more stories in that realm. Recommending the most tailored content at all times.

The second method is collaborative filtering. This is a different kind of friend. This is the one who surprises you knowing that you would never search for something like this on your own.

You need both friends. The first one keeps you well-fed but never pushes you anywhere new. The second one surprises you, but really needs to get to know you first. Together, they give you recommendations that are both relevant and unexpected.

But neither of these is uber-new technology. What’s new, or at least new-ish, is that it’s practical enough for our team to implement this right now.

That’s also where my role comes in. A few months ago, I joined The Ken as a product designer, fascinated by creating digital experiences and by writing about them too. Figuring out the best UX practices for introducing features like these on a platform like ours has been genuinely fun. As this was one of my first few projects, there were a lot of brainstorming sessions, rough sketches, design iterations, and a whole lot of “Wait…what if we did this instead?” moments. 

But what’s been far more interesting is understanding the why behind what we do. Incentive structures are the very reason certain things get built, and others don’t. Why did we, and everyone else, leave search and recommendations half-broken or barely built for years? 

We did this because we have to. Other non-subscription and ad-supported sites could do all of this, too, I suppose. They just don’t have much reason to.

We want to keep you around. And now, we’re a little bit better at showing you why you should stay. There’s a lot more we are building as The Ken’s product team, and we will be sharing more of it with you as you go. Consider this a start to that conversation.

Try poking around, search for something you’re curious about, see what the recommendations surface, and tell me what you liked, loved, found weird, or absolutely didn’t like. You can find them on The Ken'sThe Ken homepage, on every story page, and on the Listen page on the app. 

© The Ken