Strengthen your foundations with technical SEO and user experience

Iva says: “Technical SEO and user experience will remain the foundation for websites, even when optimizing for AI chatbots.”

Why do AI chatbots like user experience?

“To start with, crawlability has been a topic in SEO for years and years. It's now gaining even more importance because of the way that chatbots crawl websites. They crawl a bit differently.

We're used to Googlebot and other search engine bots adapting to what the websites are serving, and adapting to crawling them. Chatbots are not exactly the same. A lot of studies and research confirm that many of the chatbots do not crawl JavaScript, for example. So, a lot of websites that have been using JavaScript, that Googlebot has been able to crawl, are now not going to be as accessible to chatbots. GPTBot, ClaudeBot, and even Perplexity don't crawl JavaScript.

Aside from that, the bots crawl websites a little more simply, and it's really important to structure your website properly for that, so that the chatbots can access your link. Also, make sure all the links are there, avoid having 404s and redirects, have a pure structure of the website, pure links, and make sure the link juice is going on.

Even before, websites that used JavaScript were not rendered as much. A long time ago, using JavaScript, you could hide links and do spammy stuff. Later on, Googlebot developed a way to make sure your JavaScript is crawled properly, and Martin Splitt has talked about it many times.

However, it is frustrating for developers who are used to using JavaScript in frameworks like React and Vue to create their websites, especially in SaaS businesses. Now, the way that the bots will render it is a bit different. It's going to be a bit difficult for developers and SEOs to find common ground in that.”

Is it worth hanging on for bots to understand JavaScript and more complex technologies, or should SEOs make their websites simpler for them?

“I don't think you have to make them simple, per se. You just have to structure them properly. You can use server-side rendering to optimize the JavaScript rendering.

Maybe it's worth hanging on, but who knows how and when it will develop. It’s all about properly setting your priorities. If your website is built fully in JavaScript, then maybe you don’t want to compromise that. However, for businesses or websites that don’t use JavaScript that much (like small businesses or e-com shops), it might be worth having a chat with developers about how to make it better for bots to crawl it.”

Does the importance of page experience and performance mean that Core Web Vitals are also key for AI search engines?

“I believe so. Over the years, we’ve talked about Core Web Vitals. It started with just PageSpeed and Lighthouse, then moved on to Core Web Vitals, and we've even gotten a new metric in recent years. Core Web Vitals are always going to be important, and Googlebot is now crawling for AI overviews, Gemini, and regular search results. Googlebot has always taken Core Web Vitals into account.

I am yet to read up on how other bots do it, but I believe they will use it, because bad loading speeds and waiting for the website to load are going to take a while, even for bots. It's going to use a lot of resources – especially if we're talking about ethical uses of AI.

I do believe that it will be important for AI overviews and chatbots as well.”

Is there anything different that AI search engine bots are looking for in terms of web page performance?

“I'm honestly not sure. I don't think so, but then again, it's still rolling out.

We've just recently got GPT-5, while we're recording this, and it's something to be explored. Maybe it does different things, maybe it doesn't. Claude is also evolving, and there are a lot of bots rising every day. Somebody always develops a new tool for that.

It remains to be seen. I would love to give a certain answer, but it's not a territory that I've personally dived into as much.”

Why would an AI search engine bot have issues with the experience a user has on a page?

“That is not necessarily for the bots. I put that in my tip because it is important to take into account when optimizing websites.

A certain amount of user experience goes into it from the bot side, but most importantly, now that websites are losing clicks because of AI overviews, user experience needs to happen. It needed to happen for a while now. It's something that should have been a priority forever.

Now, though, we're going to get so many fewer clicks on the website, so a lot fewer users are coming to the website, and you really need to keep them there. Plus, in my opinion, the website loading speed is part of the user experience. We're coming back to the Core Web Vitals. That will be important to take care of for bots.

I believe that user experience will also impact how the chatbots crawl, but it mainly refers to how users will interact with it as well. It's on both sides.”

Are there any key elements that typical websites have to improve on?

“I've worked with a lot of websites in different industries and, from what I've seen, most of them have problems rendering the largest piece of content (Largest Contentful Paint), which is usually the hero part of a website.

That's where all the heavy copywriting, video, and image game goes, because it's the first thing the user sees – and it's important. Whenever I do an audit on a website, that is always what shows up as having trouble loading.

Another segment is the Time to Interactive. Websites are optimized, have their content loading fast, and maybe even take care of the largest content piece, but then the Time to Interactive takes a while. Out of the metrics, those are two of the most common and important ones I've seen people neglect.

It's not easy, especially if we're talking about WordPress websites that use builders like Elementor or Avada. It's not that easy to optimize the heavy content, I get it, but it really should be optimized. You also need to focus on the time it takes for a user to interact with the website because websites that want to sell fast and have users click on ‘buy’ need to make sure that they can click ‘buy’ immediately.”

Do bots for AI search engines care about mobile?

“Right now, I'm not sure. I don't believe so (except for Googlebot, because it has the Smartphone Bot). I don't think the others are doing that.

However, I've always said, don't neglect mobile because users are there. It comes back to the user experience part, which influences everything. Even if the bots do not crawl via mobile, you really should keep it mobile-friendly.”

What structured data and schema tend to be key?

“There hasn't been definitive proof that structured data is used by chatbots but, in my opinion, adding schema and having structured data on your website will help. It helped before, with Googlebot. It's going to help now. It's part of the code that gives an abbreviation of what your website is.

I have personally implemented coded schema. I've seen how the code looks, and it's a digest of what you want to highlight on your website. Instead of focussing on things like llms.txt (which really have no purpose, as of now), adding schema is such low-hanging fruit. It doesn't take a lot of energy or time, but it is helpful.

Currently, there is a debate in the SEO world. I see people on LinkedIn demanding proof that structured data is a part of chatbots. Nobody has any proof yet, but I honestly believe that it should remain part of a technical SEO strategy because it doesn't hurt, and it’s really nice to have an abbreviation to serve the bots.”

If you have a CMS like WordPress, and you use a decent, modern, trusted theme, is that enough to ensure that your code is relatively efficient?

“It really depends on what you're using. I prefer WordPress when I'm working on websites, and when I'm building them out as well, and I always advocate for using a custom-created WordPress.

WordPress is really nice to use when you custom-code and tailor it, and you’re not limited by a builder. With custom-themed WordPress, you can manage a clean-coded website. I've seen it, I've helped build it, and it's really nice – as a back-end user adding blogs or pages, and as a developer.

If we're talking about using builders, it's difficult to do that. It's not impossible (I've seen some examples). It really depends on how big your website is, what your website includes, and what you want to have – colourful elements, sliders, etc. It is more difficult to maintain a clean code on a builder website, but it's not impossible.

WordPress is widely used, it's open source, and you can build out your own website according to your needs and keep the code clean. Years ago, I came across my first website that had a lot of unnecessary code. It was a big chunk of everything, with a lot of plugins being used as well. If you turn on a slider plugin, the footer fails. If you turn on one element, like a review plugin, the whole website breaks. It's difficult to maintain those websites, and it's not nice for Googlebot, or any other bot, to see that on your website.

I've worked on a project where we took that kind of website, cleaned it up, and it really showed results. It was crawled more, it was ranked more, and it showed up in search results. Before that, there weren't many keywords ranking. After that, it didn't immediately go to the first page, but it started showing something, which meant that it was easier for Googlebot to go through it.

I'm really advocating a custom-coded website that has clean code – or if you prefer using a builder, cleaning that up a bit will help boost your website performance, your user experience, and the way your website is indexed and crawled.”

Is a theme builder like Elementor less efficient than building your own custom theme?

“Elementor is really nice because it lets you build out your website by yourself, even if you’re not tech-savvy and you’re not a developer.

It’s WYSIWYG WordPress. It's a drag-and-drop thing. You have the elements, and you can add an image here, text there, a slider there, etc. You can even add some add-ons if you're missing an FAQ or whatever. You can build it out the way you want, visually.

Also, what you see is what you get. What you see when you build it up is what you get when the users come to it. The problematic part comes with mobile optimization (which is where you need to be a bit tech-savvy) and keeping it clean. It's updated all the time. Every update brings something new or retracts something old, so you have to be in it to keep it updated and keep it performing well.

Honestly, all of those things are much easier if you code a website. If you have a custom-coded website, you don't have to be a developer to maintain it. You can hire someone to build it out and show you the ropes on how to maintain it yourself. If a problem pops up, you can contact the developer, but if it's built up properly, you can get by without a developer for a while.

This is very important because a lot of small businesses don't have the budget to pay a developer. You can just use a one-time payment for them to build out a website. A local business doesn't have as many pages as an enterprise website or e-commerce shop would. It's a worthy investment for any type or size of business.”

How do you ensure that you have a clean and efficient website structure?

“Keep your navigation – menu, footer, etc. – very clean, properly organised, and structured in tiers.

Have the most important links right there, and then move on to a second tier. There's been a lot of talk about this for years. There are a lot of infographics and diagrams showing how you have your homepage, then you dive down into the first tier, the second tier, and the third tier. Keep it all clean and in a proper hierarchy and structure so that, when a user visits, they can easily navigate, and Googlebot can easily find those links and figure out what they are.

A lot of the time, you're going to see internal links being linked as ‘Click here’, ‘Learn more’, ‘See this’, etc. That's not good. You need to click on what it is. ‘Get X service,’ ‘Get Y item’.

When we’re talking about the menu, keep it light. If you have a lot of pages, you can keep them organised by having a drop-down menu with services, then list those out, or categories, then list those out. Keep it really organised.

I'm very organised in my life, and I like to keep the websites I work on organised as well. You want your desk organised. You want your life organised. Why not have your website organised?”

Iva, what's the key takeaway from the tip you shared today?

“Technical SEO and user experience aren't optional, and they haven't been for a while. They're the baseline for optimizing websites and, without them, your content has no chance of surfacing in AI-driven search results or even traditional search results.

If crawlers can't access your website or interpret it efficiently, your audience won't either, and you won't show up for that. Focus on making sure your website is technically set and optimized for the user as well.”

Iva Jovanovic is an SEO Specialist and Conference Organiser. Find out more over at Iva-Jovanovic.com.