Rank Math’s llms.txt is slow: why and how to fix it

A few days ago, going over the Lighthouse report of one of the sites we maintain, we found that Rank Math SEO‘s llms.txt file was not even getting read.

The warning is terse, "llms.txt does not follow the recommendations", and underneath it is the real cause, because the request times out before receiving anything.

There is nothing wrong with the file, it simply does not arrive in time, and the reason lies in how the plugin generates it.

Why is Rank Math’s llms.txt slow?

The llms module does not write any file: it registers a rewrite rule on init, serves the request on template_redirect and builds the whole document on every visit.

The cost is in add_post_types_data(), inside includes/modules/llms/class-llms-txt.php, which for each entry in the listing runs a do_shortcode() over the complete content before asking Helper::replace_vars() for the excerpt.

Put another way, to get a two-line description out it renders the entire entry.

If you use a page builder, and here we use Avada, each entry is dozens of nested shortcodes that really do run, one after another, multiplied by the number of entries you have in the listing.

In our case, 283 lines of file and 66 KB of output.

The numbers

Each measurement with its setup, because otherwise the numbers mean nothing:

What really changes things is getting out of PHP, because the transient only cuts half the journey and leaves the full WordPress boot in the middle.

The second factor, which is harder to see

Caching the response looks like enough, but it is not, and the reason is not in Rank Math.

Check the headers of your own file with curl -sI https://dominio.ext/llms.txt and see whether vary: Accept-Encoding,User-Agent shows up.

If your server adds Vary: User-Agent, and in our case it is added by the .htaccess itself from Avada‘s compression block, the page cache keeps one copy for each agent family.

Your home page has the mobile copy always warm because it gets visits, but the llms.txt gets none, so when PageSpeed asks for it with the agent of a Moto G it lands on an empty copy, and what it finds is a cache miss, a full WordPress boot and, on top of that, the network and processor throttling that Lighthouse applies on mobile.

That is where the request timeout comes from.

A low-traffic route with Vary: User-Agent is doomed to always be served cold, and this applies to any PHP-generated route that gets no visits, whether feeds, exports or the entry points you have built yourself.

Down to work

The way out is to stop generating the document on every visit and to write it just once in the root, since the !-f condition of WordPress’s rewrite rule makes the server deliver the file straight from disk without going through PHP.

Specifically we have added this small piece of PHP code to the functions.php file of our child theme:

Thanks to it Rank Math carries on building the document as always, but the output is captured with ob_start() on rank_math/llms_txt/before_output and dumped to ABSPATH . 'llms.txt' on rank_math/llms_txt/after_output, which is the moment when the content is already complete.

The write is atomic, because the text goes first to a .tmp file and only when it is whole does a rename() put it in its place, so no visitor ever sees the route half written.

Checking that the output contains a section header is the safety net, so as not to leave a truncated file on disk if something goes wrong ahead of time.

The regeneration is scheduled with wp_schedule_single_event() every time an entry, a term or the plugin’s general options change, with a margin of 300 seconds that groups bursts of saving into a single pass.

And the apg_llms_txt_rebuild event does not rebuild anything on its own, it asks for index.php?llms_txt=1 without blocking the request, lets Rank Math generate the document and the same hook as before dumps it to disk again.

It has been running for months on the APG site without a single incident.

The cache of the .txt files

There is one detail worth looking at before calling this closed, and it is in the .htaccess.

The site’s cache block marks the .txt files with Cache-Control: max-age=2592000, public and also gives them an Expires of a month, so the freshly regenerated llms.txt can carry on being served in its old version long after it has been rewritten.

That does not affect your server —which always delivers the file that is on disk—, but it does affect whatever CDN you have in front and the visitor’s browser, which are not going to ask for it again for thirty days.

If you are in a hurry for the changes to show, take the txt extension out of that rule or lower its max-age, and if you are not in a hurry, leave it as it is, because an llms.txt a couple of days behind does nobody any harm.

Conclusion

Thanks to this small piece of code the llms.txt goes from the long seven seconds of dynamic generation to the 26 milliseconds of a file served from disk, and the Lighthouse warning disappears.

The underlying fix, in any case, ought to come from Rank Math, using the stored excerpt when it exists instead of rendering the whole entry to build a two-line summary.

We hope you find this little tutorial useful and we look forward to your comments and contributions.