Sitemap issue with Polylang

[saulirajala]

The language version of sitemaps is not accessible via domain.com/lang_slug/sitemap.xml, if “The front page URL contains the language code instead of the page name or page id” -setting is not enabled in Polylang Pro. This settings controls whether the frontpage is located in domain.com/lang_slug/ or in domain.com/lang_slug/page-title/.

If the url of frontpage is domain.com/lang_slug/page-title/ then the sitemap can be found in domain.com/lang_slug/page-title/sitemap.xml. Yet the domain.com/lang_slug/sitemap.xml is advertised in robots.txt:

Sitemap: https://domain.com/sitemap.xml
Sitemap: https://domain.com/lang_slug/sitemap.xml

I tested the issue both with Polylang Pro and Polylang You can replicate the issue by creating a new site with two languages. I have following settings in Polylang

• The language is set from the directory name in pretty permalinks
• Hide URL language information for default language
• Remove /language/ in pretty permalinks

Original issue in wordpress.org: https://wordpress.org/support/topic/sitemap-issue-with-polylang/

[sybrew]

Also, unsetting "Hide URL language information for default language" will cause the robots.txt URLs to add extraneous language prefixes. However, the sitemap endpoints still work on their correct URL.

I'm not sure if these issues are related; further investigation is required.

[nlenkowski]

@sybrew Just ran into this issue myself.

With the same URL configuration as @saulirajala, the following URL format is output to my robots.txt for translated languages, but is not publicly accessible:

Sitemap: https://example.com/es/sitemap.xml

This is the correct URL format, which is publicly accessible:

https://example.com/sitemap.xml?lang=es

Any word on a possible fix?

[sybrew]

Thanks for the added info, @nlenkowski.

This issue is planned to be fixed with the next update, due in the second week of November, but I may need to push it to the one after that due to time constraints. (The "responsive canonical URLs" update had gotten a bit out of control.)

The issue won't impact ranking but may impact the discovery of URLs on large sites. If Google hasn't found the translated versions of the sitemap yet, I recommend manually submitting them: https://support.google.com/webmasters/answer/7451001.

[nlenkowski]

Glad to hear! Thanks for looking into it @sybrew. In the meantime we've our sitemaps manually, as suggested.

[sybrew]

After testing, I concluded that only disabling "Hide URL language information for default language" affects this behavior.

To resolve this issue, we'd have to disable our hijack on robots.txt requests, even when template_redirect never occurred. This proves difficult to do without affecting other parts of the site or otherwise causing a performance downgrade.

But that isn't the right place to resolve this. We must query non-default languages for the robots.txt; the hijack should not be a concern because we must also be able to fetch the correct URLs after template_redirect, even though we don't at the moment.

I'll look at it again in the minor release cycle of TSF v5.1.x because we are already overdue for the major release.