Why a bilingual website collapses into one page in Google: one URL, one language

This site has always carried both languages in one page, with a switch deciding which one is shown. To make the Chinese version findable in Google we added a /zh/ URL for every page, changed the title, description and language declaration to Chinese, and pointed the two URLs at each other with hreflang.

Three days later, Search Console's URL inspection reported /zh/ as "duplicate, Google chose a different canonical than the user", and the canonical it chose was the English home page. The services page came back as "crawled, currently not indexed".

Google decides whether two URLs are the same page by comparing content. With both languages in the same HTML, the English page and the Chinese page differed only in a few lines of the head. To Google they were two addresses for one page, so it kept one.

hreflang describes the language relationship between pages whose content differs. It cannot make two pages with the same content be indexed separately. Neither can the language declaration or the title.

The source files stay bilingual. At build time they are split by language: the English URL keeps only the English nodes, the /zh/ URL keeps only the Chinese ones. The language switch changed from toggling what is shown to a link pointing at the other language's URL.

One part is easy to get backwards: do not redirect by browser language. Googlebot renders pages with an English browser language, so a /zh/ page that bounces to English on load becomes a redirect page again. We redirect only after a visitor has clicked the switch and a preference is stored, and the redirect keeps the query string and anchor.

This domain used to host a different product. Twenty-odd old URLs were still being crawled and all returned 404. A 404 means "possibly missing for now", so Google keeps coming back; a 410 means "gone", and removal is faster. The old URLs have no equivalent page on the new site, so they are not redirected to the home page either; unrelated pages redirected to the home page get treated as soft 404s.

The www and non-www hostnames each returned 200, and Search Console treated them as two duplicate pages. www now returns a 301 to the main domain. The server was also answering unknown paths with a zero-byte 404; the branded 404 page had never been wired up. That does not affect indexing, but it affects whoever lands there.