A site published in two languages does not have twice the content. It has twice the surface a crawler has to walk, and only one of those halves is usually earning anything.
Indexing gets treated as a formality: publish, submit, wait. It behaves more like a queue with a doorman. Something has to find the address, something has to fetch it, and something else entirely has to decide the result is worth keeping. Those are three separate outcomes, and a page can pass the first two and fail the third without anyone noticing for months.
In this city the queue is longer than it needs to be for a specific and avoidable reason. Companies here routinely run a full Dutch site and a full English one because both audiences are the same audience, and mirroring feels like the safe choice. The cost of that decision does not show up on the invoice. It shows up in how long it takes anything new to be seen.
Doubling the surface without doubling the demand
Take a professional services firm with 380 published pages. Mirrored, that is 760. Add paginated listings, tag archives, a filterable case-study index and the print variants somebody enabled in 2019, and the number a crawler can reach passes two thousand while the number of pages anyone would sensibly search for stays around three hundred.
None of this is fatal on a small site. It becomes expensive at the point where fetching is rationed — and it always is, in the sense that no crawler works through an unbounded list on your schedule. What gets fetched first is a function of how easy the address was to find and how much the rest of the site suggests it matters. A mirror that nobody links to internally and nobody searches for competes for attention with the pages that do both.
- Two trees, one demand curve. Where both language versions serve one bilingual reader, the second copy adds pages to fetch without adding queries to answer.
- Duplication compounds downward. A mirrored section brings its own pagination, its own archives and its own filter combinations along with it.
- Old mirrors rot quietly. The translated half is usually the one that stops being updated first, so the crawler keeps returning to pages nobody maintains.
- Nothing warns you. There is no error state for a page that exists, loads correctly and simply is not worth anyone's attention.
Found, fetched, kept — and why they fail apart
The three stages are worth naming because remedies differ by stage. Discovery is about addresses becoming known: through internal links, through a sitemap, through a submission. Fetching is the visit itself, which can fail on a timeout, a redirect chain or a server error at exactly the wrong moment. Retention is a judgement — the page is fetched, evaluated, and either kept in the index or dropped.
| Stage | What goes wrong | What actually helps |
|---|---|---|
| Discovery | The address is linked from nowhere and sits in no sitemap | Internal links from pages that already get fetched; explicit submission |
| Fetching | Errors, redirect chains, slow responses at the moment of the visit | The per-URL log, read for status rather than for reassurance |
| Retention | The page is thin, or duplicates another page you also publish | Fewer, better pages — nothing technical fixes this one |
| Refresh | An updated page keeps its old fetch date for weeks | Resubmission, and a sitemap whose dates are honest |
The mirrored-site problem sits squarely in the third row, which is the row no submission tool can touch. That is worth being blunt about before describing any tooling at all.
What the hub does, stated plainly
The indexing section of the Semalt operations panel is a tracker with a submission mechanism attached. Addresses go in individually or in bulk, submission runs through the IndexNow API — which GoogleBot and BingBot both consume — and every URL carries a record afterwards: whether a bot came, when, with what status, and with error detail where there was one.
Nothing about that description is exotic, and that is rather the point. The value is in having the submission, the receipt and the history in the same place as the search figures, so a question about one page can be answered without exporting three files and joining them in a spreadsheet.
Submission with a receipt
For anyone who needs to know whether a specific address was actually visited, and when.
- A daily allowance per account. One thousand URLs a day, which is a real constraint on a large mirrored estate and irrelevant on a small one.
- Batches of up to ten thousand. A batch can be handed over in one go; the daily allowance governs how quickly it drains.
- Delivery over IndexNow. The notification goes to GoogleBot and BingBot rather than to a queue only one engine reads.
- A log entry per address. Bot visit with a timestamp, the status returned, and the error detail when something went wrong.
Three counters run while a job is live: submitted, found, failed. They are worth watching in that order, because a rising failure count during a batch usually means something on your side — a server under load, a certificate problem, a redirect loop introduced by last night's release — rather than anything about the pages themselves.
Sitemap jobs and the case for one file per language
Sitemaps can be handed over as an uploaded file or as a URL to fetch, and nested index files are parsed recursively up to three levels deep. A single job takes as many as a thousand sitemaps. Two jobs run at once and up to twenty wait in the queue, which for most estates means the whole inventory can be resubmitted in an afternoon.
Whole inventories in one pass
For estates held together by index files rather than by a single flat list.
- File upload or fetched URL. Either route works, which matters when the sitemap is generated by a system you cannot easily change.
- Nested parsing, three levels. An index pointing at indexes pointing at sitemaps is resolved without flattening it by hand first.
- A thousand sitemaps per job. Enough for a segmented estate where each section maintains its own file.
- Two running, twenty queued. The ceiling exists so a single account cannot monopolise the workers; queued jobs are not lost.
The practical advice for a bilingual site is smaller than it sounds and pays back for years: keep one sitemap per language tree, never one merged file. The moment the trees are separated, the counters answer a question you could not otherwise ask — how much of the Dutch inventory is being visited versus the English inventory, and whether the gap between them is widening.
That reading tends to surprise people. On estates where the second language was added later, the newer tree is often visited far less often despite containing the same number of addresses, simply because almost nothing links to it from anywhere that already gets attention. A submission alone will not repair that; internal links will, and the counters make the improvement visible over the following weeks. The URL tracker Semalt provides keeps the before-and-after in one view, which is what makes the argument to whoever controls the navigation.
What the log tells you, and what it cannot
The log is the least glamorous part of the hub and the part worth opening most often. Each address carries the timestamp of the bot visit, the status returned and, where relevant, the error detail. Read across a batch it produces four recognisable patterns.
Visited, clean status, nothing in results
The technical side worked and the judgement went against you. Usually a thin page, or one that duplicates something you already publish.
- Check whether the other language tree carries the same content
- Consider consolidation rather than resubmission
Errors clustered in one section
Failures grouped by path almost always trace to one template or one route rather than to the individual pages.
- Sort by path before sorting by status
- Reproduce one case by hand before filing anything
No visit at all, weeks on
A discovery problem, not an indexing one. The address is known to your submission and to nothing else on the site.
- Link it from a page that is already fetched regularly
- Check it is actually present in the sitemap you submitted
Visited often, changes ignored
The page is fetched but the updated version does not take. Frequently a caching layer serving the bot something else.
- Compare what the log recorded against what a browser sees
- Check the dates your sitemap is publishing
What the log cannot tell you is why a page was not kept. It records the visit, not the verdict. Anyone claiming to read intent out of a fetch timestamp is guessing, and the honest use of this data is narrower and more useful: it separates the problems you can fix from the ones that need editorial decisions.
A thousand a day, applied to a real-sized estate
The figures here are a worked example rather than a client case. A creative studio in the canal belt runs its own site and eight client sites from one account. Between them: 6,400 addresses, of which roughly 2,900 are the second language version of something already published.
| Scenario | URLs to submit | Days at 1,000 per day | Reading |
|---|---|---|---|
| Everything, mirrored as-is | 6,400 | 6.4 | A full pass takes most of a working week |
| Primary tree only | 3,500 | 3.5 | Half the wait, no loss of anything that ranks |
| Changed pages, monthly | 420 | 0.4 | What the routine should actually look like |
| One client's site, in full | 710 | 0.7 | Comfortably inside a single day's allowance |
| After a site migration | 6,400 | 6.4 | The one case where a full pass is justified |
The third row is the point. Submitting everything every month is a habit, not a strategy, and on a mirrored estate it spends most of the allowance on pages whose duplicate is already indexed. Submitting what changed keeps the allowance free for the days when it matters — a launch, a migration, a section rebuilt from scratch.
Pruning a mirror without losing the audience
Reducing the surface is the only remedy in this article that addresses the cause rather than the symptom, and it is the one people avoid because deleting pages feels like going backwards. The framing that helps: you are not removing content, you are removing addresses that answer no question anyone asked.
- Start with the pages that never received a visit. Months of nothing in the log, no clicks, no impressions. Those are candidates for removal, not for another submission.
- Then the duplicated informational pages. Where one bilingual reader is served by both versions, one strong page usually outperforms two competing ones.
- Keep the commercial pages mirrored. Service and pricing pages are read in the language of the contract, and that is a real reason for both to exist.
- Fix what you keep. Language annotations only help when both versions genuinely exist and point at each other; on a half-finished mirror they add confusion rather than clarity.
Where the two versions must both stay, they should be doing different jobs — different search terms, different depth, different entry points — rather than saying the same thing to the same person twice. This is where indexing work stops being technical and turns into an editorial decision, and it is usually the conversation that produces the most improvement per hour spent. The same reasoning informs how we scope our own technical work.
A rhythm that survives a busy quarter
Most indexing work fails not because the method is wrong but because it is done once, enthusiastically, and never again. A schedule that fits around real work looks roughly like this: changed pages submitted weekly as a small batch, one sitemap job per language tree monthly, the failure counter checked after every release, and a full pass reserved for migrations.
The changed-pages batch
Anything published or meaningfully edited since the last run, submitted as one batch. Rarely more than a few dozen addresses.
- Takes minutes once the list is generated
- Leaves the daily allowance almost untouched
The failure counter
Five minutes on the counters catches redirect loops and broken routes while the deployment is still fresh in everyone's memory.
- Compare against the previous run, not against zero
- Cluster failures by path before investigating
The assistant in the My SEO area shortens the tedious part: Stream accepts lists of URLs in a batch rather than one at a time, and a router model decides per question which data blocks to pull — indexing, Search Console, results-page or campaign — loading between none and three of them by relevance. Asking it which addresses in a given section have no recorded visit is faster than assembling the same list by hand, and the answer arrives as a stream rather than as a report you wait for.
Reporting follows the same logic as elsewhere in the panel: table exports carry up to ten thousand rows in CSV or JSON, the rendered PDF stops at two hundred and fifty because a longer document does not get read. For a monthly indexing review the PDF is usually the wrong format anyway — the working file is what you want, sorted by status and filtered to one tree at a time. Both the tracker and the sitemap jobs it runs feed those exports directly.
Frequently asked questions
How long after submission should a page appear?
There is no guaranteed interval, and anyone quoting one is inventing it. The log will usually show a bot visit within days for a site that is otherwise healthy. Whether the page is then kept is a separate decision made on the page's merits, and it can take considerably longer or never happen at all.
Does submitting a page more often help?
No. Repeated submission of an unchanged page spends your daily allowance and changes nothing about how the page is judged. Submit when something actually changed. The exception is a genuine technical fix, where one resubmission after the fix is deployed is reasonable.
Should the English and Dutch versions be in one sitemap?
Technically it works; practically it wastes the information. Separate files let the counters tell you which tree is being visited and which is falling behind. On a bilingual site that comparison is one of the few indexing readings that leads directly to a decision.
What happens when the daily allowance runs out?
The batch does not fail — it drains more slowly, at a thousand addresses a day until it is finished. A ten thousand URL batch therefore takes about ten days end to end. Plan migrations with that arithmetic in mind rather than expecting a single evening to clear everything.
Is the log a substitute for server logs?
No, and it is not meant to be. It records the visits connected to what you submitted, with status and error detail. Full server logs see every request from everything, which is more complete and considerably more work to read. For most teams the per-URL record answers the question at hand.
The honest summary is short. Discovery problems are worth solving and mostly solvable; retention problems are editorial and no submission mechanism touches them. On a mirrored bilingual site the largest available gain is usually not a faster queue but a shorter one — fewer addresses, each with a reason to exist, split into sitemaps that let you see the two languages apart.
If you want to see how much of your estate has never actually been visited, connect the property, split the trees into separate files and run the first sitemap job from the Semalt dashboard. The gap between the addresses you publish and the ones anything has fetched is usually the most sobering number in the account.
Hulp Nodig met Uw SEO?
Onze Amsterdam SEO-experts kunnen u helpen deze strategieën te implementeren en uw rankings te verbeteren.
Gratis Consult Aanvragen