Marketplace Listing Sitemaps: Split Files and lastmod

A marketplace with tens of thousands of listings cannot rely on category pages alone to get every item found. A sitemap fills the gap, but only inside fixed limits: Google states (Opens in a new tab) that one sitemap is capped at 50,000 URLs or 50 MB uncompressed, whichever is reached first. Past that point you split the catalog into several files and list them in a sitemap index. The lastmod value belongs to real content changes only. If you run a classifieds platform or look after its technical SEO, the rest comes down to how you cut the files, how you keep dates honest and how you publish without breaking anything. Examples throughout use one product category: bicycle spare parts.

Where a sitemap sits in the discovery chain

Crawlers find listings by following links, and a sitemap is the fallback for pages that links do not reach well. Google's ecommerce site structure page (Opens in a new tab) says it is strongly recommended to link to every product you want indexed, and that a sitemap should be used when linking to all pages is not possible. It also notes that feeds can carry links to pages a crawler would not otherwise find. The same page adds that Google does not look at URL structure to work out how a site is organized, so a tidy folder path in a listing URL does not substitute for real navigation.

For a bicycle parts marketplace the consequence is practical. A new listing for a cassette or a set of brake pads sits on no category page until the seller has been browsed to, and an old listing may be buried deep in paginated results. The sitemap gives crawlers a direct list of those addresses. It does not promise a crawl, an index entry or a ranking. The documentation makes no such promise.

Decision rule: treat the sitemap as a second route, not the only route. A listing that can be reached only through the sitemap is an orphan, and the cause is in your link structure. For the crawl capacity side of the same problem, see the crawl budget guide.

The limits, stated precisely

Both references give the same numbers, and the sitemaps.org protocol (Opens in a new tab) adds the exact byte count. Each file may hold no more than 50,000 URLs and no more than 50 MB (52,428,800 bytes) uncompressed. A sitemap index may list no more than 50,000 sitemaps. Google accepts four formats: XML, RSS 2.0, Atom 1.0 and plain text with one URL per line, and the file-size and URL limits apply to all of them.

Table: Sitemap rules and where each one is written

The limits are published by the vendors and can change, so check the current pages before you hard-code them into a generator.

Which limit hits first

A catalog with short, clean listing paths reaches the URL count before the size. A catalog whose URLs carry long query strings for variants or fitment filters may reach the size first. Practical suggestion: compute both numbers for every file and keep a margin under each. Example scenario: a file sitting just under the 50,000-URL limit will overflow when the next batch of listings arrives.

Compressing the file does not raise the ceiling. Google reads the limit against the uncompressed size, so measure before compression.

Choosing how to split the catalog

The documentation says how large a file may be and does not prescribe how to divide a catalog. That is a design choice, and it is easier to make well when you know what you will do with the files afterwards. Practical suggestion: pick the split key that tells you where to look when something goes wrong.

Table: Four split keys compared

Decision rule: if every category fits inside one file, split by category. If one category outgrows a file, divide only that category by a second key such as creation range. Keep the scheme stable. Renaming or reshuffling files every month destroys the history you would use for diagnosis.

Example scenario: bicycle spare parts

Example scenario: a marketplace sells used and new bicycle spare parts. Its categories are drivetrain (chains, cassettes, chainrings), brakes (pads, rotors, levers), wheels and tires, and cockpit parts. Drivetrain is the largest category and passes 50,000 listings, so it becomes drivetrain-1 and drivetrain-2. Brakes, wheels and cockpit each stay in one file. A fault that shows up only in the drivetrain-2 file tells the team immediately that the newest drivetrain listings are affected, not the catalog as a whole.

Listings that sold out or were taken down leave the files at the next rebuild. What the removed URLs should return, a 404, a 410 or a redirect, is a separate decision. For checking sitemap errors, see the XML sitemap audit and error fixes article.

Building the sitemap index

Google calls the index file optional: you can create one and submit that single file. With several category files, the index becomes the only address you need to submit. The protocol allows up to 50,000 entries in it, which is far more than a marketplace will ordinarily need, but the ceiling is worth an alert in the generator.

Practical suggestion: keep the structure one level deep, with a root index and sitemap files that list URLs directly.

If you give each index entry a lastmod, it should describe the file's content. The protocol principle is that the date is the last modification of the linked resource, not the moment the sitemap was generated.

Naming and stability

File names are the first thing you read during an incident, so they should say something. Practical suggestion: use a fixed prefix, a short category name and a sequence number. Avoid dates in names, since a dated name creates a new address at every run and breaks the submission history. Keep the characters URL-safe and match each name to a row in your own generation log.

When a category is split, leave the first file's name and content scope alone and send new listings to the new file. Otherwise every split reshuffles all the files and nobody can tell which file held which listing last week.

lastmod: a field that has to stay honest

The date field is easy to fill and easy to fill wrongly. Google says it uses the lastmod value if it is consistently and verifiably accurate, and that the value should reflect the date and time of the last significant update to the page. The protocol page says the date must be set to the date the linked page was last modified, not the date the sitemap was generated.

Read together, these statements rule out the habit of stamping every listing with the generation time. If a nightly job writes the same date on every listing, the field stops being verifiable. What Google does with a field that fails the condition is not described in the documentation.

What counts as significant

The documentation gives the phrase "last significant update" without a list. Practical suggestion for a parts listing:

Table: Should lastmod change?

Writing the value

The protocol uses the W3C date-time format and accepts a plain date. Google refers to date and time together. When your system already stores the time, write the full timestamp with a time zone.

Derive the value from a dedicated content-updated field. A general row-modified timestamp also moves when a counter or a background job touches the record, which would push irrelevant changes into the sitemap. A separate timestamp, set only when title, description, price, state or photos change, keeps the sitemap clean and gives your own reports a reliable signal.

Example scenario: the midnight job

Example scenario: a nightly maintenance script recalculates search ranking scores for all listings and, as a side effect, updates each record's modified time. The next morning the sitemap shows the same date on every bicycle part. Fixing the script so it writes to a different column, and pointing the sitemap at the content-updated field, restores meaningful dates at the next generation.

Fields Google ignores

On two tags the Google page is direct: Google ignores priority and changefreq. The protocol page says priority values range from 0.0 to 1.0 with a default of 0.5, and that changefreq is a hint, not a command.

Decision rule: in a catalog that is near the 50 MB limit, leave both fields out. Every record gets shorter, and the file reaches the size limit later. If you have a reason to keep them for another consumer, treat them as optional decoration rather than a ranking lever.

Deciding which listing URLs belong

The documentation does not give a rule set for which URLs to list. It does say that URLs must be absolute and must share protocol and host with the sitemap. What follows is practical suggestion, not Google text.

Checklist: what to include

• One canonical address per listing. If a listing opens at both a parameterized and a clean URL, list one.
• Active listings whose pages still show the item. Remove sold and deleted listings whose addresses no longer carry content.
• The destination of a redirect, not the redirecting address.
• No session, tracking or search-id parameters.
• Filter and sort pages only when you deliberately want them indexed.
• No pages behind a login and none marked noindex, because listing a URL while telling crawlers to keep it out of the index sends two opposite signals.

The aim is to keep the sitemap a list of the pages you want indexed.

Submitting the files and watching them

Google names three ways to submit: the Sitemaps report in Search Console, the Search Console API and a Sitemap line in robots.txt. For Atom or RSS feeds, WebSub can broadcast changes. If your interface is not in English, report and menu names in Search Console may look different; in English the report is called "Sitemaps".

When you use an index, submit only the index address and put only that address in robots.txt. Adding a new category file then requires no further submission step.

What Search Console shows per submitted file is not described in the page that was read, so no promise is made about it. Practical suggestion: build your own generation log with one row per file.

Table: Fields for a generation log

Image and video sitemap extensions are outside the scope of this topic.

Running generation safely

Rebuild only what changed

Regenerating forty files from scratch every night risks fresh dates on untouched categories. Rebuilding only the files whose listings changed keeps the date history steady and shortens the run.

Publish atomically

Write the new file under a temporary name, validate the count, size, encoding and escaping, then rename it into place. If the run fails, the previous good file stays live.

Guard against empty output

A database timeout that produces an empty or tiny file makes the whole category look as if it had vanished from the sitemap. Practical suggestion: compare the new count with the last run and stop publishing when it falls below a threshold that suits your normal churn.

Produce XML with a library

Ampersands in parameterized URLs must be escaped under the protocol. Build the file with an XML writer rather than by concatenating strings, so escaping and encoding are handled for you.

Growing without surprises

Files that were comfortable last year will not stay that way. Let the generator compute two ratios for each file, URL count over 50,000 and size over 50 MB, and trigger a split or an alert when either reaches a threshold you set. A fast-growing category such as drivetrain parts needs a lower threshold than a slow one such as cockpit accessories. The documentation offers no recommended threshold, so base yours on listing inflow.

Example scenario: after a seasonal sale, drivetrain-2 climbs close to the ceiling. The generator opens drivetrain-3, adds it to the index and writes a new row in the log. Months later, as listings sell, drivetrain-1 and drivetrain-2 shrink. Merging them is optional; stable names and a continuous history matter more than a neat file count.

Adding a new category without gaps

A catalog changes shape: a marketplace that started with chains and brake pads adds e-bike batteries or child seats. A new category creates four tasks at once. The generator needs a query for the category, the file gets a name that follows the existing scheme, the index gains a new entry, and the generation log gains a new row. If one of the four is missed, the category's listings are live but missing from the sitemap, and nothing fails loudly.

Practical suggestion: make category creation in the admin tool trigger a checklist item for the sitemap owner, and add a nightly comparison between the list of active categories and the list of files in the index. A category with active listings but no file is an alert, not a log line.

Reviewing the setup on a schedule

Practical suggestion: once a month, open three things side by side: the index, the generation log and the report in Search Console. Check that the number of files in the index equals the number of rows in the log, that no file sits close to a limit, and that no category lost a large share of its URLs since the previous review. This takes a few minutes and catches the silent failures that no single alert covers.

Example scenario: during a monthly review the team sees that the wheels file lost much of its URLs after a deployment changed a status code in the listing table. The generator had treated the new status as inactive. Because the previous file was kept by the empty-output guard only when the count fell to zero, the partial drop passed. The team raises the guard to compare against the last run's count and restores the query.

Troubleshooting order

Table: Symptom and first check

Decision rule: check your own log first, then the file itself, then the report in Search Console. If the file and the log are correct, look at link structure and crawl capacity.

Who owns each decision

Sitemap generation touches several roles, and decisions without an owner drift.

Table: Ownership

Take reserved parts: a reserved cassette may return to sale within hours. Whether it stays listed is a product rule that the documentation does not settle.

Pre-launch checklist

Checklist: before the first publish

• Each file is under 50,000 URLs and under 50 MB uncompressed.
• Files are UTF-8 and special characters are escaped.
• Every URL is absolute and shares protocol and host with its sitemap.
• The index sits at the site root and lists every split file.
• lastmod comes from content changes, not from generation time.
• Priority and changefreq are omitted, or kept on purpose.
• Sold, deleted, redirecting and noindex addresses are excluded.
• The index is submitted and robots.txt holds one Sitemap line.
• A failed run leaves the previous file in place.
• Count and generation time are logged per file.

Mistakes to avoid

Refreshing every date on every run

It breaks the consistent and verifiable condition Google sets for lastmod, and the protocol says the date is not the generation time.

A file cut off silently at the limit

A generator that stops writing at 50,000 without moving on drops the newest listings. Make it roll over to a new file and raise an alert.

Using the sitemap as a substitute for links

Listings missing from category navigation stay weak even if they appear in the sitemap. Google's ecommerce page stresses reachability through browsing.

Mixing hosts

Listings served from a store subdomain do not belong in the main domain's sitemap, because the protocol ties the URLs to the sitemap's host.

Skipping the escape step

An unescaped ampersand makes the XML invalid. Let a library write the file.

Frequently Asked Questions

What is the maximum size of one sitemap file?

Google and sitemaps.org both give 50,000 URLs or 50 MB uncompressed per file. Limits can change, so check the current Google page before you rely on them.

Do I need a sitemap index?

Google describes the index as optional. Once the catalog is split into several files, it lets you submit one address instead of many.

Can lastmod show the generation date?

No. Google wants the last significant update of the page, and wants the value to be consistently and verifiably accurate. The protocol says the date is not when the sitemap was generated.

Should I fill in priority and changefreq?

Google ignores both. Omit them if you want smaller files.

Does a sitemap guarantee that a listing is indexed?

The documentation makes no such promise. Google recommends linking to every product you want indexed and using a sitemap where linking to all pages is not possible.

What happens to sold listings in the sitemap?

Practical suggestion: drop addresses that no longer show the item and keep only working canonical URLs.

Conclusion

A marketplace sitemap rests on three decisions: split files so each one stays under 50,000 URLs and 50 MB, gather them in one index, and write lastmod only when a listing's content really changes. Because Google ignores priority and changefreq, you can leave them out. Add a generation log, an empty-output guard and stable file names, and the setup stays understandable as the catalog grows.

A sitemap helps listings get found, yet it does not replace sound category links or visibility for the listings themselves. To put your listings in front of a wider audience, explore the ad spaces.