A competitor's public sitemap is a list of the URLs that site has chosen to announce. Find the file through robots.txt, walk down to its child files, group the URLs by path to count content types, and test the lastmod dates before trusting them. The result is a rough inventory of what the rival publishes and where it invests. It does not show what is indexed or how any page ranks. The examples follow a craft coffee roaster that wants to see how a rival organizes its shop, brewing guides and wholesale pages.
What a competitor's sitemap can and cannot tell you
Google's sitemap overview (Opens in a new tab) describes a sitemap as a file where you provide information about the pages, videos and other files on your site, and says search engines read it to crawl a site more efficiently. It adds that the file can convey when a page was last updated and the alternate language versions of a page. All of that sits at a public address, so you can use it without any special access.
What it can tell you
A sitemap records the site owner's choices. Whoever built it decided which URLs to announce. If the file lists shop pages, origin pages, brewing guides and journal posts, the proportion between them points to where the rival puts its effort.
What it cannot tell you
The overview page says a sitemap does not guarantee that all the items in it will be crawled and indexed. A URL in the file therefore proves nothing about indexing or quality. A URL missing from the file proves little as well: the page says a site of about 500 pages or fewer that is comprehensively linked internally may not need a sitemap at all. Call the result "announced content" and avoid calling it "the whole site".
| Question | The sitemap answers | The sitemap does not answer |
|---|---|---|
| Which URLs are announced | The listed URLs | Whether unlisted pages exist |
| Which content types exist | URL patterns | How good the pages are |
| When pages changed | lastmod, if accurate | What changed |
| Whether pages are indexed | Nothing | Indexing status |
Finding the file
Three routes lead to a sitemap: the robots.txt file, the conventional location at the site root, and an index file that points to child files. Try them in that order.
The Sitemap line in robots.txt
Under the sitemaps protocol (Opens in a new tab), a line such as "Sitemap: http://www.example.com/sitemap.xml" in robots.txt announces where a sitemap lives. The directive is independent of the user-agent line, so its position in the file does not matter, and more than one sitemap can be listed. Google's page on building a sitemap likewise says you can insert the line anywhere in robots.txt and that there is no limit to the number of sitemaps you can include there. Open the rival's robots.txt at the root of the domain and note every line that begins with Sitemap.
Location at the site root
Google's page on building a sitemap (Opens in a new tab) says you can host sitemaps anywhere on your site, but that a sitemap posted at the site root can affect all files on the site, which is where it recommends posting them. The protocol also recommends the root. If robots.txt has no Sitemap line, try a file named sitemap.xml at the root. If nothing turns up, stop looking. The absence of a sitemap is itself a finding.
Formats
The same Google page says it supports the formats defined by the sitemaps protocol, describes XML as versatile, accepts RSS 2.0 and Atom 1.0 feeds, and says a text file should hold nothing but URLs. A text-format sitemap cannot carry lastmod, so for that rival you can analyze URL patterns but not timing.
Scope
Under the protocol, a sitemap can include only URLs at or under its own directory, and every URL must use the same protocol and host as the sitemap file. Google adds that unless you submit a sitemap through Search Console, it affects only the descendants of its parent directory. A rival who runs the blog on a subdomain or publishes sitemaps per folder will have several files, each covering one slice of the site. If the roaster's rival keeps its journal on a subdomain, open that subdomain's robots.txt as well.
Reading the entries
An XML sitemap needs three tags: urlset, url and loc. The loc tag holds the page address, which must begin with the protocol and be under 2,048 characters. Everything else is optional.
| Field | Meaning in the protocol | Use in your analysis |
|---|---|---|
| loc | The page URL | Pattern extraction and counts |
| lastmod | Date of last modification in W3C Datetime format | A hint about update timing |
| changefreq | A hint about change frequency, not a command | Leave out |
| priority | Relative priority within the same site | Leave out |
Why changefreq and priority are skipped
Google's page on building a sitemap says Google ignores priority and changefreq values. The protocol itself says priority only ranks URLs against other URLs on the same site and that changefreq is a hint. Adding these columns to your sheet would only record the rival's own guess, so leave them out of the inventory.
Dates, encoding and escaping
The protocol accepts W3C Datetime for lastmod and allows you to omit the time portion and use YYYY-MM-DD. The file must be UTF-8 encoded, and all data values must be entity-escaped, so an ampersand in a URL appears as "&". Convert escaped characters back before counting, otherwise the same URL can land in two groups.
Image, video and news entries
The overview page notes that a video entry can specify running time, rating and age-appropriateness, an image entry can include the location of images on a page, and a news entry can include an article title and publication date. If the rival's file contains such entries, the site is investing in those formats. A roaster whose rival publishes a separate video sitemap for brewing tutorials can infer that video matters to that rival.
Walking an index file
Large sites publish an index file instead of a single sitemap. Google's page says all formats limit a single sitemap to 50MB uncompressed or 50,000 URLs, and that a larger site must split its sitemap into several. The protocol says an index file can list up to 50,000 sitemaps and can point only to sitemaps on the same site.
Reading the index
Each entry in an index file has a loc and an optional lastmod. Write every child file name into your sheet. Child file names can name the content type, such as products, categories or guides. Open each child and count its URLs.
When a child file will not open
The protocol allows gzip compression. A child file ending in .gz may not open directly in a browser or a spreadsheet. Do not force it. Mark that child "not read" and continue with the others, then say so in the summary.
What the split says about the site
If the rival keeps shop pages, brewing guides and journal posts in separate child files, the site separates its content types on purpose. If a file name does not make the type clear, open the first few URLs and name the type yourself.
Example scenario: a craft coffee roaster reads a rival's paths
Example scenario: the roaster's rival publishes an index with several child files. The roaster groups the URLs by the first path segment. The table lists patterns and how to verify each one, with no counts.
| Path pattern | Probable type | How to verify |
|---|---|---|
| /shop/... | Product pages | Open three: do they show price and add-to-cart |
| /origins/... | Coffee origin pages | Open three: is there original description or only a product copy |
| /brew-guides/... | Instructional pages | Open three: are there steps, tables or images |
| /journal/... | Articles | Open three: dates, authors, topics |
| /wholesale/... | Business offer pages | Open three: is there a form or just text |
Depth under each path
The second and third path segments carry information too. A large number of URLs under /brew-guides/espresso/ against few under /brew-guides/pour-over/ shows where the rival goes deeper. A high URL count is not a quality score, though. Parameter variants of one page can inflate it.
Language prefixes
If paths begin with language codes, count each language separately. The overview page says a sitemap can convey alternate language versions of pages, so a gap between languages shows where the rival publishes more.
Parameters and repeated URLs
When one path appears with several query-string variants, count the unique paths separately from the total URLs. A big gap between the two suggests a possible duplicate, and you should label it that way, not as a verdict.
Putting the roaster's inventory next to the rival's
Once both files are summarized with the same template, a side-by-side table shows where the two sites differ. This table is directional on purpose and carries no counts.
| Content type | The roaster | The rival | Reading |
|---|---|---|---|
| Product pages | Moderate | Moderate | Similar catalog size |
| Origin pages | Few | Broad | The rival documents each coffee in depth |
| Brewing guides | None | Moderate | An instructional gap on the roaster's side |
| Journal posts | Moderate | Few | The roaster publishes more stories |
| Wholesale pages | One | Several | The rival splits wholesale by buyer type |
Reading the origin pages gap
A broad origin folder suggests the rival writes a page for each lot or region. Before treating that as a lead, open three of them. If each page repeats the product description, there is little to learn. If each carries tasting notes, processing details and photos taken by the rival, the pages offer value that the roaster's short product copy does not.
Reading the brewing guide gap
The roaster has no guide folder at all. Instead of copying the rival's topics, list the questions a customer asks after buying coffee: grind size, water temperature, storage. Check which of those the rival answers well and which it skips. The skipped questions are the roaster's opening.
Reading the wholesale gap
Several wholesale pages may mean the rival speaks to cafes, offices and retailers separately. If the roaster has one generic page, it can ask whether a buyer type is poorly served. That question belongs to the offer, not just to content, so note it for the commercial team.
From sitemap rows to page reading
The inventory is a map that tells you which pages to open. After the count, read a handful of pages from each important type.
Choosing sample pages
For each type, pick three pages: one with a recent lastmod, one with an old lastmod, and one chosen without looking at the date. This stops a single page from standing in for the whole type. Keep the chosen URLs in their own column.
What to record on each page
Note four things: the question the page answers, whether its information appears original, whether authorship and dates are shown, and how the page links to others. The same fields work for any rival page comparison you run later.
Keeping it small
Three pages across five types is fifteen pages. That adds about half an hour to the session. Reading more stretches the task into days. Mark the types you skipped as "not read" in the summary.
Testing lastmod before trusting it
A lastmod column looks like a free window into a rival's publishing rhythm. Google's page says Google uses the lastmod value if it is consistently and verifiably accurate, for example by comparing it to the last modification of the page. The protocol says the date should reflect when the page changed, not when the sitemap was generated. Both statements imply that the value can be unreliable on some sites.
Three tests
Practical suggestion: run three checks. First, see whether every URL has the same date and time, which would point to the file's generation time. Second, open three pages and compare their visible dates with the lastmod value. Third, if you have two copies of the file from different days, check that the dates that changed belong to pages that really changed.
| Test | If it passes | If it fails |
|---|---|---|
| Are the dates varied | Per-page values are plausible | Probably a generation date, drop the column |
| Do sample pages agree | Roughly trustworthy | Do not rely on it where the page shows a date and disagrees |
| Do changes between copies match real edits | Reliability rises | Drop the column if all dates moved together |
How coarse to be
Decision rule: if lastmod passes at least two tests, sort pages into two broad groups, "recently active" and "untouched for a long time". Do not read it at day-level precision. If it fails, remove the column and write why in the sheet.
A file with no lastmod
The field is optional. A blank column means "no data", not "no updates".
Building the sheet and the summary table
One spreadsheet is enough. Give each URL a row and use the columns as a working template.
| Column | Content |
|---|---|
| URL | The loc value with escapes undone |
| Source file | Which child sitemap held it |
| Segment 1 and 2 | First and second path folders |
| Type guess | Product, guide, article, offer |
| Verified | Whether a sample page confirmed the type |
| lastmod | As a date, blank if missing |
| Doubt note | Parameter variant, possible duplicate |
The summary
Produce a short summary: URLs per type, the lastmod range, the number of verified samples, and which child files could not be read. Put the rival's name and the date you fetched the file at the top.
Checklist
- Every Sitemap line in robots.txt was noted.
- All child files were opened, or the unreadable ones were marked.
- Escaped characters were converted back before counting.
- Each path pattern was verified with at least three sample pages.
- The lastmod tests were run and the outcome recorded.
- changefreq and priority were left out of the analysis.
- The fetch date appears on the sheet.
Turning the inventory into an opportunity list
An inventory is useful when it answers a question: in which content type does the rival publish more than you, and which topics does it skip. You answer by putting your own inventory in a column next to theirs.
Compare against your own file
Extract your own sitemap with the same template and put counts per type side by side. Types the rival has and you lack, and subfolders where it goes deeper, are opportunity candidates. For judging your own inventory, see the content inventory decision matrix, and for errors in your own file see the XML sitemap audit.
Telling an opportunity from a trap
A rival folder with a large URL count may hold thin or automatically generated pages. Open three before you add the type to your list, and mark the folder "rival inflation" if the pages do not serve a visitor.
| Observation | Reading | Decision |
|---|---|---|
| Rival has a broad brewing guide folder, you have none | Investment in instructional content | Read three guides and list topic candidates |
| Rival has a broad origin section | Possible catalog-led content | Check whether pages differ in substance |
| Rival has a shallow wholesale section | A gap you could fill | Consider a fuller wholesale page |
| All lastmod dates identical | The data cannot be trusted | Drop timing analysis |
Decision rule
Decision rule: add a type to the opportunity list only if sample pages serve a visitor and you can produce better ones. Otherwise record it as skipped, with the reason.
Comparing two copies over time
A sitemap is a snapshot. Two copies from different dates show where the rival is heading, which no single file can.
Method
Put the URL columns of both copies side by side. Mark URLs found only in the newer copy as added and those found only in the older as removed. For URLs present in both, compare lastmod. Group the added URLs by path segment.
Caution on restructures
New URLs may reflect a site redesign, not new content. If the URL structure changed, old and new URLs look different while the pages stay the same. Open a few added and removed URLs before concluding that anything is new.
Schedule
Practical suggestion: quarterly for a competitive topic, twice a year for a calm one.
Boundaries of the method
A sitemap is a public file and reading it is fine. That does not extend to harvesting the rival's pages in bulk.
Read the file, not the site
Limit the work to the sitemap and a handful of sample pages. Do not run automated crawling that breaches a site's terms of use. Copy the file's text into the spreadsheet, and you will not need to fetch page content.
Incomplete by design
Some pages are not in the file. The overview page notes that a small site that is comprehensively linked internally may not need one. Listed URLs also may not be indexed. Describe findings as "announced URLs".
Mind the date
Write down when you fetched the file, and keep the old copy when you take a new one.
A one hour session
Practical suggestion: spend fifteen minutes opening robots.txt, finding the sitemap or index and listing the child files. Spend twenty-five minutes moving the URLs into the sheet, splitting paths and verifying type guesses with three sample pages each. Spend twenty minutes on the lastmod tests, the summary table and the comparison with your own file.
How to present the result
Say "the rival's announced URLs are dominated by this type" instead of "the rival invested in this type". The first sentence keeps the data's limits.
Mistakes in sitemap inventories
Treating lastmod as exact
Do not interpret a date at day level before the three tests pass.
Reading URL count as quality
A large folder is not a good folder. Open sample pages from the heavy types.
Assuming one file is the whole site
If the rival publishes sitemaps per folder or subdomain, the root file shows one part. Check the index file and the robots.txt lines.
Inferring rank from a sitemap
A rival with a long list of announced URLs is not ranking higher for that reason. A sitemap shows announcements, not results.
Frequently Asked Questions
Where do I find a competitor's sitemap?
Under the sitemaps protocol, robots.txt can contain a Sitemap line that announces the location. If there is none, try sitemap.xml at the site root. If nothing exists, note the absence as a finding.
Is every URL in a sitemap indexed?
No. Google's overview page says a sitemap does not guarantee that all the items in it will be crawled and indexed.
Can I trust the lastmod values?
Google uses lastmod if it is consistently and verifiably accurate. Run three checks on the rival's file: whether the dates vary, whether sample pages agree, and whether changes between two copies match real edits.
Should priority and changefreq go into the analysis?
Google's page says Google ignores both values. Leave them out of the inventory.
Does a rival with more announced URLs rank higher?
The documentation has no statement on that. The number of announced URLs says nothing about quality or ranking.
Is crawling a rival's pages in bulk acceptable?
The documentation has no statement on it. Practical suggestion: open the file in a browser and copy its text into a spreadsheet. Bulk crawling can breach a site's terms of use.
Conclusion
A competitor's public sitemap hands you a list of the URLs the site announces. Start at robots.txt, walk the index file down to its children, count content types from path patterns, and run the three lastmod tests before you rely on any date. Read the result as announced content, because a sitemap does not show indexing, quality or ranking.
If the inventory comparison also shows that a rival leads on external references, see the backlink packages.
Post a comment
Please log in to leave a comment. Login