URL Parameters and Tracking Codes: Crawl Control

Copy a product link from the address bar and you often copy more than the page. A campaign tag, a sort choice, a session token or a word typed into the site search box rides along after the question mark. To a person it is the same page. To a crawler it is a different address, and every extra address is something to fetch, compare and decide about. On a small brochure site this barely matters. On a marketplace with many listings, filters and campaign links, it becomes lost crawl effort, split signals and muddy reports.

This guide is for technical SEOs, e-commerce and marketplace developers, and marketers who manage campaign links. It sorts parameters into families, reports what Google's own documentation says about each, offers a decision ladder for picking a control, gives internal search result pages a section of their own and closes with a Search Console check. The general theory of duplicate content is not repeated here; see duplicate content: what it is and how to fix it for that. This piece stays with duplication that comes from parameters, tracking codes and internal search. It contains no search volumes, difficulty scores or ranking-gain figures.

Reading a Parameter URL the Way a Crawler Does

Strings, not pages

After the domain comes the path, which names the page. A question mark then opens the query string, made of key=value pairs joined by ampersands. Your server may read those pairs and change what it shows, or ignore them and return the same page. A crawler cannot know which in advance. If the characters differ, the address differs, and the only way to learn whether the content differs too is to fetch it.

Google's URL structure documentation (Opens in a new tab) states two relevant facts plainly. Google treats URLs as case-sensitive, so if your server serves differently cased paths as one page, choose a single consistent case. And removing parameters that do not change content keeps URLs short and clean. Reordering parameters is not discussed in that document, but since the address is a string, the same parameters in a different order form a different address. That is general engineering reasoning, and it is the reason applications should always emit parameters in one fixed order.

The format Google recommends

The same page recommends an equals sign between key and value and an ampersand between pairs. For several values under one key it suggests a comma or another character that does not conflict with the URL standard, and it lists colon-and-bracket or double-comma notations as not recommended. Readable words, hyphens instead of underscores and percent-encoded non-ASCII characters complete the guidance. None of this is exotic, but a surprising number of home-grown filter systems break the first rule on day one.

Fragments deserve a note. Everything after the hash sign is generally not supported by Google for changing page content, and sites that alter content with JavaScript are pointed to the History API. The canonical documentation adds that a fragment should not be designated as a canonical. A design that keeps tab or filter state only after the hash therefore shows search engines no separate pages, which may be exactly what you want or exactly what you do not, depending on what should be findable.

A one-question test

For any parameter, ask: if this parameter were deleted, would a visitor see different content? If not, it is there for tracking, session handling or presentation, and the only address worth listing is the clean one. If yes, because a different subset of listings appears, the address is a candidate page in its own right and whether it deserves indexing is a product decision rather than a purely technical one. Each rung of the decision ladder further down begins from this question.

A Field Guide to the Parameters You Will Meet

Marketplace URLs tend to accumulate the same handful of parameter families. The grouping below mixes examples from Google's documents with general engineering practice. Parameter names will differ on your site, but the underlying problem of each family stays the same.

Tracking and attribution

Campaign tags, advertising click identifiers and referral markers are appended to links and never change what the page contains. Keys beginning with utm_ are a widespread convention for source, medium and campaign names, and ad platforms attach their own click IDs. Google's canonical documentation uses exactly this situation as its example of duplicate URLs: a product page reachable both with a tracking parameter such as gclid and in its clean form. The URL structure documentation likewise lists referral and click-tracking parameters among the irrelevant parameters that can create large numbers of duplicate URLs.

Sessions

Some older platforms still write a session identifier into the address. Each visitor then receives a private copy of every page URL. Google's URL structure documentation flags session IDs in URLs as a problem and says to use cookies where possible. Cookies bring their own consent and privacy duties, which sit outside this article; here the concern is only what the address looks like.

Sorting and display

Price ascending, newest first, list view versus grid: these reorder the same set of listings. The crawl budget guide names differently sorted versions of the same page as an example of low-value URLs. These variants are rarely a destination in their own right, so the usual responses are a canonical to the default ordering or a crawl block.

Pagination

Page two and page three show different listings, which makes them unlike a sort order. Pointing every page's canonical at page one can hide the listings that only appear deeper in the sequence from discovery. Pagination mechanics are outside this article. What does belong here is stopping the page number from multiplying with sort and filter parameters into a combinatorial explosion.

Filters

Combinable filters such as price range, brand and condition trigger what the URL structure documentation calls additive filtering, producing many near-duplicate listing URLs. That topic is large enough to deserve its own treatment, which you will find in e-commerce filter URLs and faceted navigation. The rest of this article leaves filters aside.

Internal search

Every word a visitor types into the search box can create a new address, and so can every typo. The URL structure documentation lists search results among dynamic URLs you may consider blocking. They get a full section below.

The families side by side

Treat the table as a thinking aid. Each site has its own data and goals, so a call that is right for one marketplace can be wrong for another.

What Google's Documentation Says About Crawling and Duplication

Why consolidate at all

The documentation on specifying a canonical URL (Opens in a new tab) lists four reasons to pick one preferred address: to control which URL appears in search results, to consolidate signals such as links from variant URLs onto the preferred one, to simplify tracking so metrics are not split across addresses, and to save crawl budget so Googlebot spends time on new or updated pages instead of duplicates. It also notes that specifying a canonical is optional. If you say nothing, Google chooses a version itself. The cost of inaction is therefore not an emptied index but a loss of control over which address represents the page.

The reporting angle

The third reason, simplified tracking, is easy to overlook. When one listing is visible under several addresses, analytics can scatter its visits across rows. That observation comes from day-to-day work rather than from Google's text: a stakeholder opens a report, sees one listing split across three paths and cannot tell which row is the real performance. A canonical and consistent internal links shorten that argument.

Who really needs to worry about crawl budget

Google's crawl budget guide (Opens in a new tab) draws the line with concrete thresholds: sites with more than a million unique pages whose content changes about weekly, sites with more than ten thousand pages that change daily, and sites with many URLs sitting in the Discovered - currently not indexed status. The guide itself calls these rough estimates. Everyone else is told to keep the sitemap current and watch the Page indexing report. So for a small shop, parameter control is mostly about order and accurate measurement, while for a very large marketplace it is a genuine resource question. Decide which side you are on using your own page counts.

Among the things that waste crawling, the guide names duplicate content, low-value URLs such as differently sorted versions of a page, soft 404s, long redirect chains and slow or erroring servers. Parameter URLs fall mainly into the first two groups. The recommendations include consolidating duplicate content and using robots.txt to block unimportant URLs that cannot be consolidated. If crawl budget as a concept is new to you, our crawl budget optimization guide covers the foundations separately.

Two cautions worth memorising

The same guide stresses two points. Blocking URLs in robots.txt does not automatically move the freed budget to other pages; Google reallocates only when the site is already hitting its crawl capacity limit. And noindex does not save crawling, because Google still requests the page before dropping it. Together these explain why no single control on the ladder below is a universal answer.

Choosing the Control: A Decision Ladder

Four controls cover most parameter problems, and each answers a different question. Do not choose by habit. Choose by the fate you want for the address: should it vanish, merge, stay out of search or simply go uncrawled?

Redirect when the address should not exist

The canonical documentation ranks redirects as the strongest signal. A permanent redirect tells Google the old address is a worse version of the target, and the page advises using it only when retiring a duplicate. Server-side HTTP redirects act fastest. If a parameter serves nobody, for instance a legacy session token or an abandoned campaign tag, redirecting to the clean address is the most durable fix.

Canonical when the address must keep working

The next strongest is the rel canonical annotation, delivered as a link element in the HTML head or as an HTTP Link header. The header exists for non-HTML files such as PDFs, and Google recommends choosing one of the two, because using both invites errors. The same page advises a self-referencing canonical on the preferred page, absolute URLs, no sitemap address that contradicts the tag, and consistent internal links to the canonical version. A canonical is only honoured inside a valid head. For a catalogue of common mistakes, read canonical tag issues, audit and fixes.

On a tracking-tagged URL the canonical normally points to the untagged version. Visitors are unaffected: they click the campaign link, the tag stays in place and analytics keep working. The price is that Google still fetches the page to read the tag, so the canonical steers consolidation without reducing crawling.

Sitemaps as a weak, supporting signal

The canonical documentation describes sitemap inclusion as a weak canonical hint. The practical rule follows directly: your sitemap should list only the clean, preferred addresses. A sitemap pointing at one URL while the tag points at another is a contradiction.

Noindex when the page must exist but not rank

Google's documentation on blocking indexing (Opens in a new tab) explains that noindex can be sent as a meta tag or an X-Robots-Tag header, and that when Googlebot sees it the page is dropped from results even if other sites link to it. The essential condition: the page must not be blocked by robots.txt. If it is, the crawler never sees the rule and the page can still surface through links. Google must recrawl the page to notice the change, which can take months for unimportant pages. The canonical documentation also advises against using noindex to choose a canonical within one site, since it removes the page from search altogether.

Robots.txt when the goal is less crawling

The robots.txt introduction (Opens in a new tab) describes the file as a way to manage crawler traffic and server load, and notes it can keep crawlers away from unimportant or near-duplicate pages. It then adds the caveat that it is not a mechanism for keeping a page out of Google. A blocked URL that other sites link to can still appear in results, usually without a description, and the canonical documentation says not to use robots.txt for canonicalization.

For authoring, the guide to creating a robots.txt file (Opens in a new tab) reminds you that the file sits at the site root, applies only to the protocol, host and port where it is published, accepts the asterisk wildcard in rules and treats paths as case-sensitive. It offers no example with query parameters, so any parameter pattern you write should be tested against real addresses from your site.

The four controls compared

The crawling column summarises statements from the documents above. How much any of these changes your own traffic can only be measured in your server logs.

Internal Search Pages: Index, Block or Curate

Why search results are a class of their own

Other families have a limited set of variants: one campaign tag, one sort choice. Internal search is bounded only by what people can type. Each result page is typically thin, changes per visitor and may be empty when nothing matches. The pages that deserve to rank are your categories and listings; a search result is just another presentation of them. There is also an abuse angle: someone can type offensive phrases into the search box and link to the resulting addresses from other sites, and if those pages are crawlable and indexable, unwanted pages could be listed under your name. That is a general security observation, not something Google's documents describe.

Two goals that need two tools

The URL structure documentation says plainly that search results, calendars and sorting or filtering functions are examples of dynamic URLs you can block from crawling with robots.txt. Keep two goals apart. Reducing crawling is the job of robots.txt. Keeping pages out of results is the job of noindex. On a single URL they cannot work together, since a blocked page never reveals its rule.

A practical sequence

The following is an implementation suggestion derived from the two documents above, not an official Google recipe.

• If search result pages are not indexed yet, block the search path in robots.txt to cut crawling and avoid linking to those addresses from crawlable pages.
• If they are already indexed, leave them crawlable and add noindex first. Once they have dropped out, bring in the robots.txt block. Reverse the order and the noindex is never seen.
• Remember that Google must recrawl to see noindex and that this may take months for unimportant pages.
• Handle the search action on the server and do not leave zero-result queries in circulation as ordinary pages. This is a general quality measure rather than something the documents say, and it reduces the thin-content footprint of empty results.

Turning search behaviour into worthwhile pages

What visitors type is a free demand map. For a phrase that people clearly search for, a hand-built category or collection page works better than a parameterised search page: its own title, description, selected listings and a self-referencing canonical. Such a page is a navigation aid for people, and it should exist only in numbers an editorial team can maintain. Mass-producing them creates a different risk, a pile of thin pages. Which queries deserve a page depends on internal search data, the real catalogue and product strategy; this article cannot supply volume or demand figures.

Fictional example: a rental marketplace

Fictional example: a marketplace for event equipment rentals passes the search term in a parameter named q, and its result pages also accept sort and page-number parameters. Server logs show Googlebot requesting search result addresses more than anything else. The team checks the Page indexing report and finds that some of those addresses are indexed. They first add noindex to the search result template. After seeing the pages drop out, they block the search path in robots.txt and remove search addresses from internal links. In parallel, editors build collection pages for a few of the most typed equipment categories. The example shows the order of decisions and contains no figures or outcomes.

Keeping Campaign Tags Out of the Index

Campaign URLs are for sharing, not for site navigation

Campaign parameters make sense on external channels such as email, social posts and ads. Your own menus, listing cards and sitemap should never carry them. The canonical documentation asks you to link internally to the canonical URL consistently, and tagged internal links break that consistency while also corrupting attribution.

Generate the clean address in one place

Compute the canonical URL in a single function in the template layer. Whatever query string arrived, drop parameters that do not change content and join the allowed ones in a fixed order. Feed the output to the canonical tag, the sitemap and in-page links alike. Keeping the allow-list short and explicit means that when someone ships a new parameter, the default is safe: an unrecognised parameter never reaches the canonical. That design is an engineering suggestion and does not come from Google's documents.

Cleaning on the server or in the browser

If analytics must read a campaign parameter after the page loads, redirecting on the server to strip it can break attribution. Do not add such a redirect before confirming when your measurement setup reads the parameter. Another approach is to tidy the address bar with the History API after analytics has read it. That changes what the visitor sees but not what a search engine sees, and it is an engineering option that Google does not specifically recommend. Test it with whoever owns campaign measurement.

How canonical and tracking cooperate

In this setup the visitor clicks a tagged link, analytics records the visit and the canonical in the page head names the clean address. Even if a search engine sees the tagged URL, it learns which version you prefer. It is the lowest-risk pattern for consolidating without disturbing measurement, and it is a reasonable first step for most sites. Naming standards for UTM values are a separate subject and are not covered here.

Avoid contradictory signals

If the canonical names the clean URL, the sitemap lists the tagged one and an internal link points to a third variant, Google will make its own choice. One source of truth for all three is the simplest way to stay consistent. If you add a redirect, the address being redirected should not remain in the sitemap or in a canonical.

Checking the Work in Search Console

Single-address checks with URL Inspection

The Search Console URL Inspection tool (Opens in a new tab) shows Google's indexed information for a page and can run a live test of its current version. It reports the user-declared canonical separately from the Google-selected canonical, which may not match. It also shows whether robots.txt allowed the crawl, whether the page could be fetched and whether a noindex rule blocked indexing. A sitemaps section lists the submitted or robots.txt-referenced sitemaps that include the address.

Know its limits. The live test cannot predict which canonical Google will select, a valid result does not guarantee indexing, and duplicates are not detected by it. Redirects are followed without being reported, and daily limits apply to inspections and indexing requests. Treat the tool as a spot check on sample addresses, not as proof that a bulk clean-up worked.

The Page indexing report

The noindex documentation suggests using the Page indexing report to monitor pages where Googlebot found a noindex rule. Looking at how parameter addresses are grouped in that report is one of the most direct ways to see whether a fix is working. Compare before and after rather than reading a single snapshot.

A short checklist

1. Open a sample tracking-tagged URL in URL Inspection and compare the declared and selected canonicals.
2. Confirm that the sitemap contains no addresses with parameters.
3. Test the robots.txt rules for the search path, sorting and session parameters against real addresses.
4. Check that any template carrying noindex is not blocked in robots.txt.
5. Compare how parameter addresses are classified in the Page indexing report before and after the change.
6. Watch the share of Googlebot requests that go to parameter addresses in your server logs.

When analysing logs, use verified Googlebot entries; trusting the user-agent string alone can mislead. Effects take time to appear, so avoid judging a change from a single day.

Mistakes That Undo Good Intentions

• Combining noindex with a robots.txt block on the same URL: the block hides the rule from Google.
• Canonicalising every parameter page to page one: it can weaken discovery of listings that only appear deeper in a sequence.
• Using robots.txt to pick a canonical: the documentation says not to.
• Putting tagged URLs in the sitemap: it contradicts the canonical tag.
• Carrying campaign tags in internal links: it breaks canonical consistency and muddles attribution.
• Leaving session IDs in the address: the number of URLs becomes unbounded.
• Leaving the parameter list without an owner: every release can add a new parameter, and without an inventory the problem returns.
• Judging from one day of data: reports lag behind changes.

Frequently Asked Questions

Do UTM-tagged URLs get indexed?

They can be crawled and treated as duplicates of the clean URL when the content is the same. Google's canonical documentation uses a tracking-parameter URL (a gclid example) as a case of duplicate URLs. A rel canonical pointing to the clean address, plus clean internal links, steers which version Google picks. It does not guarantee the outcome.

Should internal search results be noindex or blocked in robots.txt?

Not both on the same URL: a page blocked in robots.txt never lets Google see its noindex rule. Google's URL structure documentation names search results among the dynamic URLs you can block from crawling. If you want to reduce crawling, block them. If result pages are already indexed and you want them dropped, let them be crawled with noindex first.

Is a canonical tag enough for parameter URLs?

Often it is enough to consolidate duplicate variants, but it does not cut crawling, because Google still fetches the page to read the tag. When a site generates a large number of low-value variants, stop producing them, redirect them or block them in robots.txt.

Why are session IDs in URLs a problem?

Every visitor sees the same content at a different address, so the number of duplicate URLs has no ceiling. Google's URL structure documentation recommends cookies instead of session IDs where possible.

What should I watch in Search Console while cleaning up parameters?

Compare the user-declared canonical with the Google-selected canonical in the URL Inspection tool, look at how parameter URLs are classified in the Page indexing report, and confirm that your sitemap lists only clean addresses.

Conclusion

Parameter URLs are not one problem but several families. Tracking and session parameters leave content untouched and should be consolidated or stopped at the source. Sort variants are low value. Pagination carries real content. Internal search produces an unbounded supply of thin pages. Redirects, rel canonical, noindex and robots.txt each answer a different question, and substituting one for another, or combining the two that conflict, noindex and a block on the same address, does not work. The size of your site decides whether crawl budget is a live concern. Whichever route you take, test the change in Search Console before and after.

If you need help writing copy for collection and category pages that replace parameterised search results, see the SEO content writing service at Akua Adforce; scope and terms are on the product page.