Every Search Console property has a quiet deadline built into it. The Performance report keeps a limited history, and Google's article on analysing search traffic drops suggests widening the date range to 16 months, then adds that anyone who wants more than that can use the Search Analytics API to pull the data and store it in their own systems. In other words, whatever happens before the window is gone unless you copied it. You can read the advice in Google's post on search traffic drops (Opens in a new tab).
This guide is written for analysts, agencies and data engineers who want to close that gap. It compares the three ways of getting data out, walks through a BigQuery bulk export and a lighter API-based archive, explains the limits that quietly distort the numbers, and covers who should be able to read the result. If you want a refresher on what the report itself shows, start with our walkthrough of the Search Console performance report.
What Falls Off the Edge After 16 Months
History is only valuable when someone asks a question about it, and the questions that matter tend to arrive late. A board member asks how the site did through last autumn's redesign. A client asks whether a campaign from two summers ago ever paid off in organic search. If the answer lived only inside the interface, it may already be out of reach.
The year-over-year squeeze
Seasonal businesses need to put this month next to the same month last year. That requires data from twelve months back, so a 16-month window leaves only a four-month margin before the comparison month slips away. Wanting a second year of comparison, such as this January against both of the previous two, is impossible without an archive. This is simple arithmetic rather than a Google quirk, but it explains why teams that care about trends should start saving data in their first month.
Baselines for migrations and updates
Domain moves, URL restructures and algorithm updates all demand a clean before-and-after picture. If you investigate a change fourteen months after it happened, the weeks before it may already be beyond the window, and you are left estimating a baseline instead of measuring one.
A fictional case: the rebuilt help centre
Imagine a software company that rebuilds its help centre and wants to know, a year and a half later, which old articles had been quietly bringing in the most clicks. Nobody archived anything. The team now has screenshots from a slide deck and a few remembered article titles. Another team in the same position had switched on an export from the beginning, and answers the same question with a single query. The point of the story is the cost of waiting.
What an archive cannot do
An archive preserves raw performance data. It does not annotate it. If nobody writes down when the site was redesigned, when a template changed or when a migration finished, you will hold years of numbers and still struggle to explain them. Pair the data with a dated change log.
Interface, API or BigQuery: Choosing Your Route
There are three practical ways to get Search Console data out. They overlap, but each has a different reach into the past, a different level of detail and a different cost of ownership.
Downloading from the report
Google's page about Search Console data explains that tables show up to 1,000 rows, so some rows may be omitted, while the API can return a larger selection. It also notes that values shown as a tilde or a dash in the report are exported as zeros. That second point matters if you feed downloads into a spreadsheet: a true zero and a masked value look identical afterwards. Read the details on the About Search Console data page (Opens in a new tab).
Pulling through the API
The API lets you request any dates still inside the window, so it is the only route that can rescue the older days you have not saved. It returns at most 25,000 rows per request and needs paging beyond that. It also needs a script, credentials and somewhere to put the results.
Letting BigQuery do the saving
The bulk export writes performance data to a BigQuery dataset in your Google Cloud project on a recurring schedule. After setup there is nothing to run by hand. The catch is that it does not include data from before setup.
A decision table
| Route | How far back | Rows you can get | Upkeep | Best for |
|---|---|---|---|---|
| Report download | Whatever the interface shows | Up to 1,000 per table | None | Quick one-off checks |
| Search Analytics API | Days still inside the window | 25,000 per request, 50 thousand per day per search type at most | A script, credentials and storage | Rescuing history and custom schemas |
| BigQuery bulk export | From setup onwards | Date-partitioned site and URL tables | Cloud project, billing and access roles | A permanent archive queried with SQL |
A combination that works
Most mature setups use two routes. Switch on the bulk export first, since every day of delay is a day that cannot be recovered through it. Then use the API once to backfill what still sits in the window. After that, the API script can retire, or stay as a periodic cross-check against the export.
Starter Archive: Daily API Pulls and a Simple Table Design
Not every team wants a Google Cloud project. A lightweight archive built on the API and a few well-named files is a respectable starting point, and it is also how you rescue history before the bulk export has collected any.
Pull one day at a time
Google's guide to getting all your data recommends daily queries: set the start and end date to the same day, and a single day of data should not exceed your daily quota. Data is typically available after two to three days, and you can find the latest available date by running a date-grouped query over the past ten days. The full pattern is in the guide to getting all your data (Opens in a new tab).
Page through every result
The documented maximum for rowLimit is 25,000, and the default is only 1,000. To get everything for a day, repeat the same query while raising startRow by 25,000 each time, and stop when a response comes back with zero rows. A script that forgets to set rowLimit will quietly save one thousand rows per request and look successful. Add a test that fails when a day's row count is suspiciously round.
Know the daily ceiling
The guide states that the API exposes a maximum of 50 thousand rows of data per day per search type, sorted by clicks. Rows beyond that cap are not returned. The query reference adds that the API does not guarantee all rows, only top ones. Run web, image, video and other search types as separate queries, and treat the query-level archive as a large sample of the biggest rows rather than a complete census.
Choose dataState on purpose
In the search analytics query reference (Opens in a new tab), dataState defaults to final, which returns only finalised data. The value all includes fresh data, and hourly_all adds an hourly breakdown that may be partial. For a permanent archive the default is the safer choice, because saving a partial day freezes a number that may later change. Dates are interpreted in Pacific Time, so write that on every report that mixes archive data with local calendars.
Dimensions can cost you rows
Google warns that including page or query dimensions can make some data drop, because the system drops data to compute results in a reasonable time. A useful safeguard is to store a second, coarse daily total without those dimensions. The gap between the coarse total and the sum of your detailed rows is your estimate of how much detail was lost that day.
Staying inside the quotas
The usage limits page lists, for Search Analytics calls, 1,200 queries per minute per site and per user, and per project 30,000,000 queries per day and 40,000 per minute. Short-term quota is measured in 10-minute chunks and long-term load over one-day periods. A script for one site is nowhere near those limits. An agency job that fans out over many properties at once can approach them, so add a delay between requests. See the usage limits (Opens in a new tab) for the full list.
A minimal table layout
This layout is a general suggestion, not a Google requirement. Keep one row per combination of the fields below, and store raw sums rather than precomputed rates so that you can regroup later.
| Column | Purpose |
|---|---|
| data_date | The day, in Pacific Time |
| search_type | Web, image, video or another surface, pulled separately |
| country and device | Optional breakdowns for later slicing |
| page and query | The detailed dimensions; query may be empty |
| clicks and impressions | Raw counts, never averaged rates |
| position_sum | So average position can be rebuilt after regrouping |
| pulled_at | When the row was collected, to distinguish revisions |
Save each month as one file, name it by month and search type, and keep a short summary beside it with the row count and total clicks. That summary is the cheapest integrity check you will ever have.
Switching On the BigQuery Bulk Export
This is the more durable option, and the setup order matters. The steps below follow Google's start-a-bulk-export page, which remains the authority for screen names; open the bulk export setup instructions (Opens in a new tab) alongside this section.
What you need first
You need a Google Cloud project with billing enabled, plus the BigQuery API and the BigQuery Storage API. The page says there is a free usage level, with storage and query usage above it charged. Set a budget alert before you start so a runaway query does not become a surprise invoice.
Granting access to the export
In Google Cloud, give the service account search-console-data-export@system.gserviceaccount.com two roles: BigQuery Job User and BigQuery Data Editor. During access reviews treat this account as infrastructure. Search Console writes to your project through it, so changes to its permissions should be deliberate and communicated to whoever owns the export.
Entering the project and dataset
In Search Console, open Settings and choose Bulk data export, then enter your project ID, not the project number. The dataset name defaults to searchconsole and must start with that string.
Pick the location carefully
Google warns that you cannot easily change the dataset location once exports have begun. Decide where the data may legally and organisationally live before you click through, particularly if you export on behalf of clients in regulated sectors.
The first run and the gap before it
The first export happens up to 48 hours after successful configuration and includes data for the day of the export. Nothing earlier is included; for history, the page points to the API or the reports. If a non-persistent error occurs, the export retries the next day. Because the gap before setup cannot be refilled, the sensible time to configure it is as soon as someone has access to the Cloud project, not when the report design is finished.
Leave the schema alone
Google states that schema changes will cause the export to fail. Do not add columns to the exported tables. Build views or derived tables in a separate dataset instead, and give analysts access to those.
Anonymised Queries, Row Caps and Other Blind Spots
An archive of imperfect data is still valuable, as long as the imperfections are documented. These are the blind spots most likely to cause an argument between two dashboards.
Anonymised rows
Google's table reference describes a boolean named is_anonymized_query that flags rare queries. When it is true, the query field is empty to protect the privacy of the people searching. The reference describes the empty value in two ways, as null in the field description and as a zero-length string in the site table, so write your filters to handle both. The table guidelines and reference (Opens in a new tab) explain the fields.
Why the top query is often blank
The query guidelines note that anonymised queries often make up the single most common query for a site. A naive top-queries chart will therefore lead with an empty label. Exclude empty query values from query rankings, but keep those rows in your click and impression totals, because the totals are what reconcile with the interface.
Duplicate keys need aggregation
The tables are not guaranteed to be consolidated by date, URL or site. You may find several rows for the same query on the same day and search type, so every metric should be wrapped in SUM or COUNT. The guidelines show a deliberately wrong example that returns one row per query and an aggregated version that adds them up. The query guidelines and sample queries (Opens in a new tab) include both.
Position is not a plain average
The site table stores sum_top_position, which is zero-based, and the reference gives average position as the sum of that field divided by impressions, plus one. Averaging a precomputed position column across rows would give the wrong answer. Always rebuild position from sums.
Rare queries are filtered everywhere
The About Search Console data page adds that rare queries are left out of reports to protect user privacy. No route gives you every query a site has ever appeared for. Put this limitation in the footer of reports: query-level numbers describe a privacy-filtered sample.
Time zones and delays
Both data_date and API dates use Pacific Time, and data usually arrives two to three days late. Expect daily charts built from the archive to shift slightly against a local calendar, and avoid reading the last few days as final.
Retention, Access and Cost Governance
An archive is a long-lived asset, so it needs rules from the first day: how long to keep data, who can read it and what it may cost.
The default is forever
According to the setup page, the exported data accumulates forever for your project unless you set an expiration time. The table reference repeats that tables and partitions are retained forever by default. For most teams that is the right default for an archive, but if your policy limits storage to a set period, you must configure it deliberately.
Set expiry on partitions, never on the table
After the tables exist, you can set a partition expiration of at least 14 days. The table reference warns that putting an expiration date on the whole table is probably not what you want, as it will delete all your data. One wrong setting could erase years of history, so write the policy down and have a second person review it.
Least privilege for readers
The advice in this subsection is general good practice rather than Google documentation. Raw query and page data can reveal a client's strategy. Restrict dataset read access to defined roles, name service accounts after their function, and remove people when they leave a project. In agency work, put in the contract whose Cloud project holds the dataset and what happens to it at the end of the engagement.
A word on personal data
Search queries are free text typed by people, so they can occasionally contain something personal. The anonymisation Google applies is a protection in that direction, and anonymised rows arrive without query text. Before you join the archive with other datasets or extend its purpose, review retention and lawful use with whoever advises you on data protection.
Documentation for handover
Archives often outlive the person who built them. Keep a one-page record of the project ID, dataset name and location, where any pull script runs and which credentials it uses. Add the date the archive began, because that date defines how much history anyone can honestly claim.
Keeping costs predictable
The query guidelines remind you that BigQuery charges for queries and that tables can grow large. Because exported tables are date partitioned, a WHERE clause on the date range limits how much input is scanned. In practice, avoid selecting every column, give dashboards explicit date filters and materialise frequently read summaries into small derived tables. Check the current Google Cloud pricing page for exact rates.
Watching for silent failure
The ExportLog table records each successful export, and Google states that failed exports are not recorded there. Its epoch_version field starts at 0 and rises when previously saved data for a table and date is revised. Use the log to confirm each day arrived, and the epoch to notice when a day was restated.
From Archive to Reporting
Archiving only pays off when the data reaches decisions. A small routine keeps the pipeline trusted, and a reporting layer turns it into answers.
A monthly reconciliation routine
- Confirm that every day of the previous month appears in the ExportLog table.
- Compare the month's total clicks in the site table with the same month in the interface, and note small gaps rather than hiding them.
- Check the row counts of the latest API pulls against the prior months, looking for round, suspicious numbers.
- If you keep monthly files, recompute the row count and click total in each summary.
- Look at BigQuery storage and query spending in billing.
- Review who has access and remove anyone who has left.
Feeding the monthly report
An archive makes the monthly SEO report far stronger: instead of comparing this month with last month alone, you can add the same month of the previous year and the trend between. The structure of such a report is covered in our monthly SEO report template, and the archive can supply every table in it.
Visibility dashboards
If you track impressions, query counts and page spread over time, the archive removes the cutoff from those charts. Our piece on a Google visibility dashboard describes which measures to chart; with the archive, the lines can run for years instead of months.
Mistakes that undermine trust
- Waiting to design the perfect schema: the export fixes its own schema, and each delayed day is a permanent gap.
- Editing the exported tables: schema changes make the export fail.
- Reading without aggregation: duplicate keys make unaggregated queries undercount.
- Deleting anonymised rows: totals then drift from the interface.
- Treating data as explanation: history shows what happened, not why; pair it with a change log.
Frequently Asked Questions
Does Search Console keep a copy of my data after the 16-month window?
Not for you to use. Google's own advice for anyone who wants to go beyond 16 months is to pull the data with the Search Analytics API and store it in their own systems. The bulk export does not reach backwards either, so the start date of your archive is the date you set it up.
Can I recover the months before I enabled the BigQuery export?
Not through the export itself. Google's setup page says the export does not include data from before configuration, and that the first export arrives up to 48 hours after a successful setup. For earlier days, use the API while they still sit inside the window, or download what the reports offer.
Why do my archived query totals differ from the interface?
There are several legitimate reasons. Rare queries are left out of reports to protect users' privacy, anonymised rows carry no query text, and the API is documented as returning top rows rather than guaranteeing every row. Compare site-level totals with site-level totals before blaming a pipeline.
How do I keep BigQuery queries from getting expensive?
Filter every query by date, because the exported tables are date partitioned, and select only the columns you need. Google notes that usage above the free level is charged, so check the current Google Cloud pricing page rather than relying on any figure quoted in an article.
Is it safe to set an expiry on the exported tables?
Only on partitions. Google's table reference warns that an expiry on the whole table will delete all your data. A partition expiry, which can be set after the tables are created and needs to be at least 14 days, removes only older days.
Conclusion
Keeping Search Console data beyond 16 months comes down to three moves. Enable the BigQuery bulk export now, because it does not include data from before setup. Use the API to rescue the days still inside the window, remembering the 25,000-row page size and the daily cap of 50 thousand rows per search type. And govern the archive from the first day: aggregate in every query, treat anonymised rows deliberately, set expiry only on partitions, and give readers the least access they need. A lightweight monthly file archive is a legitimate starting point when a Cloud project is not yet possible.
If you want help turning what the archive reveals into pages that earn search visibility, you can look at our SEO content writing service; scope and terms are listed on the product page.
Post a comment
Please log in to leave a comment. Login