Crawling and Indexing for Nashville Local Business Sites

Published Read 4 min read

On this page

Before a page can appear in search, Google has to find it, crawl it and decide to index it. For a small Nashville business site, the practical questions are simple: are your important pages reachable, is anything blocking them, and what do the index states in Search Console mean? Google’s own definitions answer more of those questions than the folklore around them.

Crawl budget on a small site

Google’s crawl budget guide describes crawling as a balance between a crawl capacity limit, which “can go up and down based on a few factors” such as how consistently and quickly the site responds, and crawl demand. It also says who needs to worry: “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide” (Crawl budget management).

A forty-page plumbing site is not the audience for that guide. What still matters is basic health: a fast, reliable server, and a clean set of URLs so the crawling you get lands on pages that matter.

Two index states, in Google’s words

Search Console’s Page indexing report defines the two states local sites run into (Page indexing report):

  • Discovered, currently not indexed: “The page was found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl.”
  • Crawled, currently not indexed: “The page was crawled by Google but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.”

For the first, check server speed and errors, and make sure the page is linked from pages Google already crawls. For the second, look hard at the page itself. If many of your pages sit in this state and they are near-copies of each other, such as ten city pages that differ only in the town name, making each page genuinely distinct and useful is the change within your control. Google’s help does not promise indexing either way; improve the page because it serves visitors, and check the report over the following weeks.

robots.txt that does not backfire

Google’s robots.txt introduction says a robots.txt file “tells search engine crawlers which URLs the crawler can access on your site,” and adds that it is used mainly to avoid overloading the site and “is not a mechanism for keeping a web page out of Google” (Robots.txt introduction and guide). Use noindex for pages that should stay out of search, and do not block the CSS and JavaScript files Google needs to render your pages, or your location pages, images and other content you want found. robots.txt is useful for keeping crawlers out of low-value areas such as internal search results and staging copies.

XML sitemaps

A sitemap helps Google discover your URLs. Google’s sitemap documentation is specific about these fields: Google ignores the <priority> and <changefreq> values, and it uses <lastmod> “if it’s consistently and verifiably (for example by comparing to the last modification of the page) accurate” (Build and submit a sitemap). Update lastmod only when a page changes.

Splitting the sitemap by page type, such as service pages, location pages and blog posts, lets you compare indexing by segment in Search Console. If the location-page sitemap shows few indexed pages, you have narrowed the problem to that template.

Internal links are how crawlers travel your site. Find orphan pages, the ones no other page links to, and give each a link from a relevant page. Hub pages that link to related services and locations create short, clear paths to your content.

Keep junk out of the index

Parameter variants for tracking, sorting and filtering, thin archive pages and internal search results can multiply one page into many URLs. Choose the tool by the goal:

  • Duplicates that should consolidate: a canonical tag pointing to the main URL.
  • Pages that should exist but stay out of search: noindex, with the page left crawlable so Google can see the tag.
  • Areas crawlers should not visit: a robots.txt disallow rule.

Search Console’s old URL Parameters tool is no longer available. Google announced on March 28, 2022 that it was “deprecating the URL Parameters tool in 1 month” and that “Google’s crawlers will learn how to deal with URL parameters automatically” (Spring cleaning: the URL Parameters tool).

Sources

Related posts:

  1. Local Schema Implementation for Nashville Businesses LocalBusiness structured data states your business facts in a form machines can read: your name, address, coordinates, hours and the kind of business you are. Google's documentation frames it as......
  2. Content Strategy Fundamentals for Nashville Businesses Before deciding what to write, decide which searches your site can realistically win. Search "best cardiologist in Nashville" and the first page may be filled with directories, review sites and......
  3. Content Optimization Fundamentals for Nashville Sites Optimizing a page's content means making it answer the searcher's question completely and efficiently. Length and keyword counts are the wrong targets. The useful levers are covering the subject the......
  4. Local Link Prospecting for Nashville Businesses Prospecting is the research that happens before any outreach: finding the overlap between sites that link out and sites worth earning a link from, and building a qualified list before......
  5. Local Pack Optimization for Nashville Businesses The local pack, the short list of businesses shown with a map above the regular results, is decided by relevance, distance and prominence, and Google is explicit that "There's no......
  6. URL Structure for Nashville Local Businesses A consistent URL scheme lets a business add locations and services for years without a painful site-wide change of addresses, and that matters more than where a keyword sits in......
  7. Mobile Optimization for Nashville Local Businesses A site that works on a phone is the floor, not the goal. Google crawls and indexes the mobile version of your pages, so the mobile experience is the version......
  8. Caching Strategy for Nashville Business Sites Caching is not a single switch you flip on; it is several coordinated layers, and the hard part is not turning them on but keeping them honest. A page can......
  9. Google Business Profile Optimization for Nashville Google Business Profile optimization has two parts. The foundation is completing every field accurately, and the ongoing work is keeping the profile current and useful to the people deciding between......
  10. Link Reclamation for Nashville Businesses Link reclamation recovers backlink value you already earned but lost, and it is an easier ask than cold prospecting because the relationship or the mention already exists. When a site......
  11. Content Readability for Nashville Audiences Readability means matching the writing to the person you expect to read it. Text that is too dense for its audience and text that is too simple for it fail......
  12. Log File Analysis for Nashville Local Business Sites Server logs show what Googlebot did: every request it made, the response it got and how often it came back. Search Console reports on crawling too, but its Crawl Stats......
  13. Review Response Framework for Nashville Businesses A review response is public content written for the future searchers who will read it, not only for the person who wrote the review. Google's local ranking help puts the......
  14. WordPress Technical SEO for Nashville Local Businesses WordPress out of the box is built for compatibility: its defaults let almost any theme or plugin work. A fresh install with a few plugins added is a working site,......
  15. Product Page Optimization for Nashville Local Retailers A local retailer's product page should not be built to fight on national sellers' ground, price and shipping. The fight a local store can win is local shopping intent: the......

Leave a comment

Your email address will not be published. Required fields are marked *