Publisher growth tactics for election season | WEBINAR

Learn More

SODP

SODP Media

  • Education
    • Articles
      • Audience Development
      • Content Strategy
      • Digital Publishing
      • Monetization
      • SEO
      • Digital Platforms & Tools
    • Opinion
    • Podcast
    • Events
      • All Events
  • Top Tools & Reviews
  • Research & Resources
  • Community
    • Slack Channel
    • Newsletter
  • About
    • About Us
    • Contact Us
    • Editorial Policy
  • English
sodp logo
SODP logo
    Search
    Close this search box.
    Login
    • Education
      • Podcast
      • Articles
        • Audience Development
        • Content Strategy
        • Digital Publishing
        • Monetization
        • SEO
        • Digital Platforms & Tools
        • Articles
        • Opinion
        • Podcasts
        • Events
        • Audience Development
        • Content Strategy
        • Digital Publishing
        • Monetization
        • SEO
        • Digital Platforms & Tools
        • View All
    • Top Tools & Reviews
        • Headless CMS Platforms
        • Digital Publishing Platforms
        • Editorial Calendar Software
        • Magazine Apps
        • Email Newsletter Platforms
        • More Best Tools Lists
        • Reviews
    • Research & Resources
    • Community
      • Slack Channel
      • Office Hours
      • Newsletter
        • Slack Channel
        • Newsletter
    • About
      • About Us
      • Contact Us
      • Editorial Policy
        • About Us
        • Contact Us
        • Editorial Policy
    placeholder
    SODP logo
    Become a Brand Partner
    Home > Publisher SEO Course > Chapter 2: Technical SEO > Crawl Speed and Frequency
    6

    Crawl Speed and Frequency

    Crawl Speed and Frequency
    Previous Module
    Back to Chapter
    Next Module

    Learning Objective

    After going through this module, you should understand what crawl budget is, how it’s allocated to a website and how to optimize for it.

    Video Duration

    23:27

    Answer Quiz

    Upgrade to access

    Materials

    Upgrade to access

    Resources

    Reports and Resources

    2.6.1 What Is a Crawl Budget?

    Crawl budget is the number of pages on your website a web crawler will crawl within a given timeframe.

    Each time you hit the publish button, Google needs to crawl and index the content for it to start appearing in search results. Given the scale and volume of content on the internet, crawling becomes a valuable resource that needs to be budgeted and rationed for the most efficient use.

    To put it simply, it’s hard for Google to crawl and index every single page on the internet everyday. So, Google crawls each website according to its assigned budget.

    2.6.2 How Is Crawl Budget Assigned to Websites?

    Crawl budget is assigned to websites based on two factors — crawl limit and crawl demand.

    Crawl Limit

    This is a website’s capacity and/or willingness to be crawled.

    Not every website is built to be crawled everyday. Crawling involves Googlebot sending requests to your website’s server that, if done too frequently, may stress the server’s capacity.

    Also, not every publisher wants their site crawled continuously.

    Crawl Demand

    Crawl demand is a measure of how often a particular page wants to be (re)crawled. Popular pages or pages that are updated frequently need to be crawled and recrawled more frequently.

    2.6.3 Why Should Publishers Care About Crawl Budget?

    If Google can’t crawl and index your content, that content will simply not show up on search results.

    That being said, crawl budgets are generally only a concern for medium to large publishers who have more than 10,000 pages on their website.  Smaller publishers shouldn’t need to worry overly about crawl budgets.

    Upgrade to a premium plan to access the content.

    Most Popular

    Upfront
    (1 payment)

    $1000
    • 7 chapters
    • 53 modules
    • 20+ hours of educational videos
    • Monthly 1-on-1 coaching
    • Ready-to-use templates
    Subscribe
    Fixed Price Guarantee
    Recommended

    4 Quarterly Payments Of

    $250
    • 7 chapters
    • 53 modules
    • 20+ hours of educational videos
    • Monthly 1-on-1 coaching
    • Ready-to-use templates
    Subscribe
    Fixed Price Guarantee
    Best Value

    Course Only
    (1 Payment)

    $300
    • 7 chapters
    • 53 modules
    • 20+ hours of educational videos
    • Monthly 1-on-1 coaching
    • Ready-to-use templates
    Subscribe
    Fixed Price Guarantee

    Active now

    6

    Crawl Speed and Frequency

    See more

    1

    Design and Layout

    2

    Site Architecture

    3

    Page Experience

    4

    News Sitemap

    5

    Schema

    7

    Links to Sponsored and User-Generated Content

    8

    Google Publisher Center

    9

    Bing News PubHub

    10

    Ads, Popups and Best Practices

    SODP logo

    State of Digital Publishing is creating a new publication and community for digital media and publishing professionals, in new media and technology.

    • Top tools
    • SEO for publishers
    • Privacy policy
    • Editorial policy
    • Sitemap
    • Search by company
    Facebook X-twitter Slack Linkedin

    STATE OF DIGITAL PUBLISHING – COPYRIGHT 2025