• Insights
    • Exclusive Insights
    • Podcast
    • Articles
      • Advertising & Sales
      • Audience Development
      • Content Strategy
      • Digital Analytics
      • Digital Publishing
      • Monetization
      • SEO
      • Social Media
      • Technology
    • Top Tools & Reviews
    • Features
  • Education
    • Publisher SEO Course
  • Solutions
    • SEO for Publishers
    • Content Strategy Services
    • Brand Partner Solutions
  • Community
    • Slack Channel
    • Office Hours
    • Newsletter
  • About
    • About Us
    • Contact Us
    • Editorial Policy
sodp logo
Search
Close this search box.
    SODP logo
    Login
    • Insights
      • Exclusive Insights
      • Podcast
      • Articles
        • Advertising & Sales
        • Audience Development
        • Content Strategy
        • Digital Analytics
        • Digital Publishing
        • Monetization
        • SEO
        • Social Media
        • Technology
      • Top Tools & Reviews
      • Features
        • Podcasts
        • Articles
        • Top Tools & Reviews
        • Features
        • Opinion
        • Advertising & Sales
        • Audience Development
        • Content Strategy
        • Digital Analytics
        • Digital Publishing
        • Insights
        • Monetization
        • SEO
        • Social Media
        • Technology
    • Education
      • Publisher SEO Course
        • Publisher SEO Course New
    • Solutions
      • SEO for Publishers
      • Content Strategy Services
      • Brand Partner Solutions
        • SEO For Publishers
        • Content Strategy Support
        • Brand Partner Solutions
    • Community
      • Slack Channel
      • Office Hours
      • Newsletter
        • Slack Channel
        • Office Hours
        • Newsletter
    • About
      • About Us
      • Contact Us
      • Editorial Policy
        • About Us
        • Contact Us
        • Editorial Policy
    placeholder
    State of Digital Publishing Logo
    Publisher SEO Course Chapter 2: Technical SEO Crawl Speed and Frequency
    6

    Crawl Speed and Frequency

    Crawl Speed and Frequency
    Previous Module
    Back to Chapter
    Next Module

    Learning Objective

    After going through this module, you should understand what crawl budget is, how it’s allocated to a website and how to optimize for it.

    Video Duration

    23:27

    Answer Quiz

    Upgrade to access

    Materials

    Upgrade to access

    Office Hours

    Upgrade to access

    2.6.1 What Is a Crawl Budget?

    Crawl budget is the number of pages on your website a web crawler will crawl within a given timeframe.

    Each time you hit the publish button, Google needs to crawl and index the content for it to start appearing in search results. Given the scale and volume of content on the internet, crawling becomes a valuable resource that needs to be budgeted and rationed for the most efficient use.

    To put it simply, it’s hard for Google to crawl and index every single page on the internet everyday. So, Google crawls each website according to its assigned budget.

    2.6.2 How Is Crawl Budget Assigned to Websites?

    Crawl budget is assigned to websites based on two factors — crawl limit and crawl demand.

    Crawl Limit

    This is a website’s capacity and/or willingness to be crawled.

    Upgrade to a premium plan to access the content.

    COURSE ONLY (1 payment)

    $695

    • 7 chapters
    • 53 modules
    • 20+ hours of educational videos
    • Monthly 1-on-1 coaching
    • Monthly group office hours
    • Ready-to-use templates
    • Members-only articles
    Enroll Now
    • GIFT

    Fixed Price Guarantee

    Upfront (1 payment)

    $1500

    • 7 chapters
    • 53 modules
    • 20+ hours of educational videos
    • Monthly 1-on-1 coaching
    • Monthly group office hours
    • Ready-to-use templates
    • Members-only articles
    Enroll Now
    • GIFT

    Fixed Price Guarantee

    2 quarterly payments of

    $750

    • 7 chapters
    • 53 modules
    • 20+ hours of educational videos
    • Monthly 1-on-1 coaching
    • Monthly group office hours
    • Ready-to-use templates
    • Members-only articles
    Enroll Now

    Fixed Price Guarantee

    4 quarterly payments of

    $375

    • 7 chapters
    • 53 modules
    • 20+ hours of educational videos
    • Monthly 1-on-1 coaching
    • Monthly group office hours
    • Ready-to-use templates
    • Members-only articles
    Enroll Now

    Fixed Price Guarantee

    Active now

    6

    Crawl Speed and Frequency

    See more

    1

    Design and Layout

    2

    Site Architecture

    3

    Page Experience

    4

    News Sitemap

    5

    Schema

    7

    Links to Sponsored and User-Generated Content

    8

    Google Publisher Center

    9

    Bing News PubHub

    10

    Ads, Popups and Best Practices

    Subscribe to receive a recap of our top stories in digital media.

    SODP logo

    State of Digital Publishing is creating a new publication and community for digital media and publishing professionals, in new media and technology.

    • About us
    • Contact us
    • Editorial policy
    • Privacy policy
    • Sitemap
    • Newsletter
    • Publisher SEO course
    • Become a member
    • Slack community
    • SEO for publishers
    • Content strategy support
    • Partner with us
    • Podcast
    • SEO Office Hours
    • Editorial Office Hours
    Facebook Twitter Slack Linkedin

    STATE OF DIGITAL PUBLISHING – COPYRIGHT 2023

    PUBLISHERS, THESE 20+ TOOLS WILL SUPERCHARGE YOUR GROWTH

    Subscribe to our newsletter and receive the ultimate list of smart technologies and free resources to boost your publisher SEO and content efforts.

    Join 3,000+ publishers around the world and get exclusive tips and resources delivered to your inbox.

    Being a part of the SODP Community has made a real impact on my problem-solving at work. From bouncing off paywalls solution ideas to thinking about an SEO strategy for my paywall, it’s the most value-adding professional community I’m a part of.

    Yvette Dimiri, Head of Growth, Stears

    PUBLISHERS, THESE 20+ TOOLS WILL SUPERCHARGE YOUR GROWTH

    Subscribe to our newsletter and receive the ultimate list of smart technologies and free resources to boost your publisher SEO and content efforts.

    Join 3,000+ publishers around the world and get exclusive tips and resources delivered to your inbox.

    Being a part of the SODP Community has made a real impact on my problem-solving at work. From bouncing off paywalls solution ideas to thinking about an SEO strategy for my paywall, it’s the most value-adding professional community I’m a part of.

    Yvette Dimiri, Head of Growth, Stears