Website Builders.com

  • Home
  • Builder Reviews
        • Our team have carefully reviewed a selection of premium website building platform, making it easier for you to choose the right one.

          While most website builders are compatible for generic  website building, including online stores, blogs and generic websites, some are better than others depending on what your website type.

        • Website Builder Reviews

          GoDaddy

          IM Creator

          Jimdo

          Squarespace

          Voog

          Weebly

          Wix

          WordPress

        • Store Builder Reviews

          BigCommerce

          GoDaddy Store

          Shopify

          Weebly

          Wix

          Yola

        • Comparisons

          Wix vs. Weebly

          Shopify vs. BigCommerce

  • Start A Blog
  • How To Guides
        • With over 500 pages of detailed tutorials and guides, our guides cover a large chunk of online topics.

          Learn crafty tips and tricks to make your time online smooth. Our directory is useful for internet newbies, as well as aspiring website builders.

          Check out the full directory.

        • Building a Website

          How To Build a Website

          Name Your Website – Domain

          Picking a Web Hosting Service

          Content and User Experience

          More

        • Searching and Research

          Searching the Web

          Doing Research Online

          The World of Wiki

          Finding Videos Online

          More

        • E-mails – Basics and Advanced

          Getting Started

          Sending an E-mail Message

          Opening Attachments

          Sending Attachments

          More

        • Social Media & Communication

          Instant Messaging

          Online Networking

          Facebook

          LinkedIn

          More

        • Files, Downloads, Software & Installs

          File Formats and Extensions

          Install Plug-Ins

          How to Download Files

          Downloading Software

          More

        • Making Money Online

          Making Money with Facebook

          Affiliate Marketing

          Virtual Assistant

          Writing Articles

          More

        • Online Shopping

          10 Tips for Smart and Safe Shopping

          Smart Shopping

          Protecting Personal Information

          Looking for Discounts

          More

  • Website Builder Tools Your Business Can’t Do Without (+ Links To Useful Resources)
    • HTML Character Codes
    • Advertiser Tracking Cookies: There ARE Ways To Opt Out – Here’s How
    • Website Glossary
  • Our Blog
        • Recent Posts

          When you have awesome writers, it is easy to spread wisdom. Take a look at some of our recent posts on various internet and website related topics.

          Have any suggestions on topics? Don’t hold back, send us your ideas:

          [email protected]

        • 12 Simple Rules on How to Use E-mail Politely

          Live Chat Software: Is it Worth It? Facebook Says Yes.

          Algorithmic Warfare: Is it the Beginning or the End?

          How to Promote Your Pet Business Online

          DIY Marketing: 7 Steps for any Business Owner

        • 20 Ideas to Generate Your First Online Sale

          Complete Guide to Content Marketing

          8 Best Free Photo Editors for Product Photosraphy

          Press Kits: DIY to Free Media Hype

          Why Backlinks are Essential for SEO – How To Build Them

  • About Us
    • Contact Us
    • FAQ
    • How Did You Make This Site?
    • How the Website Is Funded
    • Our Team
You are here: Home / The Complete List Of How-To Tutorials For Internet Dummies / The Interactive Glossary: Defining the Net / Spider Definition | The Interactive Glossary

Spider Definition | The Interactive Glossary

Disclosure: Your support helps keep the site running! We earn a referral fee for some of the services we recommend on this page. Learn more

Contents

  • 1 Spider
  • 2 Frequently Asked Questions
    • 2.1 Why are spiders used to build search engine databases as opposed to manually built databases?
    • 2.2 Are there any dangers associated with spiders automatically crawling a site?
  • 3 Glossary Index

Spider

A spider is a computer program used by a search engine to systematically browse and index websites on the World Wide Web.

Spiders go by many names such as web crawler, bot, web spider, ant, and automatic indexer. Regardless of the name used, spiders have one goal: to browse websites and create a thorough index of the content and links present on the site.

The way a webcrawler goes about the business of browsing websites depends on a combination of four policies:

  • Selection: Search engines cannot keep up with all of the new content that appears on the web, or with the sheer massive size of the web. As a result, search engines, and the spiders they send out, must prioritize which sites to crawl and in what order.
  • Revisit: Some websites are updated regularly, while others are only updated every few years if at all. Search engines have to determine the time interval at which they should revisit a webpage and update the indexed record of the site.
  • Politeness: Because spiders don’t require the same sort of time a human would require to process a webpage, they can request pages much more quickly than a normal website visitor. As a result, if a spider doesn’t choose to behave politely, it can send multiple requests every second, and cripple a web server in the process. Search engines must program their spiders to be appropriately polite and to space their requests out in time intervals that a web server can handle.
  • Parallelization: All search engines have multiple web crawling processes in place at the same time, so they must have policies in place to maximize the effectiveness of their spiders and to avoid having different processes crawling the same sites.

Every search engine prioritizes and crafts these strategies a little differently. The result is that the search results generated by one search engine will always look a little different than those generated by a different search engine.

Also See: Search Engine, How Search Engines Work, Searching the Web

Frequently Asked Questions

Why are spiders used to build search engine databases as opposed to manually built databases?

Spiders can make requests from a server incredibly quickly – sending multiple requests per second if not programmed to be polite as was already mentioned. Humans, on the other hand, need several seconds or several minutes to scan a webpage and create a directory record based on what they see. In addition, spiders are better at providing detailed and extensive records, and are much cheaper than a human database-building workforce.

At one time, websites were found by looking in web directories, which were manually compiled databases in which websites were organized by subject. However, once spider-indexed search engines came along, directories simply couldn’t keep up with the massive size of spider-indexed databases, or the detailed records that were searchable by spider-indexed search engines, and directories quickly fell by the wayside.

Are there any dangers associated with spiders automatically crawling a site?

Spiders are very thorough, to the point that they can unintentionally expose security vulnerabilities. A practice known as Google Hacking involves making very specific advanced queries in a search engine to identify websites with certain security vulnerabilities. The identified websites are then targeted with hacking techniques based on the vulnerabilities exposed.


Glossary Index

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Latest Posts

Hosting

Black Friday 2023: Website Builder & Web Hosting Deals For Webmasters [Updated]

Uncategorized

Why WebsiteBuilders.com Got Hacked (Yes, You Can Blame Social Warfare)

E-mail Internet

E-Mail Etiquette: 12 Basic Rules For Politely Using E-Mail

Marketing social networking

Live Chat Software: Should You Invest In It? (It’s Easier Than You Think)

Computer Science

Never Heard Of Algorithmic Warfare? Then You’d Better Read This

Marketing

How To Successfully Promote Your Pet Business Online With Minimum Effort

Most Popular Reviews

  • Website Builder Reviews
  • Wix
  • Weebly
  • Squarespace
  • GoDaddy Website Builder
  • Shopify

Recent Posts

  • Black Friday 2023: Website Builder & Web Hosting Deals For Webmasters [Updated] November 5, 2019
  • Why WebsiteBuilders.com Got Hacked (Yes, You Can Blame Social Warfare) March 23, 2019
  • E-Mail Etiquette: 12 Basic Rules For Politely Using E-Mail November 15, 2017

About WebsiteBuilders.com

Everything you need to know about the WebsiteBuilders.com team.

About Us

Our Team

How The Website Is Funded

Privacy Policy

© 2017-2022 Website Builders.com | Sitemap