Web crawler | Solutional Webdesign Antwerpen

ALLE 0-9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

A web crawler, also referred to as a spider or ant, is a tool that systematically navigates the internet^[1] to gather and index information from web pages. Starting from a base list of URLs, known as seeds, it follows links to collect and store content. This data is then archived in a repository for later use. Crucial to the functionality of search engines, web crawlers help in fetching, parsing, and storing web data to keep databases up-to-date. They operate under set policies that guide their selection, revisit, politeness, and parallelization actions. Algorithms and optimization techniques are used to enhance their efficiency, and they also face challenges like handling spam and duplicate content. Their identification is vital for preventing server overloads and for security^[2] purposes, as they pose a risk of data breaches when indexing sensitive resources.

Terms definitions

1. ↑ internet. The Internet is a global system of interconnected computer networks that use standardized communication protocols, primarily the TCP/IP, to link devices worldwide. Originating from the term 'internetted' used in 1849, the term 'Internet' was later used by the US War Department in 1945. Its development began with computer scientists creating time-sharing systems in the 1960s and further progressed with the establishment of ARPANET in 1969. The Internet is self-governed, without a central authority, and its principal name spaces are administered by the Internet Corporation for Assigned Names and Numbers (ICANN). It has significantly transformed traditional communication media and has grown exponentially over the years, with internet users increasing 20% to 50% annually. In 2019, over half of the world population used the Internet. The Internet protocol suite, which includes TCP/IP and four conceptual layers, guides internet packets to their destinations. Essential services like email and Internet telephony operate on the Internet. The World Wide Web, a global collection of interconnected documents, is a key component of the Internet.

2. ↑ security. Security, as a term, originates from the Latin 'securus,' meaning free from worry. It is a concept that refers to the state of being protected from potential harm or threats. This protection can apply to a wide range of referents, including individuals, groups, institutions, or even ecosystems. Security is closely linked with the environment of the referent and can be influenced by different factors that can make it either beneficial or hostile. Various methods can be employed to ensure security, including protective and warning systems, diplomacy, and policy implementation. The effectiveness of these security measures can vary, and perceptions of security can differ widely. Important security concepts include access control, assurance, authorization, cipher, and countermeasures. The United Nations also plays a significant role in global security, focusing on areas like soil health and food security.

Verwante artikelen:

Glossary

Web crawler (Wikipedia)

A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).

Web search engines and some other websites use Web crawling or spidering software to update their web content or indices of other sites' web content. Web crawlers copy pages for processing by a search engine, which indexes the downloaded pages so that users can search more efficiently.

Crawlers consume resources on visited systems and often visit sites unprompted. Issues of schedule, load, and "politeness" come into play when large collections of pages are accessed. Mechanisms exist for public sites not wishing to be crawled to make this known to the crawling agent. For example, including a robots.txt file can request bots to index only parts of a website, or nothing at all.

The number of Internet pages is extremely large; even the largest crawlers fall short of making a complete index. For this reason, search engines struggled to give relevant search results in the early years of the World Wide Web, before 2000. Today, relevant results are given almost instantly.

Crawlers can validate hyperlinks and HTML code. They can also be used for web scraping and data-driven programming.

" Terug naar Woordenlijst Index