The study - 'the largest ever to be conducted on the topography of the Web' - has blown apart the theory that the web is 'effectively' small and easy to connect from one site to virtually any other. One study suggested that every site can be found in just nineteen steps.
Instead, the new research paints a picture of an elite core of websites that are meshed together with hyperlinks with less fortunate satellite clusters that represent dead ends.
According to IBM Research, Compaq Corporate Research Laboratories and AltaVista who collaborated in the research, almost 10% of all web pages are 'completely isolated'. Another 22% are 'disconnected' pages that are not accessible from the 'strongly connected core' of sites that form about one third of the entire web.
Using the results from the study, the scientists have developed the Bow Tie Theory of the web which attempts to explain what the web 'looks' like. In the theory, the core set of interconnected sites is flanked on one side by 'origination' sites. These represent about one quarter of all web pages and allow surfers to reach the core from these pages but you cannot travel to origination pages from the centre.
Sites in the other flank have been labelled 'termination'. You can surf here from the central core but cannot link back again. Other sites are even more isolated.
The researchers claim the work will enable the scientific and business community to design more effective web crawling strategies and increase the effectiveness of e-commerce through the design of more effective browsing, advertising and modelling.
The results of the study were presented to the 9th International World Wide Web Conference held in Amsterdam this month.
Free daily newsletter
If you like our news and feature articles, you can sign up to receive our free daily (Mon-Fri) email newsletter (mobile friendly).
