A Note on Web Crawlers

Here are a few links that touch on a few of the ideas or topics we discussed today:

Check out http://www.drunkmenworkhere.org/219 to see how different search engines crawl the ‘webspider tree of death’.

Webrings: http://dir.webring.org/rw  (http://www.webring.org/hub/drtvgw for example)

Directories from the past: http://dir.yahoo.com/

 

 

 

 

 

 

 

 

 

 

About these ads

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

%d bloggers like this: