今天在日志中发现了新浪IaskSpider的踪迹： 219.142.78.210 - - [10/Dec/2005:00:00:36 +0800] “GET /phpMan.php/man/crond/8/ HTTP/1.0” 200 3926 “-” “iaskspider” 219.142.78.2 10.114661134144036588 219.142.78.210 - - [10/Dec/2005:00:00:36 +0800] “GET /phpMan.php/man/cytune/8/ HTTP/1.0” 200 8639 “-” “iaskspider” 219.142.78. 210.302381134144036603 219.142.118.65 - - [10/Dec/2005:00:00:50 +0800] “GET /phpMan.php/info/pathchk/ HTTP/1.0” 200 3136 “-” “iaskspider” 219.142.118 .65.3811134144050430 219.142.78.85 - - [10/Dec/2005:00:00:58 +0800] “GET /phpMan.php/man/Config::myconfig/ HTTP/1.0” 200 1998 “-” “iaskspider” 219. 142.78.85.302351134144058586

IaskSpider的访问来源和频度分布： grep -i iaskspider *|awk ‘{print $1}’ |awk -F ‘:’ ‘{print $2}’ |sort|uniq -c|sort -rn |head -10 490 219.142.78.87 444 219.142.118.62 429 219.142.78.210 390 219.142.118.63 355 219.142.118.65 352 219.142.78.59 320 219.142.78.85 306 219.142.118.64 304 219.142.78.60 247 219.142.78.179

和国内外其他主流搜索引擎的SPIDER不一样：Sohu和新浪的都不抓取robots.txt文件，这样就有可能存在一些不希望被搜索引擎收录的内容被公开的潜在威胁。

附：AWStats中增加针对Sohu Agent和IaskSpider的统计 Index: robots.pm

RCS file: /cvsroot/awstats/awstats/wwwroot/cgi-bin/lib/robots.pm,v retrieving revision 1.40 diff -r1.40 robots.pm 214a215

‘iaskspider’, 321a323 ‘sohu’, 718a721 ‘iaskspider’,‘Sina Iask’, 724a728 ‘sohu’,‘SohuSpider’,

国内搜索引擎的定义：

RCS file: /cvsroot/awstats/awstats/wwwroot/cgi-bin/lib/search_engines.pm,v retrieving revision 1.35 diff -r1.35 search_engines.pm 94c94,96 < ‘baidu.com’,‘search.sina.com’,‘search.sohu.com’,

‘baidu.com’,‘search.sohu.com’,‘iask.com’, ‘.3721.com’,’.163.com’,‘search.tom.com’,‘search.21cn.com’,’.qq.com’, ‘yisou.com’,‘search.china.com’,‘zhongsou.com’, ‘search.online.’, ‘sogou.com’, 203c205 < ‘search.sina.com’,‘sina’,

‘iask.com’,‘iask’, 204a207,216 ‘.3721.com’,‘3721’, ‘.163.com’,’netease’, ‘search.tom.com’,’tom’, ‘search.21cn.com’,‘21cn’, ‘.qq.com’,‘qq’, ‘yisou.com’,‘yisou’, ‘search.china.com’,‘china’, ‘zhongsou.com’,‘zhongsou’, ‘search.online’,‘shonline’, ‘sogou.com’,‘sogou’, 360c372,374 < ‘baidu’,‘word=’, ‘sina’, ‘word=’, ‘sohu’,‘word=’,

‘baidu’,’(bs|wd|word)=’, ‘iask’,’(_searchkey|k)’, ‘sohu’,’(query|key_word|word)=’, ‘3721’,’(p|name)=’, ’netease’,‘q=’, ’tom’,‘word=’, ‘21cn’,‘word=’, ‘qq’,’(word|w)=’, ‘yisou’,‘p=’, ‘china’,‘query=’, ‘zhongsou’,‘word=’, ‘shonline’,‘word=’, ‘sogou’, ‘query=’, 473c487,490 < ‘baidu’,‘Baidu’, ‘sina’,‘Sina’, ‘sohu’,‘Sohu’,

‘baidu’,‘Baidu’, ‘iask’,‘Iask’, ‘sohu’,‘Sohu’, ‘3721’,‘3721’, ’tom’,‘Tom’, ’netease’, ‘NetEase’, ‘21cn’,‘21CN’, ‘qq’,‘QQ’, ‘yisou’,‘YiSou’, ‘zhongsou’,‘ZhougSou’, ‘shonline’,‘SHOnline’,