今天在日志中发现了新浪IaskSpider的踪迹： 219.142.78.210 - - [10/Dec/2005:00:00:36 +0800] “GET /phpMan.php/man/crond/8/ HTTP/1.0” 200 3926 “-” “iaskspider” 219.142.78.2 10.114661134144036588 219.142.78.210 - - [10/Dec/2005:00:00:36 +0800] “GET /phpMan.php/man/cytune/8/ HTTP/1.0” 200 8639 “-” “iaskspider” 219.142.78. 210.302381134144036603 219.142.118.65 - - [10/Dec/2005:00:00:50 +0800] “GET /phpMan.php/info/pathchk/ HTTP/1.0” 200 3136 “-” “iaskspider” 219.142.118 .65.3811134144050430 219.142.78.85 - - [10/Dec/2005:00:00:58 +0800] “GET /phpMan.php/man/Config::myconfig/ HTTP/1.0” 200 1998 “-” “iaskspider” 219. 142.78.85.302351134144058586

IaskSpider的访问来源和频度分布： grep -i iaskspider *|awk ‘{print $1}’ |awk -F ‘:’ ‘{print $2}’ |sort|uniq -c|sort -rn |head -10 490 219.142.78.87 444 219.142.118.62 429 219.142.78.210 390 219.142.118.63 355 219.142.118.65 352 219.142.78.59 320 219.142.78.85 306 219.142.118.64 304 219.142.78.60 247 219.142.78.179

和国内外其他主流搜索引擎的SPIDER不一样：Sohu和新浪的都不抓取robots.txt文件，这样就有可能存在一些不希望被搜索引擎收录的内容被公开的潜在威胁。

附：AWStats中增加针对Sohu Agent和IaskSpider的统计 Index: robots.pm

RCS file: /cvsroot/awstats/awstats/wwwroot/cgi-bin/lib/robots.pm,v retrieving revision 1.40 diff -r1.40 robots.pm 214a215

‘iaskspider’, 321a323 ‘sohu’, 718a721 ‘iaskspider’,‘Sina Iask’, 724a728 ‘sohu’,‘SohuSpider’,

国内搜索引擎的定义：

RCS file: /cvsroot/awstats/awstats/wwwroot/cgi-bin/lib/search_engines.pm,v retrieving revision 1.35 diff -r1.35 search_engines.pm 94c94,96 < ‘baidu.com’,‘search.sina.com’,‘search.sohu.com’,

‘baidu.com’,‘search.sohu.com’,‘iask.com’, ‘.3721.com’,’.163.com’,‘search.tom.com’,‘search.21cn.com’,’.qq.com’, ‘yisou.com’,‘search.china.com’,‘zhongsou.com’, ‘search.online.’, ‘sogou.com’, 203c205 < ‘search.sina.com’,‘sina’,

‘iask.com’,‘iask’, 204a207,216 ‘.3721.com’,‘3721’, ‘.163.com’,’netease’, ‘search.tom.com’,’tom’, ‘search.21cn.com’,‘21cn’, ‘.qq.com’,‘qq’, ‘yisou.com’,‘yisou’, ‘search.china.com’,‘china’, ‘zhongsou.com’,‘zhongsou’, ‘search.online’,‘shonline’, ‘sogou.com’,‘sogou’, 360c372,374 < ‘baidu’,‘word=’, ‘sina’, ‘word=’, ‘sohu’,‘word=’,

‘baidu’,’(bs|wd|word)=’, ‘iask’,’(_searchkey|k)’, ‘sohu’,’(query|key_word|word)=’, ‘3721’,’(p|name)=’, ’netease’,‘q=’, ’tom’,‘word=’, ‘21cn’,‘word=’, ‘qq’,’(word|w)=’, ‘yisou’,‘p=’, ‘china’,‘query=’, ‘zhongsou’,‘word=’, ‘shonline’,‘word=’, ‘sogou’, ‘query=’, 473c487,490 < ‘baidu’,‘Baidu’, ‘sina’,‘Sina’, ‘sohu’,‘Sohu’,

‘baidu’,‘Baidu’, ‘iask’,‘Iask’, ‘sohu’,‘Sohu’, ‘3721’,‘3721’, ’tom’,‘Tom’, ’netease’, ‘NetEase’, ‘21cn’,‘21CN’, ‘qq’,‘QQ’, ‘yisou’,‘YiSou’, ‘zhongsou’,‘ZhougSou’, ‘shonline’,‘SHOnline’,

💬 历史评论

以下 5 条评论从旧版 Movable Type 博客迁移而来。原始评论时间：2002–2016 年。

shunz · 2005-12-12 15:59

219.142开头的这几个ip曾经疯狂抓取我的几个网站:(

Fenng · 2005-12-12 19:04

我用 http://www.antezeta.com/ 下载的 Robot 和 search_engines.pm ,加上你定义的国内搜索搜索引擎的定义．基本上比较全了

windtear · 2005-12-13 11:20

有时间我看看log关注一下现在只看pv系统了还真没注意访问log

sohu 的似乎抓 robots.txt 至少来自 220.181.26.74 的抓

GET //robots.txt HTTP/1.0 User-Agent: sohu-search Accept-Language: zh-cn X-Forwarded-For: 220.181.26.74 Cache-Control: max-age=36288000 Connection: keep-alive

BunnyQ · 2005-12-13 11:56

en 最近我也发现iaskspider踪迹不知它想抓走什么..

七彩 · 2006-01-07 11:03

请问一下: 这个又是什么呢? 2006-01-05 04:59:47 222.62.229.248 GET 404 3919 107 InetURL:/1.0 - 2006-01-05 07:35:40 221.137.190.189 GET 302 703 115 - - 2006-01-05 11:24:10 202.165.102.179 GET 200 16593 107 SpiderMan -