使用wget或curl来测试网站的.htaccess + robots.txt

Question

我想调试我的网站的.htaccess + robots.txt，我想用cURL或wget尝试访问使用robots.txt阻止的文件，或者应该通过的.htaccess重定向到另一个位置的网页

我已经在我的robots.txt以下尚未

User-agent: * 
Disallow: /wp/wp-admin/ 

，我仍然能够抓取

wget的

$ wget http://xxxx.com/wp/wp-admin/ 
SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc 
syswgetrc = C:\Program Files (x86)\GnuWin32/etc/wgetrc 
--2017-08-28 07:37:05-- http://xxxx.com/wp/wp-admin/ 
Resolving xxxx.com... 118.127.47.249 
Connecting to xxxx.com|118.127.47.249|:80... connected. 
HTTP request sent, awaiting response... 302 Found 
Location: http://xxxx.com/wp/wp-login.php?redirect_to=http%3A%2F%2Fxxxx.com%2Fwp%2Fwp- 
admin%2F&reauth=1 [following] 
--2017-08-28 07:37:12-- http://xxxx.com/wp/wp-login.php?redirect_to=http%3A%2F%2Fxxxx.com%2Fwp%2Fwp-admin%2F&reauth=1 
Connecting to xxxx.com|118.127.47.249|:80... connected. 
HTTP request sent, awaiting response... 200 OK 
Length: 2891 (2.8K) [text/html] 
Saving to: `[email protected]_to=http%3A%2F%2Fxxxx.com%2Fwp%2Fwp-admin%2F&reauth=1' 

100%[==============================================================================>] 2,891  --.-K/s in 0.1s 

2017-08-28 07:37:17 (22.2 KB/s) - `[email protected]_to=http%3A%2F%2Fxxxx.com%2Fwp%2Fwp-admin%2F&re 
auth=1' saved [2891/2891] 

卷曲

$ curl -L xxx.com/wp/wp-admin -o wp-admin.html 
% Total % Received % Xferd Average Speed Time Time  Time Current 
           Dload Upload Total Spent Left Speed 
100 1147 100 1147 0  0 107  0 0:00:10 0:00:10 --:--:-- 280 
0  0 0  0 0  0  0  0 --:--:-- 0:01:37 --:--:--  0 
100 2891 100 2891 0  0  17  0 0:02:50 0:02:42 0:00:08 234 

既不的wget也不卷曲尊敬的robots.txt 有没有一种方法来检查我有多么的.htaccess +的robots.txt？谢谢！

Answer 1

的robots.txt纯粹是为了搜索引擎机器人，它被大多数用户的浏览器[包括wget和卷曲]，如果您要检查您的robots.txt是可分析的，你可以使用谷歌的检查中忽略站长控制台，其中显示您的robots.txt文件可能存在的任何错误和问题。

使用.htaccess的重定向应该适用于任何浏览器，并且wget应显示这些重定向。