Christian Mehlmauer
|
7ec095d708
|
fix duplicate robots.txt entries
|
2015-08-18 15:55:10 +02:00 |
|
erwanlr
|
b5d5c4177d
|
Removes potential spaces in robots.txt entries - Ref #819
|
2015-05-08 09:50:51 +01:00 |
|
erwanlr
|
c8c126d444
|
Rspec 3.0 support
|
2014-06-02 22:06:49 +02:00 |
|
Christian Mehlmauer
|
d107613e40
|
fix issue #262
|
2013-08-12 22:01:24 +02:00 |
|
Christian Mehlmauer
|
5ea911c9b3
|
Header names are case insensitive
Move header checks to web_site
|
2013-08-10 11:49:30 +02:00 |
|
Christian Mehlmauer
|
6c008015e9
|
parse robots.txt
|
2013-08-10 11:35:17 +02:00 |
|