Christian Mehlmauer 5ea911c9b3 Header names are case insensitive ...

Move header checks to web_site

2013-08-10 11:49:30 +02:00

..

interesting_headers.rb

Header names are case insensitive

2013-08-10 11:49:30 +02:00

robots_txt.rb

parse robots.txt

2013-08-10 11:35:17 +02:00