3 Commits

Author	SHA1	Message	Date
Christian Mehlmauer	d107613e40	fix issue #262	2013-08-12 22:01:24 +02:00
Christian Mehlmauer	5ea911c9b3	Header names are case insensitive ... Move header checks to web_site	2013-08-10 11:49:30 +02:00
Christian Mehlmauer	6c008015e9	parse robots.txt	2013-08-10 11:35:17 +02:00