5 Commits

Author	SHA1	Message	Date
Christian Mehlmauer	7ec095d708	fix duplicate robots.txt entries	2015-08-18 15:55:10 +02:00
erwanlr	b5d5c4177d	Removes potential spaces in robots.txt entries - Ref #819	2015-05-08 09:50:51 +01:00
erwanlr	c8c126d444	Rspec 3.0 support	2014-06-02 22:06:49 +02:00
Christian Mehlmauer	d107613e40	fix issue #262	2013-08-12 22:01:24 +02:00
Christian Mehlmauer	6c008015e9	parse robots.txt	2013-08-10 11:35:17 +02:00