5 Commits

Author	SHA1	Message	Date
Bilal Catic	e3e47345bc	load AWS config through app config; fix ENV path	2019-09-30 09:44:19 +02:00
Bilal Catic	f93d0e738f	add delay between pages config variable	2019-09-25 08:55:00 +02:00
Bilal Catic	06d35fcb4b	move ignored usernames config to crawler specific config	2019-09-25 08:55:00 +02:00
Bilal Catic	3140fdf0c0	use function generator to index pages; crawl in parallel	2019-09-25 08:55:00 +02:00
Bilal Catic	3d46c82d3d	create new crawler and Postgres saver	2019-09-18 15:32:48 +02:00