senaduka/old-web
1
0
Fork 0
Code Issues Pull Requests 5 Actions Packages Projects Releases Wiki Activity

add crawler and crawler config for Aktido agency

Browse Source
This commit is contained in:
Bilal Catic
2019-10-28 09:14:45 +01:00
parent f5d912f02c
commit f899c96dc6
6 changed files with 438 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
development.env
View File

@@ -45,3 +45,10 @@ PROSTOR_CRAWLER_AD_TYPE=enum name of what type of ads should be crawled, check c
PROSTOR_CRAWLER_AD_CATEGORIES=comma separated list of enum names of categories to be included, check common/enums.js file for valid values
PROSTOR_IGNORED_USERNAMES=!!! This is not used for prostor crawler !!!
PROSTOR_DELAY_BETWEEN_PAGES=!!! This is not used for prostor crawler !!!
#==AKTIDO==
AKTIDO_MAX_PAGES=Restrict crawler to this number of pages
AKTIDO_MAX_RESULTS_PER_PAGE=Only this number or less results from one page will be scraped and saved
AKTIDO_CRAWLER_AD_TYPE=enum name of what type of ads should be crawled, check common/enums.js file for valid values
AKTIDO_CRAWLER_AD_CATEGORIES=comma separated list of enum names of categories to be included, check common/enums.js file for valid values
AKTIDO_IGNORED_USERNAMES=!!! This is not used for aktido crawler !!!
AKTIDO_DELAY_BETWEEN_PAGES=time in miliseconds to wait before indexing next page