Created non-concurrent request groups.
This commit is contained in:
@@ -25,6 +25,8 @@ API_MAP_KEY=(your-key-here)
|
||||
#=============== SCRAPER API SUPORT =============#
|
||||
USE_SCRAPER_API= To turn it on (1) or off (0)
|
||||
SCRAPER_API_KEY= Key for Scraper api
|
||||
NUMBER_OF_CONCURRENT_REQ_SCRAPER_API= Number of requests to send concurrently to Srcaper API proxy
|
||||
DELAY_BETWEEN_REQ_SCRAPER_API= time in miliseconds to wait before sending next req bulk to awoid server errors with Scraper API
|
||||
|
||||
#=============== AWS SDK EMAIL SETTINGS =======#
|
||||
AWS_KEY_ID=(your-key-here)
|
||||
@@ -36,6 +38,7 @@ SOURCE_EMAIL=info@saburly.com
|
||||
CRAWLER_INTERVAL=Interval to run cralwer(s), in seconds
|
||||
STOP_CRAWLER=Non-zero value will skip crawler execution
|
||||
PRINT_CRAWLER_DEBUG_INFO=Non-zero value will print crawler debugging info to the server console
|
||||
|
||||
#==OLX==
|
||||
OLX_MAX_PAGES=Restrict crawler to this number of pages
|
||||
OLX_MAX_RESULTS_PER_PAGE=Only this number or less results from one page will be scraped and saved
|
||||
@@ -43,7 +46,6 @@ OLX_CRAWLER_AD_TYPE=enum name of what type of ads should be crawled, check commo
|
||||
OLX_CRAWLER_AD_CATEGORIES=comma separated list of enum names of categories to be included, check common/enums.js file for valid values
|
||||
OLX_IGNORED_USERNAMES=comma separated list of usernames to ignore
|
||||
OLX_DELAY_BETWEEN_PAGES=time in miliseconds to wait before indexing next page
|
||||
OLX_DELAY_BETWEEN_ADS = time in miliseconds to wait before scraping next add to awoid server errors with Scraper API
|
||||
OLX_FORCE_CRAWL=Non-zero value will force crawler to crawl all pages without stopping when known real estate is found
|
||||
|
||||
#==RENTAL==
|
||||
|
||||
Reference in New Issue
Block a user