senaduka/old-web
1
0
Fork 0
Code Issues Pull Requests 5 Actions Packages Projects Releases Wiki Activity

Added Scraper API option. #100

Merged
RabbIT09-n merged 3 commits from scraper-api-support into master 2020-02-25 16:10:44 +01:00
Conversation 17 Commits 3 Files Changed 3 +20 -3
RabbIT09-n commented 2020-02-22 22:22:20 +01:00 (Migrated from gitlab.com)
Copy Link

Added option to turn on Scraper API through env variables. Just modified url to be crawled/scraped. API key also configurable.

Added option to turn on Scraper API through env variables. Just modified url to be crawled/scraped. API key also configurable.
senaduka commented 2020-02-23 04:43:24 +01:00 (Migrated from gitlab.com)
Copy Link

SHould default to USE - it is more secure option. Scraping directly is less secure so if someone forgets to set the variables it is better that scraping fails than the server being exposed to the content providers.

SHould default to USE - it is more secure option. Scraping directly is less secure so if someone forgets to set the variables it is better that scraping fails than the server being exposed to the content providers.
RabbIT09-n commented 2020-02-23 16:39:06 +01:00 (Migrated from gitlab.com)
Copy Link

changed this line in version 2 of the diff

changed this line in [version 2 of the diff](/saburly/marketalarm/web/-/merge_requests/100/diffs?diff_id=76792682&start_sha=d436d4a37bd73a4dad5f25e47b685d1183aaf120#c72f9a98e9ca3b5472a2a7eaf30c6155d3600e77_48_48)
RabbIT09-n commented 2020-02-23 16:39:07 +01:00 (Migrated from gitlab.com)
Copy Link

added 1 commit

Compare with previous version

added 1 commit <ul><li>148b2ea8 - Changed default.</li></ul> [Compare with previous version](/saburly/marketalarm/web/-/merge_requests/100/diffs?diff_id=76792682&start_sha=d436d4a37bd73a4dad5f25e47b685d1183aaf120)
RabbIT09-n commented 2020-02-23 16:39:51 +01:00 (Migrated from gitlab.com)
Copy Link

Changed default as suggested.

Changed default as suggested.
senaduka commented 2020-02-23 19:33:14 +01:00 (Migrated from gitlab.com)
Copy Link

newUrl - bad name. Why is this url used ?

newUrl - bad name. Why is this url used ?
RabbIT09-n commented 2020-02-23 20:44:41 +01:00 (Migrated from gitlab.com)
Copy Link

scraperApiUrl - better ?

scraperApiUrl - better ?
senaduka commented 2020-02-23 21:04:22 +01:00 (Migrated from gitlab.com)
Copy Link

better!

better!
senaduka commented 2020-02-23 21:05:31 +01:00 (Migrated from gitlab.com)
Copy Link

but is it better ? If USE_SCRAPER_API is false then that url does not contain scraperApiUrl ...

but is it better ? If `USE_SCRAPER_API` is false then that url does not contain scraperApiUrl ...
RabbIT09-n commented 2020-02-23 21:15:28 +01:00 (Migrated from gitlab.com)
Copy Link

Right, urlAdaptedForScraping ? if USE_SCRAPER_API is false then it is "ordinary url"; if it is true then it is from scraper api

Right, urlAdaptedForScraping ? if USE_SCRAPER_API is false then it is "ordinary url"; if it is true then it is from scraper api
senaduka commented 2020-02-23 21:24:22 +01:00 (Migrated from gitlab.com)
Copy Link

Ok that's better ! 😁

Ok that's better ! 😁
edazdarevic commented 2020-02-23 22:39:22 +01:00 (Migrated from gitlab.com)
Copy Link

I second Senad's comments. Looks good otherwise.

I second Senad's comments. Looks good otherwise.
RabbIT09-n commented 2020-02-23 23:11:41 +01:00 (Migrated from gitlab.com)
Copy Link

changed this line in version 3 of the diff

changed this line in [version 3 of the diff](/saburly/marketalarm/web/-/merge_requests/100/diffs?diff_id=76804232&start_sha=148b2ea863178267f97a45d8d125001ac1d35a71#b497d48d079a75c2aeaf501b33f5b0537711062b_14_14)
RabbIT09-n commented 2020-02-23 23:11:42 +01:00 (Migrated from gitlab.com)
Copy Link

added 1 commit

Compare with previous version

added 1 commit <ul><li>ce857ddc - Renamed var.</li></ul> [Compare with previous version](/saburly/marketalarm/web/-/merge_requests/100/diffs?diff_id=76804232&start_sha=148b2ea863178267f97a45d8d125001ac1d35a71)
RabbIT09-n commented 2020-02-23 23:14:58 +01:00 (Migrated from gitlab.com)
Copy Link

Renamed var to urlAdaptedForScraping

Renamed var to urlAdaptedForScraping
edazdarevic commented 2020-02-25 13:53:48 +01:00 (Migrated from gitlab.com)
Copy Link

🚢 🇮🇹

:ship: :it:
RabbIT09-n commented 2020-02-25 16:10:41 +01:00 (Migrated from gitlab.com)
Copy Link

mentioned in commit 08ad9edfe1

mentioned in commit 08ad9edfe1377d6c3fc981c436d6143951929b8e
RabbIT09-n commented 2020-02-25 16:10:44 +01:00 (Migrated from gitlab.com)
Copy Link

merged

merged
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
senaduka
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: senaduka/old-web#100
Delete Branch "scraper-api-support"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?