senaduka/old-web
1
0
Fork 0
Code Issues Pull Requests 5 Actions Packages Projects Releases Wiki Activity

Crawler optimisation #20

Merged
nedimu merged 2 commits from crawler-optimisation into master 2019-07-08 10:01:49 +02:00
Conversation 3 Commits 2 Files Changed 2 +261 -148
nedimu commented 2019-07-05 17:16:23 +02:00 (Migrated from gitlab.com)
Copy Link

Smoke test

  1. Run crawler

  2. Find link of your realestate request and open int in browser, check how many pages your result generate

  3. Find Final crawler results number, it should contain same nnumber od results as olx page(1 page 30 results) page number * 30

  4. Notice HUGE Improvment on crawler speed. multiple request are sent, no more wating for one request to finish before other starts.

Smoke test 1) Run crawler 2) Find link of your realestate request and open int in browser, check how many pages your result generate 3) Find Final crawler results number, it should contain same nnumber od results as olx page(1 page 30 results) page number * 30 4) Notice HUGE Improvment on crawler speed. multiple request are sent, no more wating for one request to finish before other starts.
nedimu commented 2019-07-05 17:18:54 +02:00 (Migrated from gitlab.com)
Copy Link

added 1 commit

  • 039b1a63 - Optimiset crawlers , and pagingation

Compare with previous version

added 1 commit <ul><li>039b1a63 - Optimiset crawlers , and pagingation</li></ul> [Compare with previous version](/saburly/marketalarm/web/merge_requests/20/diffs?diff_id=47425233&start_sha=deb5e7f9cef08b32f23243b5d255270385da03b4)
bilal.catic commented 2019-07-08 10:01:49 +02:00 (Migrated from gitlab.com)
Copy Link

merged

merged
bilal.catic commented 2019-07-08 10:01:49 +02:00 (Migrated from gitlab.com)
Copy Link

mentioned in commit 2dd1eaa5fd

mentioned in commit 2dd1eaa5fdc929c0d5d78a1ef566962c0ce4ada5
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
senaduka
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: senaduka/old-web#20
Delete Branch "crawler-optimisation"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?