old-kiviscrapbalancer/README.md

kiviscrapbalancer

Kivi scraping load balancer

Setup

Copy example.env to the .env and set desired values

Deployment procedure

1. Compile source code with go build

2. Transfer generated binary to the Kivi load balancer server

   Command for sending files to the remote computer :

   scp -i path/to/the/key.pem path/to/the/binary username@host:/path/to/destination

   Path on the server where load balancer resides is : /home/ubuntu/kivi_scraper/load_balancer

3. Add .env file next to the binary on the server and set correct values

   Note: for listening addresses, use :1337 and :1338 or other ports but in this format, so the server can be reached from outside

   Note : make sure firewalls(on the server and one from the platform where server is hosted) is not blocking selected ports

4. If monit is not installed on the server, install it (sudo apt install monit)

5. Create starth.sh with contents

    #!/bin/sh
   
    cd /home/ubuntu/kivi_scraper/load_balancer
    ./kiviscrapbalancer
   

6. Create stop.sh with contents

    #!/bin/sh
   
    cd /home/ubuntu/kivi_scraper/load_balancer
    sudo kill `cat sample.pid`
   

7. Edit monit configuration with sudo nano /etc/monit/monitrc

8. Around line 208 is example how to set process check. Add new line below example

9. Add to the file :

       check process kivi_load_balancer with pidfile /home/ubuntu/kivi_scraper/load_balancer/sample.pid
         start program = "/home/ubuntu/kivi_scraper/load_balancer/start.sh"
         stop program  = "/home/ubuntu/kivi_scraper/load_balancer/stop.sh"
         if failed port 1337 protocol http then restart
         if failed port 1338 then restart
         if 5 restarts within 5 cycles then timeout
   

10. Reboot with sudo reboot

Notes

• To check monit logs, check /var/log/monit.log

• To check on which ports system is listening, use sudo lsof -i -P -n | grep LISTEN