[![CircleCI](https://circleci.com/gh/SunPower/Helix_Roof_Calculator.svg?style=svg&circle-token=aacf2ae59dae99075992ed10d1e27f119e223e75)](https://circleci.com/gh/SunPower/Helix_Roof_Calculator)


## Helix Calculator

- [Staging](https://sp-helix-staging.herokuapp.com)
- [PreProd](https://sp-helix-preprod.herokuapp.com)
- [Production](https://helix-calculator.sunpower.com/)


## Local Development and Testing with Docker

##### Dependencies

- Docker: [docs.docker.com/engine/installation](https://docs.docker.com/engine/installation/)
- CircleCi CLI: [Installing and using CircleCi locally](https://circleci.com/docs/2.0/local-jobs/#installing-the-cli-locally)

- Current CircleCi and development image: 
[ivannnn/heroku-cedar14-extras:2.0](https://hub.docker.com/r/ivannnn/heroku-cedar14-extras/)

| ivannnn/heroku-cedar14-extras:2.0 |
| ------------- |
| ruby 2.3.5p376 |
| Python 3.5.4   |
| postgresql-9.5  |   
| postgresql-client-9.5|   
| postgresql-client-9.6 |   
| postgresql-client-common  |   
| postgresql-common |   
| postgresql-contrib-9.5 |   
| postgresql-server-dev-9.6 |   
| redis-server/trusty,now 2:2.8.4-2 amd64 |
| redis-tools/trusty,now 2:2.8.4-2 amd64  |
| google-chrome-stable/stable,now 62.0.3202.94-1 amd64 |
| nodejs/unknown,now 6.12.0-1nodesource1 amd64 |


##### Environment Variables

Set the environment variables for your specific environment. Use the `.env` file below as reference:

```
AWS_S3_BUCKET="..."
AWS_ACCESS_KEY_ID="..."
AWS_SECRET_ACCESS_KEY="..."

SFDC_BASE_URL="https://test.salesforce.com"
SFDC_ACCESS_KEY_ID="..."
SFDC_SECRET_ACCESS_KEY="..."
SFDC_API_URL="https://sunpower--qa.cs8.my.salesforce.com"
```


##### Set up for Develop and Testing

- Run the docker container for the 1st time

```docker run -d --name helix -p 5000:5000 --expose 5000 -t ivannnn/heroku-cedar14-extras:2.0```

- Or re start the docker conatiner 

```docker container start helix```

- Copy the code to the container

```docker cp . helix:project```

- Log in the docker container

```docker exec -ti helix bash```

- Run this commands inside the docker conatiner

```
cd project && \
rm -rf env && \
gem install bundler && \
service postgresql start &&\
service redis-server start &&\
export PGPASSWORD=password &&\
export CIRCLE_TEST_REPORTS=$(pwd) &&\
python3.5 -m venv env &&\
source env/bin/activate &&\
pip install invoke &&\
invoke install &&\
npm install &&\
invoke db_migrate
```


##### Important
- Open another terminal tab, this script watches for changes and pass them 
to the docker conatiner, which will restart the web server
```./watch.sh``` 


##### Run the server
```invoke serve_debug```


##### Test locally without CircleCi
```invoke test_ci```


##### Test locally with CircleCi

- Go to the project dir
- ```circleci build```


## OS steps wihout Docker

#### macOS systems

This assumes you already have [homebrew](https://brew.sh) installed as well as an [ssh key added to your github account](https://help.github.com/articles/generating-an-ssh-key/).
This also assumes your username is pivotal, see the [circleci configuration](circleci.yml) for examples of how to add a pivotal user to postgres.

```sh
xcode-select --install
brew install python3 node nodenv redis phantomjs imagemagick@6 postgres brew-services
brew services start postgres
createdb test
createdb pivotal
git clone git@github.com:SunPower/Helix_Roof_Calculator.git
cd Helix_Roof_Calculator
pyvenv env
source env/bin/activate
pip install invoke
invoke install
npm install
invoke db_migrate
```

This installs python3, the [invoke utility](https://github.com/pyinvoke/invoke), xcode, node, imagemagick, redis, and postgres. It also sets up a virtual environment for the calculator, installs all the python dependencies into the virtual environment, and installs all the node dependencies.
Note that invoke should be installed after the virtual env is installed to ensure that the python 3.x version of invoke is used (instead of the system version, which is typically 2.7.x).

#### Linux/Debian-based systems

While most of this README assumes you're running macOS, for linux/debian based systems (e.g. ubuntu), follow these instructions. See the mac section for an explanation.

##### PostgreSQL

Postgres needs to be installed and configured before we do anything else.

```
sudo apt-get install postgresql postgresql-client postgresql-client-common postgresql-contrib libpq-dev
# Modify /etc/postgresql/common/*/pg_hba.conf and set local connections to trust
sudo -u postgres createdb test
sudo -u postgres createdb pivotal
# create pivotal role in postgres, grant it access to the test/pivotal databases.
echo "CREATE ROLE pivotal WITH UNENCRYPTED PASSWORD 'password';" | psql -U postgres
echo "ALTER ROLE pivotal WITH LOGIN;" | psql -U postgres
echo "GRANT SELECT, UPDATE, INSERT ON ALL TABLES IN SCHEMA test.public TO pivotal;" | psql -U postgres
echo "GRANT CREATE, CONNECT ON DATABASE test TO pivotal;" | psql -U postgres
echo "GRANT SELECT, UPDATE, INSERT ON ALL TABLES IN SCHEMA pivotal.public TO pivotal;" | psql -U postgres
echo "GRANT CREATE, CONNECT ON DATABASE pivotal TO pivotal;" | psql -U postgres
```

This installs postgres and creates the database and roles that helix looks for in test/local usage. Because ubuntu is systemd, you can start/stop/restart the server by running something along the lines of `sudo systemctl [start|stop|restart] postgresql.service`. Please see [this ubuntu postgres documentation page](https://help.ubuntu.com/community/PostgreSQL) for more information.

The comment for the second line tells you how to enable connections to postgres without requiring a password. See http://dba.stackexchange.com/questions/83164/remove-password-requirement-for-user-postgres for more information

##### Other dependencies/the project

```sh
sudo apt-get install git python3 python3-dev python3-venv nodejs npm redis-server imagemagick libmagickcore-dev libxml2-dev libxslt1-dev ruby ruby-dev
sudo gem install bundler
python3 --version  # make sure you have >3.5 installed!
git clone git@github.com:SunPower/Helix_Roof_Calculator.git
cd Helix_Roof_Calculator
pyvenv env
source env/bin/activate
pip install invoke
invoke install
npm install
invoke db_migrate
```

##### PhantomJS

```sh
sudo apt-get install build-essential chrpath libssl-dev libxft-dev libfreetype6 libfreetype6-dev libfontconfig1 libfontconfig1-dev
export PHANTOM_JS=phantomjs-2.1.1-linux-x86_64
wget https://bitbucket.org/ariya/phantomjs/downloads/$PHANTOM_JS.tar.bz2
sudo targ xvjf $PHANTOM_JS.tar.bz2

sudo mv $PHANTOM_JS /usr/local/share
sudo ln -sf /usr/local/share/$PHANTOM_JS/bin/phantomjs /usr/local/bin
```

Note that it doesn't work to use the nodejs-provided phantom, nor does apt-get install phantomjs work. Yes, it's annoying.

### Adding Dependencies

Because of how we're separating production dependencies and test dependencies, it's recommended that each new dependency be manually added to either the [requirements.txt](requirements.txt) or the [requirements.test.txt](requirements.test.txt), depending on whether it's a production or test dependency, respectively.

### Adding database migrations

To manage the database, we use [SQLAlchemy](http://www.sqlalchemy.org/).  
To manage database migrations, we use [Alembic](http://alembic.zzzcomputing.com/en/latest/).

First, make your changes to the sql models (in the models/sql) package to reflect your new vision of the sql schema. Next, we need to take those changes and generate an alembic migration from them. Run `PYTHONPATH=. alembic revision -m "(migration description)"` to generate that migration. Now, add the changes (and revert instructions!) to the generated migration file. Finally, run `invoke db_migrate` to run that migration.

##### Downgrading a migration

There currently is not an invoke task to revert migrations. Instead, just directly use the alembic cli to do this, something along the lines of `PYTHONPATH=.:$PYTHONPATH alembic downgrade -1`. Setting PYTHONPATH to include the current directory will ensure that the correct sql models get picked up when alembic does the migration. To redo a migration using alembic, run `PYTHONPATH=.:$PYTHONPATH alembic upgrade +1`.

Please refer to the [Alembic Documentation](http://alembic.zzzcomputing.com/en/latest/) for more information on using Alembic.

### Running Tests

First, install PhantomJS (`brew install phantomjs`) (this should already be done as part of the setup).

Run `invoke test` from the project directory will run every test.

Note that we use [Python's built-in unittest library](https://docs.python.org/3/library/unittest.html) for our python tests. Additionally, there are a few other helper libraries we use in tests. See the [requirements.test.txt](requirements.test.txt) file for those.

##### Running Javascript Tests

`invoke test` will, by default, run both the python tests and the javascript tests. Because the python tests can take a while, it's advantageous to be able to run the javascript tests (used for the array visualization) separately, especially when working entirely on the visualization frontend.

To do this, run `invoke test_js` from your command. This spins up an instance of [karma](https://karma-runner.github.io/0.13/index.html) attached to (by default) Chrome which watches the [spec](spec) and the [helix/javascript](helix/javascript) directories for changes, and (usually) re-runs the javascript tests if anything changes.

Note that we use [Jasmine](http://jasmine.github.io/2.4/introduction.html) for our javascript specs.

### Building JS

Part of the application (as of now only files in `helix/javascript/array_summary`) are built using Webpack and leveraging ES6.

To run the build use this command from the root of the repository
```bash
./node_modules/.bin/webpack
```

### Running Locally

First, install redis (`brew install redis`), and then (in a separate terminal) run `redis-server /usr/local/etc/redis.conf` to run redis locally.

Next, install postgres (`brew install postgres`), and then start it (`brew services start postgres`), follow the instructions at the top for additional configuration information.

Then, tell alembic to migrate your postgres db to the latest (`invoke db_migrate`).

Run `invoke serve` from the project directory to run locally.

This will run the project on [port 5000](http://localhost:5000/).

##### Running in debug mode

Run `invoke serve_debug` to start the project in debug mode.

##### The image tests

Note that you may see image comparison tests failing for erroneous reasons. These are flaky and machine-dependent. If you've done anything to the image generation code, please verify visually that you're getting the output you want.

### Circle CI

[CI (Continuous Integration)](https://en.wikipedia.org/wiki/Continuous_integration) is a service we use that automatically runs all the tests whenever a new commit is pushed. This is important because it runs the code in a context-free (the container the code runs in is set up for each run, then destroyed afterwards) environment. We're using [Circle CI](http://circleci.com/) here because Rachel recommended it. In addition to automatically running tests, circle also automatically deploys successful builds to cloud foundry for us.

### Deploying to Heroku (locally)

Install the Heroku CLI by running `brew install heroku` from your terminal.

Add heroku git endpoints for the different environments you want to push to. Typically, you'll just want to push to staging (especially seeing that preprod/production are automatically pushed to heroku for you). The heroku git endpoints are:

- staging: `https://git.heroku.com/sp-helix-staging.git`
- preprod: `https://git.heroku.com/sp-helix-preprod.git`
- production: `https://git.heroku.com/sp-helix-production.git`

login to heroku with `heroku login`

Finally, push to heroku using git! (Something like `git push heroku master`)

Note that circle ci does all that for us!

#### Deploying to preprod

To deploy to preprod, rebase master (up to the commit you want) onto preprod, then push preprod to github. CI will automatically deploy to preprod assuming that tests pass.

To push the current state of master to preprod:

```bash
git pull -r # Pull from origin (assuming origin is github)
git checkout preprod # switch to preprod branch
git rebase master # rebase head of master onto preprod
git push origin preprod # push preprod to github
```

To push a past commit to preprod:

```bash
git pull -r # Pull from origin (assuming origin is github)
git log
# [grab the sha of the commit you want]
git checkout preprod # switch to preprod branch
git rebase f00d # replace f00d with the sha of the commit you want
git push origin preprod # push preprod to github
```

#### Deploying to production

To deploy to production, tag a commit as `release-<number>` (for example, `release-2`) and push the commit (`git push origin release-2`). CI will automatically deploy to production assuming that tests pass.

Please increment the release count each time.

### Setting up PyCharm

[PyCharm](https://www.jetbrains.com/pycharm/) is an IDE from Jetbrains for python. It can be installed by running `brew cask install pycharm`.

Open pycharm, input a license, and select to install the command line tool. Then, open the project in pycharm by running `charm .` from the project directory.

Right now, you can start to use pycharm as an editor, but it's not very useful as an IDE, there's a bit more configuration stuff before it's useful as an IDE.

#### Use the correct Python

By default, PyCharm assumes you're using Python 2.7.x, we need to tell it to not only use 3.5.x, but use the 3.5.x for our local virtual environment.

To do this, open the preferences window, and select `Project: Helix_Roof_Calculator` (our project directory is named helix instead of Helix_Roof_Calculator, so the screenshots will be different than what you see), then select Project Intepreter and change that to `3.5.2 virtualenv at [projectdirectory]/env`.

![Screenshot describing this](documentation/setup_pycharm_correct_python.png)

#### Configuring targets

For this, we're going to reference the target configuration menu on pycharm (this is the top-right triangle-esque menu)

![See here](documentation/setup_pycharm_target_configuration.png)

##### Running Tests

Make sure to change the python tests configuration for nosetests and unittests (target selection menu (top-right) -> Edit Configurations -> Python Tests -> Nosetests) so that the working directory is the project directory (not the test directory!).

![See this screenshot for unittests, nosestest looks similar](documentation/setup_pycharm_unittests_configuration.png)

Additionally, you can set up an 'All Tests' target, just click the + button, name the target something like 'All Tests', and tell it to run all in folder, and point it at the test subdirectory of the project. Your configuration should look something like the attached screenshot

![Configuring an All Tests target](documentation/setup_pycharm_alltests_target.png)

##### Running local server in PyCharm

Again, we need to edit the target configurations. This time add (or change, if it already exists) a python target named `helix`. It should be configured to run the `helix/main.py` script (on pivotal's machine, that field is filled with `/Users/pivotal/workspace/helix/helix/main.py`), with a working directory of the project directory (on pivotal's machine, that would be `/Users/pivotal/workspace/helix`). Additionally, the `FLASK_DEBUG` environment variable should be set to `1`. ![See this screenshot](documentation/setup_pycharm_server_target.png)

#### PhantomJS w/ PyCharm

You may need to ensure that phantomjs is in your PyCharm PATH.  One way to do this is to symlink it into the project's env/bin folder
(for example `ln -s /usr/local/bin/phantomjs env/bin/`).

### Using Git

See [Github's documentation](https://help.github.com/). We prefer to commit directly to master, with small-ish commits that always have passing tests.

### Environment set-up using Docker
In order to set up complete environment using `docker-compose` you need to execute following line from project root directory:

```
docker-compose up -d
```

On Windows remember that repository should be inside Users dir. Additionally you need to set envvar:
export COMPOSE_CONVERT_WINDOWS_PATHS=1

Dependencies (including `node.js` packages) are resolved when building an image.
Environment will be exposed via ssh and accessible via following command:

```
ssh -p 2222 root@localhost
```
ssh password is: `screencast`

It can be seamlessly used via PyCharm by setting proper python interpreter:
`File -> Settings... -> Project: Helix_Roof_Calculator -> Project Interpreter`

![See this screenshot](documentation/pycharm_docker.png)

It's necessary to set path mappings when configuring targets:

![See this screenshot](documentation/pycharm_docker_config.png)

To initialize database, use following command:

```
ssh -p 2222 root@localhost 'cd /code && invoke db_migrate'
```

or:

```
docker exec -it helixroofcalculator_helix_1 invoke db_migrate
```

Debugging is possible directly from PyCharm, all code changes are transparent between host and container.


## Database schema


![Database Schema](documentation/db_schema.png)