Helix Calculator
Local Development and Testing with Docker
Dependencies
-
CircleCi CLI: Installing and using CircleCi locally
-
Current CircleCi and development image: ivannnn/heroku-cedar14-extras:2.0
| ivannnn/heroku-cedar14-extras:2.0 |
|---|
| ruby 2.3.5p376 |
| Python 3.5.4 |
| postgresql-9.5 |
| postgresql-client-9.5 |
| postgresql-client-9.6 |
| postgresql-client-common |
| postgresql-common |
| postgresql-contrib-9.5 |
| postgresql-server-dev-9.6 |
| redis-server/trusty,now 2:2.8.4-2 amd64 |
| redis-tools/trusty,now 2:2.8.4-2 amd64 |
| google-chrome-stable/stable,now 62.0.3202.94-1 amd64 |
| nodejs/unknown,now 6.12.0-1nodesource1 amd64 |
Environment Variables
Set the environment variables for your specific environment. Use the .env file below as reference:
AWS_S3_BUCKET="..."
AWS_ACCESS_KEY_ID="..."
AWS_SECRET_ACCESS_KEY="..."
SF_BASE_URL="https://test.salesforce.com"
SFDC_ACCESS_KEY_ID="..."
SFDC_SECRET_ACCESS_KEY="..."
Set up for Develop and Testing
- Run the docker container for the 1st time
docker run -d --name helix -p 5000:5000 --expose 5000 -t ivannnn/heroku-cedar14-extras:2.0
- Or re start the docker conatiner
docker container start helix
- Copy the code to the container
docker cp . helix:project
- Log in the docker container
docker exec -ti helix bash
- Run this commands inside the docker conatiner
cd project && \
rm -rf env && \
gem install bundler && \
service postgresql start &&\
service redis-server start &&\
export PGPASSWORD=password &&\
export CIRCLE_TEST_REPORTS=$(pwd) &&\
python3.5 -m venv env &&\
source env/bin/activate &&\
pip install invoke &&\
invoke install &&\
npm install &&\
invoke db_migrate
Important
- Open another terminal tab, this script watches for changes and pass them
to the docker conatiner, which will restart the web server
./watch.sh
Run the server
invoke serve_debug
Test locally without CircleCi
invoke test_ci
Test locally with CircleCi
- Go to the project dir
circleci build
OS steps wihout Docker
macOS systems
This assumes you already have homebrew installed as well as an ssh key added to your github account. This also assumes your username is pivotal, see the circleci configuration for examples of how to add a pivotal user to postgres.
xcode-select --install
brew install python3 node nodenv redis phantomjs imagemagick@6 postgres brew-services
brew services start postgres
createdb test
createdb pivotal
git clone git@github.com:SunPower/Helix_Roof_Calculator.git
cd Helix_Roof_Calculator
pyvenv env
source env/bin/activate
pip install invoke
invoke install
npm install
invoke db_migrate
This installs python3, the invoke utility, xcode, node, imagemagick, redis, and postgres. It also sets up a virtual environment for the calculator, installs all the python dependencies into the virtual environment, and installs all the node dependencies. Note that invoke should be installed after the virtual env is installed to ensure that the python 3.x version of invoke is used (instead of the system version, which is typically 2.7.x).
Linux/Debian-based systems
While most of this README assumes you're running macOS, for linux/debian based systems (e.g. ubuntu), follow these instructions. See the mac section for an explanation.
PostgreSQL
Postgres needs to be installed and configured before we do anything else.
sudo apt-get install postgresql postgresql-client postgresql-client-common postgresql-contrib libpq-dev
# Modify /etc/postgresql/common/*/pg_hba.conf and set local connections to trust
sudo -u postgres createdb test
sudo -u postgres createdb pivotal
# create pivotal role in postgres, grant it access to the test/pivotal databases.
echo "CREATE ROLE pivotal WITH UNENCRYPTED PASSWORD 'password';" | psql -U postgres
echo "ALTER ROLE pivotal WITH LOGIN;" | psql -U postgres
echo "GRANT SELECT, UPDATE, INSERT ON ALL TABLES IN SCHEMA test.public TO pivotal;" | psql -U postgres
echo "GRANT CREATE, CONNECT ON DATABASE test TO pivotal;" | psql -U postgres
echo "GRANT SELECT, UPDATE, INSERT ON ALL TABLES IN SCHEMA pivotal.public TO pivotal;" | psql -U postgres
echo "GRANT CREATE, CONNECT ON DATABASE pivotal TO pivotal;" | psql -U postgres
This installs postgres and creates the database and roles that helix looks for in test/local usage. Because ubuntu is systemd, you can start/stop/restart the server by running something along the lines of sudo systemctl [start|stop|restart] postgresql.service. Please see this ubuntu postgres documentation page for more information.
The comment for the second line tells you how to enable connections to postgres without requiring a password. See http://dba.stackexchange.com/questions/83164/remove-password-requirement-for-user-postgres for more information
Other dependencies/the project
sudo apt-get install git python3 python3-dev python3-venv nodejs npm redis-server imagemagick libmagickcore-dev libxml2-dev libxslt1-dev ruby ruby-dev
sudo gem install bundler
python3 --version # make sure you have >3.5 installed!
git clone git@github.com:SunPower/Helix_Roof_Calculator.git
cd Helix_Roof_Calculator
pyvenv env
source env/bin/activate
pip install invoke
invoke install
npm install
invoke db_migrate
PhantomJS
sudo apt-get install build-essential chrpath libssl-dev libxft-dev libfreetype6 libfreetype6-dev libfontconfig1 libfontconfig1-dev
export PHANTOM_JS=phantomjs-2.1.1-linux-x86_64
wget https://bitbucket.org/ariya/phantomjs/downloads/$PHANTOM_JS.tar.bz2
sudo targ xvjf $PHANTOM_JS.tar.bz2
sudo mv $PHANTOM_JS /usr/local/share
sudo ln -sf /usr/local/share/$PHANTOM_JS/bin/phantomjs /usr/local/bin
Note that it doesn't work to use the nodejs-provided phantom, nor does apt-get install phantomjs work. Yes, it's annoying.
Adding Dependencies
Because of how we're separating production dependencies and test dependencies, it's recommended that each new dependency be manually added to either the requirements.txt or the requirements.test.txt, depending on whether it's a production or test dependency, respectively.
Adding database migrations
To manage the database, we use SQLAlchemy.
To manage database migrations, we use Alembic.
First, make your changes to the sql models (in the models/sql) package to reflect your new vision of the sql schema. Next, we need to take those changes and generate an alembic migration from them. Run PYTHONPATH=. alembic revision -m "(migration description)" to generate that migration. Now, add the changes (and revert instructions!) to the generated migration file. Finally, run invoke db_migrate to run that migration.
Downgrading a migration
There currently is not an invoke task to revert migrations. Instead, just directly use the alembic cli to do this, something along the lines of PYTHONPATH=.:$PYTHONPATH alembic downgrade -1. Setting PYTHONPATH to include the current directory will ensure that the correct sql models get picked up when alembic does the migration. To redo a migration using alembic, run PYTHONPATH=.:$PYTHONPATH alembic upgrade +1.
Please refer to the Alembic Documentation for more information on using Alembic.
Running Tests
First, install PhantomJS (brew install phantomjs) (this should already be done as part of the setup).
Run invoke test from the project directory will run every test.
Note that we use Python's built-in unittest library for our python tests. Additionally, there are a few other helper libraries we use in tests. See the requirements.test.txt file for those.
Running Javascript Tests
invoke test will, by default, run both the python tests and the javascript tests. Because the python tests can take a while, it's advantageous to be able to run the javascript tests (used for the array visualization) separately, especially when working entirely on the visualization frontend.
To do this, run invoke test_js from your command. This spins up an instance of karma attached to (by default) Chrome which watches the spec and the helix/javascript directories for changes, and (usually) re-runs the javascript tests if anything changes.
Note that we use Jasmine for our javascript specs.
Building JS
Part of the application (as of now only files in helix/javascript/array_summary) are built using Webpack and leveraging ES6.
To run the build use this command from the root of the repository
./node_modules/.bin/webpack
Running Locally
First, install redis (brew install redis), and then (in a separate terminal) run redis-server /usr/local/etc/redis.conf to run redis locally.
Next, install postgres (brew install postgres), and then start it (brew services start postgres), follow the instructions at the top for additional configuration information.
Then, tell alembic to migrate your postgres db to the latest (invoke db_migrate).
Run invoke serve from the project directory to run locally.
This will run the project on port 5000.
Running in debug mode
Run invoke serve_debug to start the project in debug mode.
The image tests
Note that you may see image comparison tests failing for erroneous reasons. These are flaky and machine-dependent. If you've done anything to the image generation code, please verify visually that you're getting the output you want.
Circle CI
CI (Continuous Integration) is a service we use that automatically runs all the tests whenever a new commit is pushed. This is important because it runs the code in a context-free (the container the code runs in is set up for each run, then destroyed afterwards) environment. We're using Circle CI here because Rachel recommended it. In addition to automatically running tests, circle also automatically deploys successful builds to cloud foundry for us.
Deploying to Heroku (locally)
Install the Heroku CLI by running brew install heroku from your terminal.
Add heroku git endpoints for the different environments you want to push to. Typically, you'll just want to push to staging (especially seeing that preprod/production are automatically pushed to heroku for you). The heroku git endpoints are:
- staging:
https://git.heroku.com/sp-helix-staging.git - preprod:
https://git.heroku.com/sp-helix-preprod.git - production:
https://git.heroku.com/sp-helix-production.git
login to heroku with heroku login
Finally, push to heroku using git! (Something like git push heroku master)
Note that circle ci does all that for us!
Deploying to preprod
To deploy to preprod, rebase master (up to the commit you want) onto preprod, then push preprod to github. CI will automatically deploy to preprod assuming that tests pass.
To push the current state of master to preprod:
git pull -r # Pull from origin (assuming origin is github)
git checkout preprod # switch to preprod branch
git rebase master # rebase head of master onto preprod
git push origin preprod # push preprod to github
To push a past commit to preprod:
git pull -r # Pull from origin (assuming origin is github)
git log
# [grab the sha of the commit you want]
git checkout preprod # switch to preprod branch
git rebase f00d # replace f00d with the sha of the commit you want
git push origin preprod # push preprod to github
Deploying to production
To deploy to production, tag a commit as release-<number> (for example, release-2) and push the commit (git push origin release-2). CI will automatically deploy to production assuming that tests pass.
Please increment the release count each time.
Setting up PyCharm
PyCharm is an IDE from Jetbrains for python. It can be installed by running brew cask install pycharm.
Open pycharm, input a license, and select to install the command line tool. Then, open the project in pycharm by running charm . from the project directory.
Right now, you can start to use pycharm as an editor, but it's not very useful as an IDE, there's a bit more configuration stuff before it's useful as an IDE.
Use the correct Python
By default, PyCharm assumes you're using Python 2.7.x, we need to tell it to not only use 3.5.x, but use the 3.5.x for our local virtual environment.
To do this, open the preferences window, and select Project: Helix_Roof_Calculator (our project directory is named helix instead of Helix_Roof_Calculator, so the screenshots will be different than what you see), then select Project Intepreter and change that to 3.5.2 virtualenv at [projectdirectory]/env.
Configuring targets
For this, we're going to reference the target configuration menu on pycharm (this is the top-right triangle-esque menu)
Running Tests
Make sure to change the python tests configuration for nosetests and unittests (target selection menu (top-right) -> Edit Configurations -> Python Tests -> Nosetests) so that the working directory is the project directory (not the test directory!).
Additionally, you can set up an 'All Tests' target, just click the + button, name the target something like 'All Tests', and tell it to run all in folder, and point it at the test subdirectory of the project. Your configuration should look something like the attached screenshot
Running local server in PyCharm
Again, we need to edit the target configurations. This time add (or change, if it already exists) a python target named helix. It should be configured to run the helix/main.py script (on pivotal's machine, that field is filled with /Users/pivotal/workspace/helix/helix/main.py), with a working directory of the project directory (on pivotal's machine, that would be /Users/pivotal/workspace/helix). Additionally, the FLASK_DEBUG environment variable should be set to 1. 
PhantomJS w/ PyCharm
You may need to ensure that phantomjs is in your PyCharm PATH. One way to do this is to symlink it into the project's env/bin folder
(for example ln -s /usr/local/bin/phantomjs env/bin/).
Using Git
See Github's documentation. We prefer to commit directly to master, with small-ish commits that always have passing tests.
Environment set-up using Docker
In order to set up complete environment using docker-compose you need to execute following line from project root directory:
docker-compose up -d
On Windows remember that repository should be inside Users dir. Additionally you need to set envvar: export COMPOSE_CONVERT_WINDOWS_PATHS=1
Dependencies (including node.js packages) are resolved when building an image.
Environment will be exposed via ssh and accessible via following command:
ssh -p 2222 root@localhost
ssh password is: screencast
It can be seamlessly used via PyCharm by setting proper python interpreter:
File -> Settings... -> Project: Helix_Roof_Calculator -> Project Interpreter
It's necessary to set path mappings when configuring targets:
To initialize database, use following command:
ssh -p 2222 root@localhost 'cd /code && invoke db_migrate'
or:
docker exec -it helixroofcalculator_helix_1 invoke db_migrate
Debugging is possible directly from PyCharm, all code changes are transparent between host and container.
Database schema
