Programming Python (DONE)

Note: Based on Programming Python on Udacity

Table of Contents

Chapter 1 - Programming Python

Install Python on OSX

  • Download Python 2 of 3, which includes:
    • Python language interpreter
    • IDLE IDE

    • pip - Python package management system pip --help

    • Note: pip and easyinstall download from Python Package Index Server
  • Download latest version of Python (i.e. 3.6.0) from https://www.python.org/downloads/
  • Note that the version will be created in: /Library/Frameworks/Python.framework/Versions/3.6
  • Use virtualenv to create a virtual Python environment using this version v.mk --python=python3.6 python_test_env_3.6.0
$ which python # => /Library/Frameworks/Python.framework/Versions/2.7/bin/python
$ which pip # => /Library/Frameworks/Python.framework/Versions/2.7/bin/pip
$ which idle # => /Library/Frameworks/Python.framework/Versions/2.7/bin/idle
  • Download Tcl/Tk dependency (ActiveTcl) for tkinter on MacOSX for IDLE IDE via this reference
  • Show where installed and version
  • Run Tk Widget Demo based on Tcl/Tk code from Wish GUI
$ which tclsh # => /usr/bin/tclsh
$ which wish # => /usr/bin/wish
$ tclsh
% info patchlevel # => 8.5.9
$ wish
% info patchlevel # => 8.5.9

Note: IGNORE THIS Alternatively install via Homebrew brew install python (BUT pyenv appears to be better than brew https://gist.github.com/Bouke/11261620)

  • Setup development environment
    • Create GitHub repo (i.e. PythonTest)
git clone https://github.com/ltfschoen/PythonTest
cd PythonTest

Packages Installed

  • Output packages installed with pip freeze, i.e.
pip freeze

dotenv==0.0.5
httplib2==0.9.2
py==1.4.31
pydash==3.4.5
pytest==3.0.3
pytz==2016.7
six==1.10.0
twilio==5.6.0
vboxapi==1.0

Product

  • Brainstorm to create time tracking app
    • Create program tracks time
    • Loops
      • Wait x hours
      • Then open browser
  • Share code with friend to truly understand

Using IDLE IDE

  • Run IDLE IDE
    • CMD+SPACE (Spotlight)
    • Type “IDLE”
    • New - Go to Menu: File > New File
    • Save - Go to Menu: File > Save
    • Run - Go to Menu: Run > Run Module
    • Stop - CTRL+C in Console Window >>>
  • Autocompletion
    • Start typing
    • Press CTRL+SPACE
    • Press up/down to desired method
    • Press SPACE to execute

Debugging IDLE

  • Debugging
    • CTRL+C in IDLE terminal to terminate any running app
    • Click in the Python 2.7.12 Shell window (not a code file)
    • Debug - Go to Menu: Debug > Debugger
    • Run - Go to Menu: Run (or FN + F5)
    • Go To Line Number: CMD+J

IntelliJ with Python Plugin and source level debugging setup

  • After fresh installation of IntelliJ IDEA 2016.2.5 where Python 2.7.12 already installed

  • Check Python Plugin installed (i.e. Python Plugin Version: 2016.2.162.43)
    • IntelliJ IDEA > Preferences > Plugins
  • Add Python as recognized file type
    • Preferences > Editor > File Types (shortcut CMD+,)
    • Click “+” to add a new “Recognized File Type”
    • Add name value of: Python
    • Add line comment value of: #
    • Add block comment start: “””
    • Add block comment end: “””
    • Select all checkboxes (i.e. Support paired …, Support string escapes)
    • Click “1” under “Keywords” and click “+” to add for syntax highlight colour 1: %, from, import
    • Click “2” under “Keywords” and click “+” to add for syntax highlight colour 2: def
    • Click “3” under “Keywords” and click “+” to add for syntax highlight colour 3: for, if, while, yield, next
    • Click “4” under “Keywords” and click “+” to add for syntax highlight colour 4: print
    • Click “OK” to save
  • Add Python file types
    • Preferences > Editor > File Types (shortcut CMD+,)
    • Click “+” under “Registered Patterns” and enter: *.py
    • Click “OK”

  • Check no Python files that you need to edit are listed under “Ignore files and folders”

  • Restart IntelliJ IDEA

  • Go back to check recognized file types where suddely “Python” (with *.pjw) and “Python Stub” (with *.pyi) are shown (were previous efforts were in vain???)
    • Preferences > Editor > File Types (shortcut CMD+,)
    • Click “OK”
  • Add Python Interpreter http://stackoverflow.com/questions/24769117/how-do-i-configure-a-python-interpreter-in-intellij-idea-with-the-pycharm-plugin
    • File > Project Structure > Platform Settings > SDKs
    • Click “+” and select Python SDK > Local > enter directory where Python installed (check in terminal with which python). Note that this added “Python 2.7.12” for me
    • File > Project Structure > Project Settings > Project > Project SDK > Select “Python 2.7.12”
    • Click “OK”
  • Configure Python
    • Preferences > Languages & Frameworks > Python Template Languages (i.e. Django)
    • Preferences > Languages & Frameworks > BDD (i.e. Behave)
    • Preferences > Tools > Python External Documentation
    • Preferences > Tools > Python Integrated Tools > Default Test Runner (i.e. unittest, py.test)
  • Configure Python debug configuration for source level debugging
    • Preferences > Build, Execution, Deployment > Debugger > Python Debugger

    • Preferences > Build, Execution, Deployment > Console > Python Console > Click “Always show debug console”

    • Run > Edit Configuration > Default > Python Tests > py.test

      Add “Target” of project directory Check “Options” and add: -v –color=yes –exitfirst –showlocals –durations=5

    • Run > Edit Configuration > Click “+”

      select “Python Tests > py.test” > Rename to “Python tests custom” > Change “Target” to project directory

    • Note: “Python Interpreter” should now be “Use specified interpreter” with value “Python 2.7.12 …”

    • Click “OK”

    • Run “Python tests custom”

  • Problem: No breakpoints appear when I click in the sidebar in Python app!!! (however using a Ruby app breakpoints work!!)

    • Attempt to fix #1: File > Invalidate Caches and Restart - DID NOT WORK
    • Attempt to fix #2: Uninstalled the Python Plugin Version: 2016.2.162.43 and reinstalled same version - DID NOT WORK
    • Solution Attempt to fix #2: Tried adding wildcard “.py” as a Registered Pattern but a popup appears “This wildcard is already registered by ‘Python’ filetype” with options Cancel or Reassign Wildcard (even though the only pattern listed is “.pyw” and none of the other registered filetypes use that wildcard). Clicking “Reassign Wildcard” allowed me to add “*.py” to the list. WORKED !!!!!

Python Configuration Information

  • System Path
import sys
print sys.path
import time
time.ctime() # => 'Sun Oct 30 09:21:06 2016'

Change Python Version (i.e. from 2.7 to 3.5) - Use virtualenv successfully instead

Package Versioning

  • http://stackoverflow.com/questions/458550/standard-way-to-embed-version-into-python-package

Python Dependency Management

  • Reference with excellent links
  • Note: Python comes with primitive package management tool easy_install (i.e. easy_install pip)
  • Python library dependencies are recommended to be installed using pip when a virtualenv is created
  • Pip downloads and installs app dependencies from central PyPi repo (similar to RubyGems or NPM)
  • Virtualenv instance stores a copy of Python interpreter and dependencies in the global site-packages directory (i.e. /Library/Python/2.7/site-packages/) to isolate an apps dependencies from system-level packages
  • Virtualenvwrapper Stores all Virtualenv’s in a single directory on the file system
  • requirements.txt specifies an apps pegged (precise versions or Git tags) dependencies
  • setup.py used to specify dependencies for Reusable Components (that will be imported as dependencies by other projects)
  • Anaconda http://stackoverflow.com/questions/38217545/the-different-between-pyenv-virtualenv-anaconda-in-python

  • Python Docker Images (to switch between Python environments)

  • List currently installed dependencies with pip freeze

  • Create a requirements.txt file in the root directory of the app and past the output of pip freeze

  • Reference Links
    • https://www.fullstackpython.com/application-dependencies.html
    • http://jonathanchu.is/posts/virtualenv-and-pip-basics/
  • Example Virtualenv and Virtualenvwrapper using Yolk for isolated environments using various Python Interpreters (i.e. v2.7 or v3.5)
$pip install virtualenv
$ which virtualenv
/Library/Frameworks/Python.framework/Versions/2.7/bin/virtualenv
  • Cont’d…
    • Install latest Virtualenvwrapper https://virtualenvwrapper.readthedocs.io/en/latest/
pip install virtualenvwrapper
  • Cont’d…
    • Update ~/.bash_profile with directory (i.e. ~/.virtualenvs) where we will store all virtualenv’s
$ echo $HOME
/Users/x
$ export WORKON_HOME=$HOME/.virtualenvs
$ source /Library/Frameworks/Python.framework/Versions/2.7/bin/virtualenvwrapper.sh
$ source ~/.bash_profile
$ echo $WORKON_HOME
/Users/x/.virtualenvs
  • Cont’d…
    • Add the following to Base Profile: Create aliases to save keystrokes http://blog.doughellmann.com/2010/01/virtualenvwrapper-tips-and-tricks.html
export WORKON_HOME=$HOME/.virtualenvs

alias v='workon'
alias v.deactivate='deactivate'
alias v.mk='mkvirtualenv --no-site-packages -v'
alias v.mk_withsitepackages='mkvirtualenv'
alias v.rm='rmvirtualenv'
alias v.switch='workon'
alias v.add2virtualenv='add2virtualenv'
alias v.cdsitepackages='cdsitepackages'
alias v.cd='cdvirtualenv'
alias v.lssitepackages='lssitepackages'

source /Library/Frameworks/Python.framework/Versions/2.7/bin/virtualenvwrapper.sh
  • Cont’d…
    • Show current location of Python Interpreter before creating a virtual environment
$ which python
/Library/Frameworks/Python.framework/Versions/2.7/bin/python
  • Cont’d…
    • Create new completely clean virtual environment (notice how command prompt changes at the end) with a specific Python version
$ v.mk -python=python2.7 python_test_env
New python executable in /Users/x/.virtualenvs/python_test_env/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /Users/x/.virtualenvs/python_test_env/bin/predeactivate
virtualenvwrapper.user_scripts creating /Users/x/.virtualenvs/python_test_env/bin/postdeactivate
virtualenvwrapper.user_scripts creating /Users/x/.virtualenvs/python_test_env/bin/preactivate
virtualenvwrapper.user_scripts creating /Users/x/.virtualenvs/python_test_env/bin/postactivate
virtualenvwrapper.user_scripts creating /Users/x/.virtualenvs/python_test_env/bin/get_env_details
(python_test_env) $
  • Cont’d…
    • Show that the new virtual env contains its own copy of a specific Python Interpreter:
(python_test_env_2.7) $ which python
/Users/Ls/.virtualenvs/python_test_env_2.7/bin/python
  • Cont’d…
    • Show where the new python_test_env has been created:
(python_test_env) $ ls $WORKON_HOME
...		python_test_env
  • Cont’d…
    • Deactivate using python_test_env and then show globally installed packages
(python_test_env) $ deactivate
$
$ pip freeze
click==6.6
coverage==4.2
httplib2==0.9.2
pbr==1.10.0
py==1.4.31
pydash==3.4.5
pytest==3.0.3
python-dotenv==0.6.0
pytz==2016.7
six==1.10.0
stevedore==1.18.0
twilio==5.6.0
vboxapi==1.0
virtualenv==15.0.3
virtualenv-clone==0.2.6
virtualenvwrapper==4.7.2
  • Cont’d…
    • Restart python_test_env again
v python_test_env
  • Cont’d…
    • Install Python page Yolk, a utility to list packages installed for environment, and then list them, and show locally installed packages
$ pip install yolk
$ yolk -l
Python          - 2.7.12       - active development (/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload)
pip 9.0.0 has no metadata
setuptools 28.7.1 has no metadata
wheel 0.30.0a0 has no metadata
wsgiref         - 0.1.2        - active development (/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7)
yolk 0.4.3 has no metadata

$ pip freeze
yolk==0.4.3
  • Cont’d…
    • Install requirements i.e. pip install x y z
  • Cont’d…
    • Send a snapshot of all the requirements and current versions into the requirements.txt file
(python_test_env) $ pip freeze > requirements.txt
(python_test_env) $ cat requirements.txt 
yolk==0.4.3
  • Cont’d…
    • Install the dependencies when cloned in another environment

$ pip install -r requirements.txt

  • Cont’d…
    • Remove virtual env
deactivate
rmvirtualenv python_test_env
  • Cont’d…
    • Change to a different virtual env v.switch python_test_env3.5.2
    • Show current Python version
python -V
  • Cont’d…
    • Create a virtual environement with a specific Python Interpreter (i.e. 3.5) version
v.mk --python=python3 python_test_env_3.5
  • Cont’d…
    • Install dependencies from requirements.txt
(python_test_env_3.5) $ pip install -r requirements.txt
  • Cont’d…

    • Try running a program to see if it breaks using the newer version of Python

    • Switch to a different virtual env

(python_test_env_3.5) $ v.switch python_test_env
(python_test_env) $

Pipenv

  • https://github.com/kennethreitz/pipenv

  • Use suite such as:
    • pylint & PEP 8 - Style Guide
    • mypy & PEP 484 - Type Hints
    • pipenv & PEP 508 - Dependency spec
    • pytest - Tests

  • Install pipenv, initialise in python3 mode, start pipenv shell, install project dependencies, run tests

      pip install pipenv
  pipenv --three
  pipenv shell -c
  pipenv install
  pytest

Mypy

  • Create stubs

    • https://github.com/python/mypy/wiki/Creating-Stubs-For-Python-Modules

    • i.e. mkdir stubs, then run export MYPYPATH=/Users/<your_username>/code/aind-projects/clones/nd889/2_isolation/stubs then Copy and paste the timeit.pyi stub posted on the typeshed GitHub repo here into the stubs directory to remove Mypy error isolation/isolation.py:11: error: No library stub file for standard library module 'timeit'

Convert Python 2 to Python 3

  • Use 2to3 utility on all files http://stackoverflow.com/questions/37891188/convert-python-2-code-to-3-in-pycharm 2to3 myfile.py -w
  • Correct fixes for exceptions manually

  • rename pip to pip2 using brace expansion cd /usr/local/bin && mv /usr/local/bin/{pip,pip2}

  • created symlink for pip command to points at pip3 so ln -s /usr/local/bin/pip3 /usr/local/bin/pip

Jupyter Notebook

If get errors in Jupyter Notebook when running code snippets like: ModuleNotFoundError: No module named 'numpy'

Then note that Jupyter Notebook uses Python 3 by default, and the issue can be resolved by installing the packages using pip3 install numpy matplotlib (pointing to Python 3.6) instead of just pip ... (which pointed to Python 2.7) and then opening Jupiter Notebook and performing Run All. pip3 list shows which packages are installed. Confirm which installation directory of Python 3.6 that Jupyter Notebook was using by running the following commands within the code snippets in Jupyter Notebook:

print("{}".format(sys.version))
# show Python 2.7 executable directory
!which pip
# show Python 3.6 executable directory
!which pip3
# show jupyter config paths http://jupyter.readthedocs.io/en/latest/projects/jupyter-directories.html
!jupyter --paths

Which returned the following output:

3.6.0 (v3.6.0:41df79263a11, Dec 22 2016, 17:23:13)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
/Users/<my_username>/miniconda3/envs/aind/bin/pip
/Library/Frameworks/Python.framework/Versions/3.6/bin/pip3
config:
    /Users/<my_username>/.jupyter
    /Library/Frameworks/Python.framework/Versions/3.6/etc/jupyter
    /usr/local/etc/jupyter
    /etc/jupyter
data:
    /Users/<my_username>/Library/Jupyter
    /Library/Frameworks/Python.framework/Versions/3.6/share/jupyter
    /usr/local/share/jupyter
    /usr/share/jupyter
runtime:
    /Users/<my_username>/Library/Jupyter/runtime

Notice how pip3 is in a subdirectory of /Library/Frameworks/Python.framework/Versions/3.6/, whereas pip was not

Pip Autcompletion

  • Run the following to setup and enable, then use with pip i[PRESS_TAB]
pip completion --bash >> ~/.bashrc
source ~/.bashrc

Python Standard Library

Python 3rd Party Library

pip install twilio

Google Style Guide for Python

Environment Variables

  • Python Dotenv

  • Alternatively add your credentials to your shell environment

echo "export TWILIO_ACCOUNT_SID='xyz'" >> ~/.bashrc
echo "export TWILIO_AUTH_TOKEN='xyz'" >> ~/.bashrc
source ~/.bashrc

Python Packages by Ranking

Python Standard Library Built-In Functions

Iterable Data Sets

  • itertools https://docs.python.org/3/library/itertools.html

Importing Files in Separate Directories

  • Modules and Packages
  • Note from x import y means it binds y pointing to module attribute x.modules[‘x’].y) but the rest of module x is still imported. there is not performance difference to just using import, it’s just a more granular code style making it cleared what is specifically being imported
    • i.e. from twilio.rest import TwilioRestClient means in twilio there is a folder called rest, and inside that folder is a class called TwilioRestClient (defined in init.py). Use the class with TwilioRestClient(...).
    • Alternatively use from twilio import rest and use the class with rest.TwilioRestClient(...) to call the classes init() function and create an instance
import site
print ("Importing subfolder: %s") % (sys.path[0]+'/helpers')
site.addsitedir(sys.path[0]+'/helpers')  
import reusable

Python Path

  • Add directory to $PYTHONPATH
export PYTHONPATH=$PYTHONPATH:/home/path1:/home/path2
echo $PYTHONPATH

Python Functional Programming (similar to Lodash)

pip install pydash
  • Example:
from pydash import py_
random_filename_suffix = str(randint(0,99))
filename = "sample42"
randfile_ints_found = re.findall('\d+', filename) # array of numbers found in filename
suffix_matches_found = py_(randfile_ints_found).map(lambda x: x == random_filename_suffix).each(print).value()
return True if any(element for element in suffix_matches_found) else False

Python Frameworks

  • Pyramid
  • Flask
  • Django
  • Web2Py

Python Production

  • AppHosted
  • djangozoom
  • Djangy
  • ep.io
  • Gondor.io
  • Google App Engine
  • Heroku
  • Joyent
  • Pydra
  • stable.io

Python Package System (similar to RVM/RBEnv)

Python Versioning

  • Show current version being used python -V
  • Indicate that a file only supports certain Python versions 
    import sys
assert sys.version_info >= (3,5,2)

Python Unit Testing

  • UnitTest
    • Standard module testing framework (aka pyUnit similar to xnit)
  • PyTest
    • py.test lightweight unit test framework without boilerplate of unittest
    • Test discovery requires all test filenames prefixed test_
    • PyTest Fixtures i.e. before_each, load sample data, etc
pip install -U pytest
pytest --version
  • Cont’d
  • Nose
    • Task runner that runs functions in modules whose name starts with ‘test’
    • Test generators implement data-driven tests
    • Runs unittest tests
    • Coverage plugin available
    • Google App Engine plugin
  • Nose2

  • DocTest

  • Other

Continuous Integration with Travis CI

  • Enable Travis CI to use GitHub app - https://travis-ci.org/auth
  • Check build status of GitHub app - https://travis-ci.org/ltfschoen/PythonTest
  • Create .travis.yml file - https://docs.travis-ci.com/user/getting-started/
  • Customise .travis.yml file - https://docs.travis-ci.com/user/customizing-the-build/
  • View other example .travis.yml files - https://docs.travis-ci.com/user/languages/python/#Examples

PyPy

  • Alternative to Python

Python Syntax

  • Inputs
    • input evaluates expression upon entry
    • raw_input does not evaluate immediately
  • Strings
    • Join ''.join(['/', samples_folder_name, '/'])
  • Booleans (first letter capitalised)
    • True, False
  • Print Output
    • print("Before - Index: %s, Filename: %s") % (index, filename)
  • Control Flow
  • Array
    • Iterate array by index from specific starting index
    • Examples http://stackoverflow.com/questions/1514553/how-to-declare-an-array-in-python
for i, val in enumerate(data, 1):
    print i, val
  • Class
    • Define template/blueprint (class) from which copies (instances/objects) may be made as types of it
    • Calling the init() function to create space in memory for new instance of the class that can access all class methods
    • Class variables are those with values common to all instances
    • Pre-existing class variables include doc so you can create documentation for a class within """blah""" https://discussions.udacity.com/t/exploring-built-in-functions-f/16116/2715
class A(object):     # deriving from 'object' declares A as a 'new-style-class'
    def foo(self):
        print "foo"
    @classmethod
    def bar(self):
        print("bar")

A().foo() # => foo
A().bar() # => AttributeError: 'A' object has no attribute 'bar'
A.foo()   # => TypeError: unbound method foo() must be called with A instance as first argument (got nothing instead)
A.bar()   # => AttributeError: 'A' object has no attribute 'bar'

class B(A):
    def foo(self):
        super(B, self).foo()   # calls 'A.foo()'
    @classmethod
    def bar(self):
        super(B, self).bar()   # calls 'A.bar()'

B().foo() # => foo
B().bar() # => AttributeError: 'B' object has no attribute 'bar'
B.foo()   # => TypeError: unbound method foo() must be called with B instance as first argument (got nothing instead)

B.bar()   # => AttributeError: type object 'B' has no attribute 'bar'
b = B()
b.foo()   # => foo
b.bar()   # => AttributeError: 'B' object has no attribute 'bar'
  • Hash
    • Dictionary using hash tables
D = {}
for k,v in D.items():
  print k,':',v
  • Random Numbers
    • str(randint(0,99)) random integer between 0 and 99 inclusive
  • Random Strings
import random
import string
''.join(random.choice(string.ascii_letters) for char in range(char_qty))
  • Modules
    • File with Python definitions and syntax
    • Filename is .py
    • __name__ is available in Module as module name
  • File Handling

  • Regular Expressions

    • Replace after last instance of character being queried in string
test_path = "/Users/x/code/apps/PythonTest" 
split_on_char = "/"
split_on_char.join(test_path.split(split_on_char)[:-1]) # => /Users/x/code/apps

Python Test Coverage

Python Developers Slack Forum

Udacity Forum

TODO

  • Behave - https://pypi.python.org/pypi/behave
  • Tox - https://tox.readthedocs.io/en/latest/
  • Jenkins and Python - https://wiki.jenkins-ci.org/display/JENKINS/Python+Projects
  • PyAtom - https://github.com/pyatom/pyatom
  • Random - http://mathieu.agopian.info/presentations/2015_06_djangocon_europe/?full#test-files
Written on October 30, 2016