Hi! Hope you're enjoying this blog. I have a new home at www.goldsborough.me. Be sure to also check by there for new posts <3
Showing posts with label python. Show all posts
Showing posts with label python. Show all posts

Tuesday, August 11, 2015

Creating and Distributing a Python Project

I just spent the last week giving birth to ecstasy, my first full-fledged, fully-distributed, fully-fancied-up Python project. It was an amazing adventure and I would like to share all the insights and knowledge I acquired over the last few days. Moreover, I’d like to spare you all the trouble I’ve had and headaches I got and tears I cried.

Description


It’s not that I hadn’t already written tons of stuff in Python and hadn’t already open-sourced multiple projects on Github, but this time I wanted all the fancy bells and whistles. With that I mean, and this is what I’ll show you how to do here:


I will be writing this article as if you hadn’t written anything yet, but had only an idea for your next project, which I’ll refer to as your_app. Nevertheless, it’s quite likely that you’re here right in the heat, looking for a solution to a sub-problem of this whole process and are thus not even reading this sentence but are somewhere below. So, since no-one is reading this sentence, I’ll just take this opportunity to say hi to my mom. Hi.

But on a more serious note, this article can be followed in a linear way, but needn’t be. Also note that I will be sharing valuable resources regarding certain topics that I used to acquire the knowledge I’m sharing with you and that you may also want to laugh at. I mean read. Regarding the "level", I'll try to be more explicit and explain more rather than not. You can always skip what you already know.

We’ll start with a basic directory structure and then add more and more layers of cream and cherries and chocolate, spray some sugar and love on it, spread caramel and marshmallow all over and lastly have it blessed by the almighty Octocat after which we’ll unite in a wild frenzy of excitement and exhilaration and dance around a fire. Sounds good?

Getting Started


A basic directory structure can look like follows:

your_app/
├── docs
├── (examples)
├── tests
└── your_app

  • your_app/ is the top-level directory, containing your entire project.
  • your_app/your_app/ contains your application’s source code.
  • your_app/docs/ includes everything related to documentation.
  • your_app/examples/ could include small examples of how to use your application. I put this in parentheses because I haven’t seen that many projects on Github with such an examples directory. Personally I quite like the idea of it, but you could also argue that you’d rather put examples into the documentation. Your call.

We’ll want to do three things now:
  1. Make sure you have pip so we can get any and all packages we want.
  2. Initialize a local git repository.
  3. Setup a virtual environment (virtualenv).

The Cheese Shop


pip is a command-line utility, more precisely a “package-manager”, for installing Python packages from PyPI — the Python Package Index (pronounced pie-pee-ie) — also referred to as The Butchery … no wait, that doesn’t seem right. Also referred to as The Cheese Shop. Yes, better. But if you’ve ever used npm for Node.js, RubyGems for Ruby or Bower for Javascript and all things web, then you know what’s up. Either way, we will need it for many, many things so make sure you have pip holstered and ready to fire.

Installing it is easy if your Python distribution’s version ($ python -- version) is 2.7.9 or 3.4.0 (and higher): you don’t have to. For Python 2 < 2.7.9 and Python 3 < 3.4, you can use this command to get it off the command-line:

$ curl --silent --show-error https://bootstrap.pypa.io/get-pip.py | sudo python

Resources:

Git


At this point, it's a good idea to initialize a local git repository in your application's root directory (your_app/) to record whatever we are doing and save us in case we mess anything up. I won't tell you when to commit anything for the remainder of this article, that's up to you, but remember: commit early and often.

your_app$ git init

At the same time, grab a suitable .gitignore file for Python off Github:

your_app$ curl https://raw.githubusercontent.com/github/gitignore/master/Python.gitignore > .gitignore

Virtualenv


We'll want to run everything we do inside a virtual environment. What's that? A virtual environment is an isolated container or environment that we can create with the virtualenv command, that acts just like your normal, global system environment with one important difference: whatever we install with pip will only be installed for the virtual environment (which is really a directory) and not globally -- i.e. what happens in a virtualenv stays in a virtualenv. Thereby, we can have different requirements for different projects, not pollute our global environment and also keep track of exactly what we need to execute and test our application. This last bit is important, because whatever we need to install in our virtual environment, users and contributors will also have to install. Thus, in summary, a virtualenv, in combination with pip, will greatly ease tracking our project's requirements. 

To install virtualenv, use pip:

$ sudo pip install virtualenv

To then create a virtual environment called env for our project, use this command inside your_app's directory:

your_app$ virtualenv env

This creates a virtualenv directory env (already ignored by .gitignore). To activate the virtualenv, you would:

your_app$ source env/bin/activate

Once you're done, you can deactivate the virtual environment via: 
your_app$ deactivate

And at last, a trick, coz ain't nobody got time for typing source env/bin/activate, add this to your .bash_profile or .bash_rc: 

venv() { source ${1:-env}/bin/activate;  }

So that, now, you can just type venv to activate your virtualenv called "env", and else call venv <env_name> if you decide to give your virtual environment a different name.

Resources:

Tests


I'll be taking the well-established approach of test-driven-development (TDD) here and first setup tests. In this section, I'll show you how to setup the tests directory, write a simple test module and test it with unittest2 (I'll explain why not the built-in unittest). We'll also use coverage to see how much of our code our tests really cover (execute). Regarding the actual source code: you have yours of course, but for the purpose of this tutorial I'll be writing an application that squares numbers. Yes I'd like 1 Nobel prize please and some fries, thanks.

Unit Tests


First of all, let's turn the tests/ directory into a package by touching an empty __init__.py file (note that we're in the virtual environment now):

(env)your_app$ touch tests/__init__.py

Then, we'll have one test module test_code.py that will test your_app's code, which we'll later on write. For this, we need unittest2. Unittest2 is a testing framework built on Python's own unittest package, but that is backported to Python 2.6. We'll use it because why not have that extra bit of support with no semantic changes (Note: there are many other testing frameworks out there, such as py.test or doctest, which are equally capable but have quite different semantics and flavors, which you can also check out). Get unittest2 with pip:

(env)your_app$ pip install unittest2

Then you can write your unit tests, for example tests/test_code.py (testing methods must start with 'test'):

import os
import sys
import unittest2

sys.path.insert(0, os.path.abspath(".."))

import your_app.code

class TestCode(unittest2.TestCase):

 def test_squares_positive_numbers_correctly(self):

  self.assertEqual(your_app.code.square(2), 4)

 def test_squares_negative_numbers_correctly(self):

  self.assertEqual(your_app.code.square(-3), 9)

Note that we have to make the parent directory along with our source package your_app visible to our test modules' path, which you can see in line 5. Now, to run our tests, we can use the unit2 command that came with unittest2 (which performs test discovery and will find our tests from the root your_app directory). The -v switch causes more verbose output and shows which tests run successfully and which fail. Note that you can also call unittest2 as an executable Python module, like you would unittest (if you've used it before), i.e.:

(env)your_app$ python -m unittest2

In any case, here's what unit2 does:

(env)your_app$ unit2 -v
======================================================================
ERROR: tests.test_code (unittest2.loader._FailedTest)
----------------------------------------------------------------------
Traceback (most recent call last):
ImportError: Failed to import test module: tests.test_code
Traceback (most recent call last):
  File "/Users/petergoldsborough/Documents/Projects/your_app/env/lib/python2.7/site-packages/unittest2/loader.py", line 456, in _find_test_path
    module = self._get_module_from_name(name)
  File "/Users/petergoldsborough/Documents/Projects/your_app/env/lib/python2.7/site-packages/unittest2/loader.py", line 395, in _get_module_from_name
    __import__(name)
  File "/Users/petergoldsborough/Documents/Projects/your_app/tests/test_code.py", line 7, in &ltmodule>
    import your_app.code
ImportError: No module named code


----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)

If all goes well, everything should fail. Of course: we haven't written any code yet (there isn't any module named your_app.code yet). Let's do that. After also creating an __init__.py file in your_app/your_app, the package that will contain all of our application's source code, we can write our main module your_app/code.py:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

def square(x):
    return x * x

Now running the testing command from above should give us happy messages and make us smile:

(env)your_app$ unit2 -v
test_squares_negative_numbers_correctly (tests.test_code.TestCode) ... ok
test_squares_positive_numbers_correctly (tests.test_code.TestCode) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

Coverage


It's a good idea to know about our code's test coverage, i.e. how much of our project's code we're testing (and thereby avoiding bugs). For this, we use the coverage utility, which describes itself as "[...] a tool for measuring code coverage of Python programs. It monitors your program, noting which parts of the code have been executed, then analyzes the source to identify code that could have been executed but was not." Grab coverage with pip:

(env)your_app$ pip install coverage

You use coverage by simply running your Python modules like you would with the python command, but substituting python with coverage run. For our tests, that means changing

(env)your_app$ python -m unittest2

to 

(env)your_app$ coverage run -m unittest2

This will run the same code as the previous command, but now we can call coverage report (add the -m switch for uncovered line numbers). We'll want to specify which modules we want to have included in our report, because, at least for me, coverage also reports on coverage of modules in Python's site-packages directory. For this, we add the --include="./*" option. Now, this should happen:

(env)your_app$ coverage report -m --include="./*"
Name                Stmts   Miss  Cover   Missing
-------------------------------------------------
tests/__init__          0      0   100%   
tests/test_code        10      0   100%   
your_app/__init__       5      0   100%   
your_app/code           3      0   100%   
-------------------------------------------------
TOTAL                  18      0   100%

Looks like we did a good job writing tests! What an achievement for such a large codebase!

Resources:

That's it for this section, here's what your current directory should look like if you're following this tutorial in a linear way:

├── docs
├── examples
├── temp
│   └── code.py
├── tests
│   ├── __init__.py
│   └── test_code.py
└── your_app
    ├── __init__.py
    └── code.py


Documentation


Now that we have finished writing our enchanting code and also have our tests running perfectly, we can work on documentation. This section will cover setting up the docs/ directory, writing our code's docstrings ready to be parsed by the Sphinx documentation generation tool, using a special super-legible syntax enabled by the Napoleon extension to Sphinx, and lastly writing additional documentation in reStructuredText that will later go on ReadTheDocs. You can find the documentation I generated for my project here, to have a look what you'll get: http://ecstasy.readthedocs.org/

Sphinx


What is Sphinx? According to sphinx-doc.org: "Sphinx is a tool that makes it easy to create intelligent and beautiful documentation". Basically, Sphinx is the go-to solution for Python documentation. Anything to be parsed by Sphinx will have to be written in reStructuredText, a markup language like Markdown, but with, of course, an entirely different syntax.

Let's setup Sphinx. Download it with pip:

(env)your_app$ pip install sphinx

Then change into the docs/ directory. Fortunately, Sphinx comes with a command-line utility called 'sphinx-quickstart' which will let you setup your basic Sphinx configuration by asking you a small set of questions about your favorite food and the number of cats you have. From what I remember. In any case, invoking this command (from within your_app/docs/):

docs$ sphinx-quickstart

will trigger a setup wizard. You can configure your project's documentation to your liking or just stick with the defaults (press enter), but do answer yes ('y') when it asks about splitting your build directory into a build and source folder (question 2) and also yes when it asks about autodoc. The latter will ensure that Sphinx reads our source code's documentation (from its docstring). You'll definitely want a Makefile too, so please don't disable it (it's 'yes' by default).  Here's my setup:

docs$ sphinx-quickstart
Welcome to the Sphinx 1.3.1 quickstart utility.

Please enter values for the following settings (just press Enter to
accept a default value, if one is given in brackets).

Enter the root path for documentation.
> Root path for the documentation [.]: 

You have two options for placing the build directory for Sphinx output.
Either, you use a directory "_build" within the root path, or you separate
"source" and "build" directories within the root path.
> Separate source and build directories (y/n) [n]: y

Inside the root directory, two more directories will be created; "_templates"
for custom HTML templates and "_static" for custom stylesheets and other static
files. You can enter another prefix (such as ".") to replace the underscore.
> Name prefix for templates and static dir [_]: 

The project name will occur in several places in the built documentation.
> Project name: your_app
> Author name(s): you

Sphinx has the notion of a "version" and a "release" for the
software. Each version can have multiple releases. For example, for
Python the version is something like 2.5 or 3.0, while the release is
something like 2.5.1 or 3.0a1.  If you don't need this dual structure,
just set both to the same value.
> Project version: 0.1.0
> Project release [0.1.0]: 

If the documents are to be written in a language other than English,
you can select a language here by its language code. Sphinx will then
translate text that it generates into that language.

For a list of supported codes, see
http://sphinx-doc.org/config.html#confval-language.
> Project language [en]: 

The file name suffix for source files. Commonly, this is either ".txt"
or ".rst".  Only files with this suffix are considered documents.
> Source file suffix [.rst]: 

One document is special in that it is considered the top node of the
"contents tree", that is, it is the root of the hierarchical structure
of the documents. Normally, this is "index", but if your "index"
document is a custom template, you can also set this to another filename.
> Name of your master document (without suffix) [index]: 

Sphinx can also add configuration for epub output:
> Do you want to use the epub builder (y/n) [n]: 

Please indicate if you want to use one of the following Sphinx extensions:
> autodoc: automatically insert docstrings from modules (y/n) [n]: y
> doctest: automatically test code snippets in doctest blocks (y/n) [n]: n
> intersphinx: link between Sphinx documentation of different projects (y/n) [n]: n
> todo: write "todo" entries that can be shown or hidden on build (y/n) [n]: y
> coverage: checks for documentation coverage (y/n) [n]: y
> pngmath: include math, rendered as PNG images (y/n) [n]: n
> mathjax: include math, rendered in the browser by MathJax (y/n) [n]: y
> ifconfig: conditional inclusion of content based on config values (y/n) [n]: y
> viewcode: include links to the source code of documented Python objects (y/n) [n]: y

A Makefile and a Windows command file can be generated for you so that you
only have to run e.g. `make html' instead of invoking sphinx-build
directly.
> Create Makefile? (y/n) [y]: y
> Create Windows command file? (y/n) [y]: n
...

Now, you'll want to fully enable Sphinx's ability to grab documentation out of your source code by editing the conf.py it created. Open it in your_app/docs/source/conf.py. At the very top, after the import statements, you'll see a commented line saying "# If extensions (or modules to document with autodoc) are in another directory ...". Comment out the single line of Python code below (with sys.path) and add the path to your_app's root directory, which is two relative directories up conf.py (see below).

While we're already in conf.py, let's do two more things. First of all, we'll want a minimum Sphinx version of 1.3 (second configuration option, needs_sphinx). Then, add 'sphinx.ext.napoleon' to the list of extension you see right below. I'll explain what that is in just a bit, but first confirm your conf.py looks like this now (the extensions list may vary according to the settings you selected, but napoleon should be there):

# ...
sys.path.insert(0, os.path.abspath('../../'))

# -- General configuration ------------------------------------------------

# If your documentation needs a minimal Sphinx version, state it here.
needs_sphinx = '1.0'

# ...
extensions = [
    'sphinx.ext.autodoc',
    'sphinx.ext.todo',
    'sphinx.ext.coverage',
    'sphinx.ext.mathjax',
    'sphinx.ext.ifconfig',
    'sphinx.ext.viewcode',
    'sphinx.ext.napoleon'
]

Now, we can add documentation to our source code. Usually, the standard reStructuredText + Sphinx way of adding method, class, module, parameter and return value descriptions, exception specifications and other elements of documentation would look something like this (for your_app/your_app/code.py):

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
Your application's badass source code.
"""

def square(x):
 """
 Squares a value.

 Equivalent to calling x**2.

 :param x: The value to square.
 :type x: int|float 
 :returns: The squared value.
 :rtype: int|float
 """

 return x * x


Look at that mess in square()'s docstring. The default reStructuredText syntax for docstrings is ugly and makes me cry. That's where napoleon comes in (the extension we added above). It lets us transform the above into the below:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

"""
Your application's badass source code.
"""

def square(x):
    """
    Squares a value.

    Equivalent to calling x**2.

    Arguments:
    x (int|float): The value to square.

    Returns:
    The squared value.
    """

    return x * x

Yes! Beauty! Of course, the output in Sphinx and ReadTheDocs will be the same, but if there's one thing I love then that's cats and source code documentation that looks good in the source code and in the output. You can read all about Napoleon and how to document your source code with it here. Note that this is not specifically "Napoleon Style", but actually the docstring layout recommended by the Google Style Guide for Python.

Next, we can have Sphinx generate documentation for us with the sphinx-apidoc command:

(env)docs$ sphinx-apidoc -o source ../your_app/

To the -o flag we must pass the output directory as its positional argument, in this case the 'source' directory Sphinx generated earlier. The last argument is then the package containing your source code, where Sphinx will pick up the documentation from your modules' docstrings. This will generate  one .rst file per module in source. Then, you can use the Makefile to generate the actual HTML documentation. After building, you'll find an index.html file under your_app/docs/build/html.

(env)docs$ make html
sphinx-build -b html -d build/doctrees   source build/html
Running Sphinx v1.3.1
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 0 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
no targets are out of date.
build succeeded.

Build finished. The HTML pages are in build/html.
(env)docs$ open build/html/index.html

Yey! Now it's really up to you regarding documentation -- the sky's the limit. I'll add some dummy text from www.lebowskiipsum.com and a picture of my cat to index.rst, just so you have an idea what could follow. In this case, I'll also delete your_app.rst and move its contents into modules.rst, because it's not worth having the extra file. Here are the now two remaining relevant files for your_app's documentation:

your_app/docs/source/index.rst: 

your_app
========

\

Contents
========

* `Introduction`_

* `Unbaking a cake with your_app`_

* `kbyethx`_

* `Source Code`_

\

Introduction
============

Lebowski ipsum stay out of Malibu, Lebowski! Dolor sit amet, consectetur adipiscing elit praesent ac magna justo pellentesque ac lectus. Vee could do things you only dreamed of, Lebowski. Quis elit blandit fringilla a ut. Darkness warshed over the Dude— darker'n a black steer's tookus on a moonless prairie night. There was no bottom. Turpis praesent felis ligula, malesuada suscipit malesuada non, ultrices non. We want that money, Lebowski. Urna sed orci ipsum.


Unbaking a cake with your_app
=============================

#. Take cake out of oven
#. Run your_app on it (may require root access rights)
#. Extract eggs, flour, sugar etc. 
#. Cure cancer
#. Profit

kbyethx
=======

.. image:: https://goo.gl/IpUmJn
 :alt: <3
 :align: center


Source Code
===========

.. toctree::
 :maxdepth: 2

 modules

your_app/docs/source/modules.rst: 

your_app
========

your_app.code
-------------

.. automodule:: your_app.code
    :members:
    :undoc-members:
    :show-inheritance:


One last thing to mention is that Sphinx supports quite a few themes. The default one, which you should be seeing if you've followed the previous steps, is called "alabaster". Others include the one from ReadTheDocs and also some using Bootstrap. They're quite easy to setup and play around with. Note also that alabaster is very, very customizable (see more on its Github page).

That's it for documentation (we'll send it to ReadTheDocs once we have a GitHub repo). In the next section, we'll setup a few more files before we go live, first on PyPI and then GitHub and everywhere else.

Resources:

Additional Files


A proper repository needs a bunch of more files to be ready for distribution and publicity and fame.
These are:
  • README.rst
  • LICENSE
  • HISTORY.rst

README.rst


The README should be clear. It is the main entry point to your project and should make clear what problem you are solving and how you are bettering the world; a basic usage example if your project happens to be a library; how to install your application; where to find documentation; the license you're using and whatever else you'd like and that makes sense. Have a look at these resources prior to writing a README:

Note that your README must be in reStructuredText and not in Markdown, because PyPI will only parse reStructuredText. 

LICENSE


A project without a license is implicitly closed-source. We don't want that and thus need a license. There are many options, though I always choose the MIT-license. It's permissive and lovely. One thing that's especially wonderful is that you can get your own, personal, free and ever-online MIT-license hosted under mit-license.org. See this repo for more information. Here's my license. Note that the license file doesn't need a file extension (common practice not to add one).

HISTORY.rst


This should be your changelog file (often called CHANGELOG.rst or CHANGE.rst), where you would add a list of your most important commits for every new version release.

Getting Ready for Distribution


We are very close to being able to upload our project to PyPI and then Github. However, now, we'll need a few more more files. These are very important:
  • setup.py
  • setup.cfg
  • MANIFEST.in
  • requirements.txt
  • Makefile
However, first, we'll have to edit another file: your_app/your_app/__init__.py. As I've come to see, it is a common idiom to store relevant project information such as the project version, the author's name and e-mail address, the project's license and more inside this file. For your_app, it should look like this:

# -*- coding: utf-8 -*-

__title__ = 'your_app'
__version__ = '0.1.0'
__author__ = '<you>'
__license__ = 'MIT'
__copyright__ = 'Copyright 2015 <you>'

Setup.py


setup.py is the Python module used to build and install your project. Here's the file for your_app:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

"""
setup.py script for setuptools.
"""

import re

from setuptools import setup, find_packages

version = ''

with open('your_app/__init__.py') as init:
    version = re.search(r'^__version__\s*=\s*[\'"]([^\'"]*)[\'"]',
       init.read(), re.MULTILINE).group(1)

with open('README.rst') as readme:
    long_description = readme.read()

requirements = [
    # Your project's requirements
]

test_requirements = [
    "unittest2==1.1.0"
]

setup(
    name='your_app',

    version=version,

    description='Your Python application.',
    long_description=long_description,

    author='You',
    author_email='your@email',

    license='MIT',

    classifiers=[
    'Development Status :: 4 - Beta',

    'Intended Audience :: Developers',
    'Topic :: Software Development',

    'License :: OSI Approved :: MIT License',

    'Programming Language :: Python',
    'Programming Language :: Python :: 2',
    'Programming Language :: Python :: 2.6',
    'Programming Language :: Python :: 2.7',
    'Programming Language :: Python :: 3',
    'Programming Language :: Python :: 3.2',
    'Programming Language :: Python :: 3.3',
    'Programming Language :: Python :: 3.4',
    ],

    keywords='your_app',

    include_package_data=True,

    packages=find_packages(exclude=['contrib', 'docs', 'tests*']),

    install_requires=requirements,

    test_suite="tests",

    tests_require=test_requirements
)

  • Now, first of all, in setup.py, we must import the setup method from the setuptools module, which takes a great number of keyword arguments that enable us to configure our project's metadata. We'll also need setuptools find_packages to make it easy for us to specify which packages setup.py should handle.
  •  Then we grab our project's version string by doing a regular expression search in our __init__.py under the your_app/your_app directory (for this we need the re module). Note that we cannot directly import our module your_app and just retrieve its __version__ directly, as someone who just got your project and has not yet executed the setup procedure may not have all your project's dependencies. Therefore, certain imports may fail.
  • For PyPI, we'll want a long_description, which is simply our README.
  • Requirements/dependencies to run your project's application code successfully should be put into the requirements list. your_app doesn't have any, but I'm leaving it here for you to see. When someone then setup.py installs or pip installs your project, this will also install the dependencies.
  • Test requirements go into the appropriately named list. your_app, for example, requires unittest2, version 1.1.0. Note that each element in these lists of requirements are commands for pip install, i.e. when setting up your project, pip install <element> will be called for each element in these lists.
  • Inside setup(), you'll see a lot of meta-data arguments, which you can fill out for your persona and application
  • Regarding classifiers, you can see the list of available classifiers you can choose from here.
  • By setting include_package_data to True, we can later on specify which additional data files we want to include in our distribution. setup.py will then look for a manifest file (MANIFEST.in) which we'll write in a bit. By default, Python files are included, your README is included and a few other files.
  • For the packages we want to distribute, we let setuptools' find_packages function handle that, as can be seen above.
  • We specify install and test requirements with install_requires and test_requires.
  • By handing over the path to our test directory ("tests") to the test_suite argument, we will later on be able to run our tests with unittest2 directly from the setup file (see below).

Once we're done, you'll be able to build your distribution files with:

(env)your_app$ python setup.py build sdist bdist_wheel

which builds a source and pre-built distribution (using the Python wheel format), and/or install your project with:

(env)your_app$ python setup.py install

and/or test your project with:

(env)your_app$ python setup.py test

and/or do many other things, such as registering your application with PyPI (more on that soon), which you can see by issuing: 

(env)your_app$ python setup.py --help

Resources:

setup.cfg


As quickly touched upon above, there are two main variants of distribution files for your project, the first is a source distribution (sdist) -- the standard distribution --- and the second is a wheel distribution (bdist_wheel). According to the documentation for wheels: "A wheel is a ZIP-format archive with a specially formatted filename and the .whl extension. It is designed to contain all the files for a PEP 376 compatible install in a way that is very close to the on-disk format. ". The main advantage of a wheel is that it is quicker to install than a source distribution. Note that you must have the wheel package installed to use wheels, but you should already have it if you're in a virtual environment. In any case, we can provide additional instructions on how to build wheels with a setup.cfg file. Most importantly, we'll want only one single wheel distribution for all Python versions. For this, setup.cfg should hold these two lines:

[wheel]
universal=1

MANIFEST.in


In our setup.py script, we set include_package_data to True. As a consequence, setup.py will look for a MANIFEST.in file, where we can specify additional data files such as README.rst, LICENSE and others, that we want included in our distribution. It can look like this:

include *.rst
include LICENSE
include Makefile

recursive-include tests *
recursive-exclude * __pycache__
recursive-exclude * *.py[co]

recursive-include docs *.rst conf.py Makefile

Resources:

requirements.txt


This file contains a list of pip packages needed to run your code successfully, explicitly put here alongside having them specified in setup.py. Each line in requirements.txt is a command to pip install.  For example, we'll need unittest2 version 1.1.0 to run our tests, so requirements.txt would contain at least this one line: "unittest2==1.1.0". If you then run

$ pip install -r requirements.txt

pip will execute each line in requirements.txt as if you had called pip install <line>. The best way of getting all requirements for your project is to use:

(env)your_app$ pip freeze

which prints a list of packages installed with pip, either directly by us or as dependencies of those packages. Note that the value of the virtual environment really comes to show here, as pip freeze will only show what we installed for this specific project, whereas it would show an amalgamation of packages for different projects with different requirements if we hadn't used a virtual environment (and also not for other projects). In any case, for your_app, pip freeze shows this:

(env)your_app$ pip freeze
alabaster==0.7.6
Babel==2.0
docutils==0.12
Jinja2==2.8
linecache2==1.0.0
MarkupSafe==0.23
pluggy==0.3.0
py==1.4.30
Pygments==2.0.2
pytz==2015.4
six==1.9.0
snowballstemmer==1.2.0
Sphinx==1.3.1
sphinx-rtd-theme==0.1.8
traceback2==1.4.0
unittest2==1.1.0
virtualenv==13.1.0
wheel==0.24.0

Note that most of these packages were installed as dependencies of the few packages we installed (unittest2, virtualenv and Sphinx -- wheel is installed in a virtualenv by default). We can then generate a requirements.txt file by directing pip freeze's output into it:

(env)your_app$ pip freeze > requirements.txt

Resources:

Makefile


We'll want a Makefile to make life easy for us. Mainly taken from here, but modified for our setup:

.PHONY: clean-pyc clean-build docs clean

help:
 @echo "clean - remove all build, test, coverage and Python artifacts"
 @echo "clean-build - remove build artifacts"
 @echo "clean-pyc - remove Python file artifacts"
 @echo "clean-test - remove test and coverage artifacts"
 @echo "lint - check style with flake8"
 @echo "test - run tests quickly with the default Python"
 @echo "coverage - check code coverage quickly with the default Python"
 @echo "docs - generate Sphinx HTML documentation, including API docs"
 @echo "release - package and upload a release"
 @echo "dist - package"
 @echo "install - install the package to the active Python's site-packages"

clean: clean-build clean-pyc clean-test

clean-build:
 rm -rf build/
 rm -rf dist/
 rm -rf .eggs/
 find . -name '*.egg-info' -exec rm -rf {} +
 find . -name '*.egg' -exec rm -f {} +

clean-pyc:
 find . -name '*.pyc' -exec rm -f {} +
 find . -name '*.pyo' -exec rm -f {} +
 find . -name '*~' -exec rm -f {} +
 find . -name '__pycache__' -exec rm -rf {} +

clean-test:
 rm -rf .tox/
 rm -f .coverage
 rm -rf htmlcov/

lint:
 pylint ecstasy

test:
 python setup.py test

coverage:
 coverage run --include="./*" -m unittest2
 coverage report -m
 coverage html
 open htmlcov/index.html

docs:
 $(MAKE) -C docs clean
 $(MAKE) -C docs html
 open docs/build/html/index.html

release: clean
 twine upload sdist
 twine upload bdist_wheel

dist: clean
 python setup.py sdist
 python setup.py bdist_wheel

install: clean
 python setup.py install

Tox


I'll just talk about one more thing before we send your_app online: tox. The tox automation project, according to itself, "aims to automate and standardize testing in Python". What's great about it is that it lets us test whether or not our project can be built and distributed correctly across multiple Python versions, without having to setup virtual environments ourselves (done by tox). I've waited till now for this, because we needed a setup.py script. Install tox with pip:

(env)your_app$ pip install tox

You'll then need a tox.ini configuration file, in which we can specify which python versions we'll want to test our distribution for, how to install requirements and what command to use to run tests. Note that what Python versions you can use tox with depends on which Python versions you have installed on your system, though you always grab more Python versions with pyenv or brew. tox.ini could look like this:

[tox]
envlist = py26, py27, py31, py34

[testenv]
deps = -rrequirements.txt
commands = python setup.py test

Now you run tox to see if all is well for your application in multiple environments:

(env)your_app$ tox
...
_________________________ summary __________________________

  py26: commands succeeded
  py27: commands succeeded
  py31: commands succeeded
  py34: commands succeeded
  congratulations :)

PyPI


We are now ready to send our application to the cheese shop (PyPI). First, we'll send it to the test (staging) server and then to the real one. Either way, we'll want to build our project. Run:

(env)your_app$ python setup.py build sdist bdist_wheel

This creates the appropriate distributions in the your_app/dist/ directory. You may want to execute

your_app$ python setup.py check

before doing anything next, to make sure your project is ready (but don't care about the url for now, we'll add it later).

TestPyPI


Sign up for the staging server of PyPi here. Note that the staging server is cleaned periodically, so your user profile and repository may vanish from the staging server (not the real one after) at one point in the future. In any case, next, create a .pypirc file in your root directory. ~/pypirc should contain, for now (replace the username and password):

[distutils]
index-servers =
    test

[test]
repository = https://testpypi.python.org/pypi
username = <your_username>
password = <your_password>

Then, you can register your application on the test server (note that this will warn about a missing url, we'll add it later):

(env)your_app$ python setup.py register -r test

Go have a look on testpypi.python.org if all looks well for your application on the testing server. If so, we can then upload the distribution files (it's only registered for now). For this, you can either run (-r stands for repository)

(env)your_app$ python setup.py sdist bdist_wheel upload -r test

or use twine. Twine is like the command you just saw, but more secure. You can even use PGP encryption. Get twine with pip

(env)your_app$ pip install twine

Then issue this command to upload your files (pass the -s flag to sign it with PGP):

twine upload -r test dist/*

Either way, you could now create a new virtual environment and test installing your package from the test server:

(env)your_app$ virtualenv test
Using real prefix '/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7'
New python executable in test/bin/python2.7
Also creating executable in test/bin/python
Installing setuptools, pip, wheel...done.
(env)your_app$ venv test
(test)your_app$ pip install -i https://testpypi.python.org/pypi your_app
Collecting your-app
  Using cached https://testpypi.python.org/packages/2.7/y/your_app/your_app-0.1.0-py2.py3-none-any.whl
Installing collected packages: your-app
Successfully installed your-app-0.1.0
(test)your_app$ deactivate 
your_app$ rm -r test
your_app$ venv env

PyPI


We can now go for the the real thing. Register on PyPI, if you haven't yet (separate registration from the test server). Then add change ~/.pypirc so that it looks like this:

[distutils]
index-servers =
    pypi
    test

[pypi]
repository = https://pypi.python.org/pypi
username = <your_username>
password = <your_password>

[test]
repository = https://testpypi.python.org/pypi
username = <your_username>
password = <your_password>

And repeat the process for the live server:

your_app$ python setup.py register
...

Registering your_app to https://pypi.python.org/pypi
Server response (200): OK

and then either 

your_app$ python setup.py sdist bdist_wheel upload

or

your_app$ twine upload dist/*

Voilà, c'est on PyPI, as my French dog would always say. If I had a French dog. Who could speak.

Resources:

Github


Register your project on Github now -- I'm sure you know how to do it. Once you have a URL, add it to your_app/your_app/__init__.py and the setup.py script. The url for this tutorial repository is https://github.com/goldsborough/your_app and I'll use it as a placeholder for your project's url from now on.

your_app/your_app/__init__.py:

...
__title__ = 'your_app'
__url__ = "https://github.com/goldsborough/your_app"
__version__ = '0.1.0'
...

setup.py:

setup(
...
author='You',
author_email='your@email',

url="https://github.com/goldsborough/your_app",

license='MIT',
...
)

Now that we have a little bit more meta-data, we can re-register the project on PyPI. This will only re-configure the meta data:

your_app$ python setup.py register

Adding the PyPI badge


First, we can add a "PyPI version" badge to our project. For this, visit this web page. There, you can enter the name of your application and retrieve the reStructuredText code to include in your README.rst file. For your_app, that would be:

.. image:: https://badge.fury.io/py/your_app.svg
    :target: http://badge.fury.io/py/your_app

On Github, your README will then show a badge like this:


Adding your MIT License badge


If you're awesome and have your own mit-license.org subdomain (I talked about it earlier), you can also get a badge for that, by including the following code in your README.rst:

.. image:: https://img.shields.io/github/license/mashape/apistatus.svg
 :target: http://goldsborough.mit-license.org

In the target command, you would replace goldsborough with your name for mit-license.org. That gives you this badge:



You can change the color by making your own badge on http://shields.io.

Adding the Gratipay badge


Gratipay, formerly known under the, IMHO, much nicer name, Gittip, is a way for people to tip you for your great work. Sign up here: https://gratipay.com/

And either go to https://gratipay.com/~<your_user_name>/widgets/ or just change this code for you:

.. image:: http://img.shields.io/gratipay/goldsborough.svg
:target: https://gratipay.com/~goldsborough/

This will give you this badge (feel free to click and donate <3):


Setting up Landscape.io


Landscape.io is a web tool for automated linting, code correctness and style rating. Everytime you push a commit to your repo, landscape.io will run pylint on it. Register for free and add your repository, no additional configuration required. On your repository's overview page on landscape.io, you'll see the badge in the top right corner. Clicking on it redirects you to badge heaven. The badge looks like this (with your code health):


Setting up Code Climate


Code Climate is another web application that performs code rating, or more specifically "static analysis". It differs from Landscape.io in that it really analyzes your code for quality in terms of how well it is structured and how maintainable is, rather than only checking for bugs and style as Landscape.io does. For example, for my latest project, ecstasy, it's giving me a pretty low GPA (2.8) because I have "high cyclomatic complexity" in certain areas. At the same, your_app gets a 4.0 (hehe). Sign up on their website and add your repository. You can get this kind of badge:



by visiting Settings > Badges in your Code Climate dashboard and adding the reStructuredText to your README.rst.

Setting up Coveralls


At the start of this article, I talked about code coverage. We examined our code's coverage with the command-line utility of the same name to understand how much of our code is really being executed when running our tests. Coveralls.io is the web counterpart for this -- sign up there and add your repository. Once you're done, you can see the badge being offered at the top. It looks like this:



However, there's a little extra configuration needed here. First of all, download python-coveralls with pip (and do a pip freeze > requirements.txt again, and also add it to your test_requirements in setup.py):

(env)your_app$ pip install python-coveralls

This utility will let us interact with coveralls from the command-line. Then, add this .coveralls.yml file to your repository (we'll setup Travis-CI in a bit):

your_app/.coveralls.yml:

service_name: travis-ci

Once you've added your repository on coveralls.io, you'll see setup-instructions for coveralls and Ruby. We don't need the instructions, but you'll see your repo_token in the instructions. Copy it to your clipboard and issue this command, replacing <your_repo_token> with your actual repo token:

(env)your_app$ COVERALLS_REPO_TOKEN=<your_repo_token> coveralls

That'll and you should see results once we've setup Travis CI and pushed a commit afterwards.

Resources:

Setting up Travis-CI


Travis CI (Continuous Integration) is an awesomely awesome distributed build system for continuous integration in the open-source community, used for running your tests online. It is very, very helpful when working in teams or on open-source projects with multiple contributors working on improvements and features and fixes simultaneously. Sign up on travis-ci.org, the free version for open-source projects, and not on travis-ci.com, their paid counterpart. After adding your project, you'll see the badge in your overview page. Clicking on it and switching to 'rst' yields the reStructuredText code you can add to your README.rst to get this badge:



More configuration. Add this .travis.yml file to your root directory.

your_app/.travis.yml:
language: python
python:
  - "2.6"
  - "2.7"
  - "3.2"
  - "3.3"
  - "3.4"
install:
  - "pip install ."
  - "pip install -r requirements.txt"
script: coverage run --include="./*" -m unittest2
notifications:
  email: 
    - petergoldsborough@hotmail.com
after_success:
  - coveralls

Here, we specify

  • language: The language of our project ("python")
  • python: Against which Python versions to test (e.g. "3.4")
  • install: What shell commands to run to install the project
  • script: What shell commands to run to execute our tests. Notice that we use coverage here like I showed you above. This is important for the after_success step and integration with coveralls.io. If you don't want coverage, you can just execute the "unit2" (make sure unittest2 is in your dependencies).
  • notifications: How and where to notify you of failures and changes (you can change this on travis-ci.org)
  • after_success: What to do when all our tests have passed (there's also after_failure). We want to call coveralls here, which will send the report generated by the coverage report to coveralls.io (python-coveralls interacts with the coveralls.io API).


There's more you can do with this configuration file. The full life cycle of a Travis "job" is described by these YAML statements:
  1. before_install
  2. install
  3. after_install
  4. before_script
  5. script
  6. after_script
  7. after_success or after_failure

See here: http://docs.travis-ci.com/user/customizing-the-build/#The-Build-Lifecycle

Maybe you'd also like a command-line client: https://github.com/travis-ci/travis.rb

In any case, you should now have Travis-CI and coveralls.io ready, so commit and push and check everything is wonderful on travis-ci.org and on coveralls.io.

Resources:

Setting up Gitter


The last thing we'll want to do for our kick-ass project is to add Gitter. Gitter is a chatroom for Github repositories where you can talk with fellow contributors about your project. Sign up on gitter.im and click on "create a room" in the bottom left corner, then on "repository" and lastly on your Github repo. This will create a room on Gitter. Say hi on your_app's: https://gitter.im/goldsborough/your_app

Now you'll want to setup Gitter and integrate other services. If you're in your room on Gitter, it'll say "Configure your integrations" on the right sidebar. Click that and then integrate GitHub, Coveralls and Travis CI. There are nice instructions for all of that. 

Once you're done you have to patience yourself a little, but Gitter will send you a Pull Request on GitHub to suggest adding its marvelous badge, which looks like so (click on it to shoot me a message in your_app's chatroom):


Setting up ReadTheDocs


The very last thing we'll do, and I almost forgot about it, is setup ReadTheDocs. Sign up on https://readthedocs.org and click on "Add Project" under your user profile. You can then select to "Import from Github" and, after possibly syncing your repos, you can select your Github repo. You'll then have to "Build" it and ReadTheDocs will automagically search your docs/ folder for everything it needs. After, you can click on "View Docs" and voilà, your Sphinx docs. Here are those from your_app: http://your-app.readthedocs.org/

For every <thing>.readthedocs.org there's also an alternative domain under <thing>.rtfd.org (read the fabulous docs -- right?), like your_app.rtfd.org

Farewell


I hope I could share with you what I learnt while setting up my own project. Creating and distributing such a project can be quite tough and there's so, so much to do when setting up a project other than coding (unfortunately). There's most likely many things I did wrong here, too, so please feel free to offer suggestions. 

One last resource I'd like to share with you is cookiecutter, which automates much of the process I just outlined.

But otherwise, thanks for reading and farewell!

Posted by at 1 comment :
Labels: , , , , , ,

Wednesday, April 1, 2015

Python's built-in zip() function in C++

We all know, use and love Python's built-in zip() function.

In Python, the implementation of the zip() function probably looks like, or at least could be replicated by this cute little lambda:

zip = lambda *args: [[i[n] for i in args] for n in range(len(min(args,key=len)))]

Python, being a weakly typed programming language with support for list comprehensions as well as elements of functional programming such as the lambda function feature shown here, makes it easy to create these super concise, variable length, who-cares-about-types-anyway functions.

Now, if we leave type utopia and enter the world of strongly typed programming languages such as C++, things look a little different. Types suddenly matter and variable length parameter lists take a little more thought to implement than simply adding an asterisk (note to self: great success, no longer spelt it asterix). As a matter of fact, parameter packs and variadic template functions, the combination of features we'll use in this tutorial, were only added in C++11, but provide some much-needed and really, really cool functionality.

At the end of this tutorial, this C++ code will work:

int main(int argc, char * argv [])
{
    std::vector<int> a = {1,2,3};
    std::vector<int> b = {4,5,12,44};
    std::vector<int> c = {6,7,8,9};
    
    for (auto i : zip(a, b, c))
    {
        for (auto j : i)
        {
            std::cout << j << "\t";
        }
        
        std::cout << std::endl;
    }
}

Pretty concise, no?

Before I show you and describe the code that makes the above behavior possible, let me write a few words about variadic template functions in C++. Basically, they enable us to have template functions that take a variable number of parameters. Example:

void foo()
{ }

template<typename T, typename... Args>
void foo(const T& head, const Args&... tail)
{
    std::cout << head << std::endl;
    
    foo(tail...);
}

Variadic template functions are created by having a parameter pack argument for a template function or class. This parameter pack type is denoted by the "typename" keyword followed by an ellipsis ("...") and a name, in the function above "Args" (seems to be the conventional name for C++ parameter packs). When a function then takes such a parameter pack as an argument, as does foo() above, you can call the function with a variable number of arguments, e.g. foo(1, 2, 3, 4) or foo("lorem", "ipsum").

The problem is that you cannot actually retrieve any values from this parameter pack, you can only expand it, i.e. you can't do args[2] to get the third element of a parameter pack "args", you can only do args... (note the ellipsis) which will yield the full list of parameters passed to it.

To tackle this problem, we have two template types: T and Args. Args is the template parameter pack I already introduced you to. By having a second template type T as the first argument, we can capture the first item passed the function, which will enable us to do things with it! We can then recursively call this variadic template function each time making one item from the parameter pack the "head" (type T) which we can use, while packing all other arguments into the parameter pack, the "tail" (of type Args...), which we pass on to the next stack. When the tail is empty at the very end, the overload of the function taking no parameters is called.

So, what happens when we call foo(1,2,3)? The "head" of type T will be 1 at first, while 2 and 3 are stored in the parameter pack Args. Then we call foo(tail...), thereby expanding the parameter pack. This is equivalent to foo(2,3). In the next function stack, head is equal to 2 and 3 is stored in the tail. The parameter pack expansion is then equivalent to foo(3), thus setting head equal to 3 in this stack while the parameter pack is empty. When we call foo(tail...) one last time, it will be equivalent to foo(). This is why we need a parameter-less overload of foo(), even though we don't have to do anything in it. It will just start the movement back down the stacks.

The output would therefore be:

>> foo(1,2,3);
>> 1
>> 2
>> 3

Now that the behavior of C++ variadic template functions should be clear(er), I can show you the code for the zip() function in C++ (also found here as a gist):

template<typename T>
class Zip
{
    
public:
    
    typedef std::vector<T> container_t;
    
    template<typename... Args>
    Zip(const T& head, const Args&... args)
    : items_(head.size()),
      min_(head.size())
    {
        zip_(head, args...);
    }
    
    inline operator container_t() const
    {
        return items_;
    }
    
    inline container_t operator()() const
    {
        return items_;
    }
    
private:
    
    template<typename... Args>
    void zip_(const T& head, const Args&... tail)
    {
        // If the current item's size is less than
        // the one stored in min_, reset the min_
        // variable to the item's size
        if (head.size() < min_) min_ = head.size();
        
        for (std::size_t i = 0; i < min_; ++i)
        {
            // Use begin iterator and advance it instead
            // of using the subscript operator adds support
            // for lists. std::advance has constant complexity
            // for STL containers whose iterators are
            // RandomAccessIterators (e.g. vector or deque)
            typename T::const_iterator itr = head.begin();
            
            std::advance(itr, i);
            
            items_[i].push_back(*itr);
        }
        
        // Recursive call to zip_(T, Args...)
        // while the expansion of tail... is not empty
        // else calls the overload with no parameters
        return zip_(tail...);
    }
 
    inline void zip_()
    {
        // If min_ has shrunk since the
        // constructor call
        items_.resize(min_);
    }
 
    /*! Holds the items for iterating. */
    container_t items_;
    
    /*! The minimum number of values held by all items */
    std::size_t min_;
    
};
 
template<typename T, typename... Args>
typename Zip<T>::container_t zip(const T& head, const Args&... tail)
{
    return Zip<T>(head, tail...);
}

As you can see, the zip() function is a variadic template function, but it is only an extra level of indirection to a template class, Zip<T>, whose constructor is again a variadic template function. The code is commented, but I will explain the general procedure of a call to zip().

Just to re-cap: the aim of the zip function and the Zip class is to restructure a variable number of containers into a new set of containers, where each container in the new set holds all items for a single column of the old set. Because we want each container of the new set to have the same length, we must use the minimum number of columns in the old set for our restructuring process. This sounds kind of complicated, but we're essentially just transposing the old set of containers (switching rows for columns).

We can't completely escape strong typing, even with the use of templates, because we have to store the values from the old set of containers into some other set while restructuring. Therefore, Zip<T> is a template class where T is the type of container we're passing, e.g. std::vector<int> in the main loop at the top of this post. When we call zip with a set of vectors of ints, the private variable "items_" will be of type std::vector<std::vector<int>> (typedef container_t). items_ is the object we return from zip(), by the way, so the return type of zip() is Zip<T>::container_t.

So a function call to zip(T, Args) will be forwarded to Zip<T>(T, Args), which again forwards to its own variadic template function that does the actual work of unpacking and restructuring the containers: zip_(T, Args), which has the overload for when Args is empty: zip_().

So how do we make the restructured containers / items retrievable? I decided that the best and most intuitive way would be to overload the implicit conversion operator to container_t, as well as overload the function call operator of the Zip<T> class. The overload of the conversion operator is useful any time we want to directly convert the Zip object to the container_t, e.g. as we do in the zip() function (the implicit conversion happens when we return the object). However, I figured that one case in which this would not work is if passing the Zip object as a template parameter to a function that expects the zipped container and not the Zip class, as there would be no implicit conversion in that case. To solve this problem, we have the possibility of explicitly retrieving items_ through the function call operator:

template<typename T>
void foo(const T& t)
{
    std::cout << t.size() << std::endl;
}

Zip<std::list<int>> zipObject(a,b,c);

foo(zipObject); // Error, Zip has no member size()
    
foo(zipObject()); // All okay, Zip::container_t has member size()

Given that the Zip object is supposed to simulate function behavior as well as possible, I chose the function call operator overload over having an explicit get() member or anything, thus making Zip into a true functor object. Of course, ideally, we shouldn't be handling the Zip object ourselves, but just be calling the zip() function which returns Zip::container_t in a much more intuitive way.

So after all this explaining business, what actually is the output of the zip() function. Does it even work?

int main(int argc, char * argv [])
{
    std::vector<int> a = {1,2,3};
    std::vector<int> b = {4,5,12,44};
    std::vector<int> c = {6,7,8,9};
    
    for (auto i : zip(a, b, c))
    {
        for (auto j : i)
        {
            std::cout << j << "\t";
        }
        
        std::cout << std::endl;
    }
}

Output:

>>> 1    4    6 
>>> 2    5    7 
>>> 3    12   8

Yey! Thanks for reading! :)
Posted by at 2 comments :
Labels: , , ,
Location: United States

Saturday, September 20, 2014

Building a text editor with PyQt




I have always enjoyed building beautiful Graphical User Interfaces (GUIs) to the back-end computations, number-crunching and algorithms of my programs. For Python, my GUI library of choice is the Python binding for Qt, PyQt. This tutorial will show you how you can use PyQt to build a simple but useful rich-text editor. The first part of the tutorial will focus on the core features and skeleton of the editor. In the second part of the tutorial we'll take care of text-formatting and in the third part we'll add some useful extensions like a find-and-replace dialog, support for tables and more.

Before we get started, two things:


Once you`re set up and ready to go, we can embark on our journey to create a totally awesome text editor.

An empty canvas

We start out with an empty canvas, a bare-minimum PyQt application:

import sys
from PyQt4 import QtGui, QtCore
from PyQt4.QtCore import Qt

class Main(QtGui.QMainWindow):

    def __init__(self, parent = None):
        QtGui.QMainWindow.__init__(self,parent)

        self.initUI()

    def initUI(self):

        # x and y coordinates on the screen, width, height
        self.setGeometry(100,100,1030,800)

        self.setWindowTitle("Writer")

def main():

    app = QtGui.QApplication(sys.argv)

    main = Main()
    main.show()

    sys.exit(app.exec_())

if __name__ == "__main__":
    main()

The first thing we need to do is import the sys module, which PyQt needs to start our application, as well as all the necessary modules from the PyQt4 package (PyQt5 if you have the newer version).

We'll call our class Main and let it inherit from PyQt's QMainWindow class. In the __init__ method, we initialize the parent class as well as all the UI settings for our application. The latter we just pack into the initUI() method. At the moment, the only settings we need are those concerning position on the screen, the size of the window and the window's title. We can set the first two using the setGeometry() method, which lets us set the x and y coordinates of the window on the screen, the width and lastly its height. We also set our application's window title using the setWindowTitle() method. For simplicity, we'll just call our text editor Writer.

Lastly, we need a main function that takes care of instantiating and displaying our window. We do so by creating a new Main object and calling its show() method.

And there was text

 Now that we have a basic PyQt application up and running, we can start making it look more like a text editor:

def initToolbar(self):

  self.toolbar = self.addToolBar("Options")

  # Makes the next toolbar appear underneath this one
  self.addToolBarBreak()

def initFormatbar(self):

  self.formatbar = self.addToolBar("Format")

def initMenubar(self):

  menubar = self.menuBar()

  file = menubar.addMenu("File")
  edit = menubar.addMenu("Edit")
  view = menubar.addMenu("View")

def initUI(self):

    self.text = QtGui.QTextEdit(self)
    self.setCentralWidget(self.text)

    self.initToolbar()
    self.initFormatbar()
    self.initMenubar()

    # Initialize a statusbar for the window
    self.statusbar = self.statusBar()

    # x and y coordinates on the screen, width, height

    self.setGeometry(100,100,1030,800)

    self.setWindowTitle("Writer")

I left out everything that stayed unchanged from the previous code. As you can see in the initUI() function, we first create a QTextEdit object and set it to our window's "central widget". This makes the QTextEdit object take up the window's entire space. Next up, we need to create three more methods: initToolbar(), initFormatbar() and initMenubar(). The first two methods create toolbars that will appear at the top of our window and contain our text editor's features, such as those concerning file management (opening a file, saving a file etc.) or text-formatting. The last method, initMenubar() creates a set of drop-down menus at the top of the screen.
As of now, the methods contain only the code necessary to make them appear.

For the initToolbar() and initFormatbar() methods, this means creating a new toolbar object by calling our window's addToolBar() method and passing it the name of the toolbar we're creating. Note that in the initToolbar() method, we need to also call the addToolBarBreak() method. This makes the next toolbar, the format bar, appear underneath this toolbar. In case of the menu bar, we call the window's menuBar() method and add three menus to it, "File", "Edit" and "View". We'll populate all of these toolbars and menus in a bit.

Lastly, in the initUI() method, we also create a status bar object. This will create a status bar at the bottom of our window.

An icon is worth a thousand words

 Before we start injecting some life into our text editor, we're going to need some icons for its various features. If you had a look at the GitHub repository I linked to at the top of this post, you might have noticed that it contains a folder full of icons. I recommend that you download the repo (if you haven't yet) and copy the icons folder into your working directory. The icons are from iconmonstr, completely free and require no attribution.

File management

 Now that we have a basic text editor skeleton in place, we can add some meat to the bone. We'll start with the functions concerning file management.

__init__():

def __init__(self, parent = None):
    QtGui.QMainWindow.__init__(self,parent)

    self.filename = ""

    self.initUI()

initToolbar(): 

def initToolbar(self):

  self.newAction = QtGui.QAction(QtGui.QIcon("icons/new.png"),"New",self)
  self.newAction.setStatusTip("Create a new document from scratch.")
  self.newAction.setShortcut("Ctrl+N")
  self.newAction.triggered.connect(self.new)

  self.openAction = QtGui.QAction(QtGui.QIcon("icons/open.png"),"Open file",self)
  self.openAction.setStatusTip("Open existing document")
  self.openAction.setShortcut("Ctrl+O")
  self.openAction.triggered.connect(self.open)

  self.saveAction = QtGui.QAction(QtGui.QIcon("icons/save.png"),"Save",self)
  self.saveAction.setStatusTip("Save document")
  self.saveAction.setShortcut("Ctrl+S")
  self.saveAction.triggered.connect(self.save)

  self.toolbar = self.addToolBar("Options")

  self.toolbar.addAction(self.newAction)
  self.toolbar.addAction(self.openAction)
  self.toolbar.addAction(self.saveAction)

  self.toolbar.addSeparator()

  # Makes the next toolbar appear underneath this one
  self.addToolBarBreak()

initMenubar():

file.addAction(self.newAction)
file.addAction(self.openAction)
file.addAction(self.saveAction)

Below the initUI() method:

def new(self):

    spawn = Main(self)
    spawn.show()

def open(self):

    # Get filename and show only .writer files
    self.filename = QtGui.QFileDialog.getOpenFileName(self, 'Open File',".","(*.writer)")

    if self.filename:
        with open(self.filename,"rt") as file:
            self.text.setText(file.read())

def save(self):

    # Only open dialog if there is no filename yet
    if not self.filename:
        self.filename = QtGui.QFileDialog.getSaveFileName(self, 'Save File')

    # Append extension if not there yet
    if not self.filename.endswith(".writer"):
        self.filename += ".writer"

    # We just store the contents of the text file along with the
    # format in html, which Qt does in a very nice way for us
    with open(self.filename,"wt") as file:
        file.write(self.text.toHtml())

As you might have noticed, all the actions we'll be creating for our text editor follow the same code pattern:
  • Create a QAction and pass it an icon and a name
  • Create a status tip, which will display a message in the status bar (and a tool tip if you hover the action)
  • Create a shortcut
  • Connect the QAction's triggered signal to a slot function
Once you've done this for the "new", "open" and "save" actions, you can add them to the toolbar, using the toolbar's addAction() method. Make sure you also call the addSeparator() method, which inserts a separator line between toolbar actions. Because these three actions are responsible for file management, we want to add a separator here. Also, we want to add these three actions to the "file" menu, so in the initMenubar() method, we add the three actions to the appropriate menu.
Next up, we need to create the three slot functions that we connected to our action in the initToolbar() method. The new() method is very easy, all it does is create a new instance of our window and call its show() method to display it.

Before we create the last two methods, let me mention that we'll use ".writer" as our text files' extensions. Now, for open(), we need to open PyQt's getOpenFileName dialog. This opens a file dialog which returns the name of the file the user opens. We also pass this method a title for the file dialog, in this case "Open File", the directory to open initially, "." (current directory) and finally a file filter, so that we only show ".writer" files. If the user didn't close or cancel the file dialog, we open the file and set its text to our text editor's current text.

Lastly, the save() method. We first check whether the current file already has a file name associated with it, either because it was opened with the open() method or already saved before, in case of a new text file. If this isn't the case, we open a getSaveFileName dialog which will again return a filename for us, given the user doesn't cancel or close the file dialog. Once we have a file name, we need to check whether the user already entered our extension when saving the file. If not, we add the extension. Finally, we save our file in HTML format (which stores style as well), using the QTextEdit's toHTML() method.

Printing

 Next, we'll create some actions for printing and preview-ing our document.

initToolbar():

self.printAction = QtGui.QAction(QtGui.QIcon("icons/print.png"),"Print document",self)
self.printAction.setStatusTip("Print document")
self.printAction.setShortcut("Ctrl+P")
self.printAction.triggered.connect(self.print)

self.previewAction = QtGui.QAction(QtGui.QIcon("icons/preview.png"),"Page view",self)
self.previewAction.setStatusTip("Preview page before printing")
self.previewAction.setShortcut("Ctrl+Shift+P")
self.previewAction.triggered.connect(self.preview)

Further below:

self.toolbar.addAction(self.printAction)
self.toolbar.addAction(self.previewAction)

self.toolbar.addSeparator()

initMenubar():

file.addAction(self.printAction)
file.addAction(self.previewAction)

Below the initUI() method:

def preview(self):

    # Open preview dialog
    preview = QtGui.QPrintPreviewDialog()

    # If a print is requested, open print dialog
    preview.paintRequested.connect(lambda p: self.text.print_(p))

    preview.exec_()

def print(self):

    # Open printing dialog
    dialog = QtGui.QPrintDialog()

    if dialog.exec_() == QtGui.QDialog.Accepted:
        self.text.document().print_(dialog.printer())

We create the actions following the same scheme as we did for the file management actions and add them to our toolbar as well as the "file" menu. The preview() method opens a QPrintPreviewDialog and optionally prints the document, if the user wishes to do so. The print() method opens a QPrintDialog and prints the document if the user accepts.

Copy and paste - undo and redo

 These actions will let us copy, cut and paste text as well as undo/redo actions:

initToolbar():

self.cutAction = QtGui.QAction(QtGui.QIcon("icons/cut.png"),"Cut to clipboard",self)
self.cutAction.setStatusTip("Delete and copy text to clipboard")
self.cutAction.setShortcut("Ctrl+X")
self.cutAction.triggered.connect(self.text.cut)

self.copyAction = QtGui.QAction(QtGui.QIcon("icons/copy.png"),"Copy to clipboard",self)
self.copyAction.setStatusTip("Copy text to clipboard")
self.copyAction.setShortcut("Ctrl+C")
self.copyAction.triggered.connect(self.text.copy)

self.pasteAction = QtGui.QAction(QtGui.QIcon("icons/paste.png"),"Paste from clipboard",self)
self.pasteAction.setStatusTip("Paste text from clipboard")
self.pasteAction.setShortcut("Ctrl+V")
self.pasteAction.triggered.connect(self.text.paste)

self.undoAction = QtGui.QAction(QtGui.QIcon("icons/undo.png"),"Undo last action",self)
self.undoAction.setStatusTip("Undo last action")
self.undoAction.setShortcut("Ctrl+Z")
self.undoAction.triggered.connect(self.text.undo)

self.redoAction = QtGui.QAction(QtGui.QIcon("icons/redo.png"),"Redo last undone thing",self)
self.redoAction.setStatusTip("Redo last undone thing")
self.redoAction.setShortcut("Ctrl+Y")
self.redoAction.triggered.connect(self.text.redo)

Further below:

self.toolbar.addAction(self.cutAction)
self.toolbar.addAction(self.copyAction)
self.toolbar.addAction(self.pasteAction)
self.toolbar.addAction(self.undoAction)
self.toolbar.addAction(self.redoAction)

self.toolbar.addSeparator()

initMenubar():

edit.addAction(self.undoAction)
edit.addAction(self.redoAction)
edit.addAction(self.cutAction)
edit.addAction(self.copyAction)
edit.addAction(self.pasteAction)

As you can see, we don't need any separate slot functions for these actions, as our QTextEdit object already has very handy methods for all of these actions. Note that in the initMenubar() method, we add these actions to the "Edit" menu and not the "File" menu.

Lists

 Finally, we'll add two actions for inserting lists. One for numbered lists and one for bulleted lists:

initToolbar():

bulletAction = QtGui.QAction(QtGui.QIcon("icons/bullet.png"),"Insert bullet List",self)
bulletAction.setStatusTip("Insert bullet list")
bulletAction.setShortcut("Ctrl+Shift+B")
bulletAction.triggered.connect(self.bulletList)

numberedAction = QtGui.QAction(QtGui.QIcon("icons/number.png"),"Insert numbered List",self)
numberedAction.setStatusTip("Insert numbered list")
numberedAction.setShortcut("Ctrl+Shift+L")
numberedAction.triggered.connect(self.numberList)

Further below:

self.toolbar.addAction(bulletAction)
self.toolbar.addAction(numberedAction)

Below the initUI() method:

    def bulletList(self):

        cursor = self.text.textCursor()

        # Insert bulleted list
        cursor.insertList(QtGui.QTextListFormat.ListDisc)

    def numberList(self):

        cursor = self.text.textCursor()

        # Insert list with numbers
        cursor.insertList(QtGui.QTextListFormat.ListDecimal)

As you can see, we don't make these actions class members because we don't need to access them anywhere else in our code, we only need to create and use them within the scope of initToolbar().
Concerning the slot functions, we retrieve our QTextEdit's QTextCursor, which has a lot of very useful methods, such insertList(), which, well, does what it's supposed to do. In case of bulletList(), we insert a list with the QTextListFormat set to ListDisc. For numberList(), we insert a list with ListDecimal format.

Final changes

 To finish off this part of Building a text editor with PyQt, let's make some final changes in the initUI() method:

self.text.setTabStopWidth(33)


Because PyQt's tab width is very strange, I recommend you set the QTextEdit's "tab stop width" to 33 pixels, which is around 8 spaces. 

self.setWindowIcon(QtGui.QIcon("icons/icon.png"))


Now that we have icons, we can add an icon for our window.

self.text.cursorPositionChanged.connect(self.cursorPosition


By connecting our QTextEdit's cursorPositionChanged signal to a function, we can display the cursor's current line and column number in the status bar. Here is the corresponding slot function, below initUI():

def cursorPosition(self):

    cursor = self.text.textCursor()

    # Mortals like 1-indexed things
    line = cursor.blockNumber() + 1
    col = cursor.columnNumber()

    self.statusbar.showMessage("Line: {} | Column: {}".format(line,col))

We first retrieve our QTextEdit's QTextCursor, then grab the cursor's column and block/line number and finally display these numbers in the status bar.

That'll be it for this part of the series. See you soon.

Originally published on BinPress.
Posted by at No comments :
Labels: , ,

Friday, February 7, 2014

Matrix rain in Python

Feeling Matrixy? How about making it digitally rain - in Python.



On the wikipedia article about the matrix digital rain, I found out what letters are typically used for that matrix effect, namely a form of Katakana (Japanese) letters. Moreover, it is useful to know the ord() and the chr() functions. The former transforms a character into its Unicode equivalent and chr() does the opposite: it transforms a number into a Unicode character.

The code is a piece of cake, for Python 3.x:


import time,random

items = [chr(i) for i in range(0x30a1, 0x30ff + 1)] # katakana


for i in range(1,11): # spaces and numbers
    items.append(str(i))
    items.append(" "*i)

def rain(row,column):         # rows,columns
     for i in range(row): #for every row
             s = ''      #new string
             for j in range(column): #for every column (or character)
                     ri = random.randrange(len(items)) #random index
                     s += items[ri]
             print(s)
             time.sleep(0.05) # change this to whatever makes the rain a good speed




For Python 2.x, do this for the unicode:


items = [unichr(i) for i in range(0x30a1,0x30ff + 1)]



First, import random and time. Then we need to add all the Unicode Katakana characters to the items list, the list where all possible items are stored. This is done via a list comprehension that goes through the range of the Katakana Unicode values. These values are in hexadecimal. You could also just do:


items = [chr(i) for i in range(12449,12544)]


I did it this way because those are the official Unicode values and the range of the Katakana characters is between 0x30a1 and 0x30ff. Next we put some numbers and spacing in.

The function takes the number or rows and the number of columns as a parameter. Then it loops through the ranges and fills a string with characters. This is done by getting a random index and adding the character at that position in items to the string. At the end of the row, we print out the string. The time.sleep() is there so you can control the speed of the rain.

Do not do this in IDLE! IDLE is just too slow, it won't look good. The best thing is to do it in your terminal. If your terminal let's you change the layout, try to get that black background/neon green text look.
Posted by at 6 comments :
Labels: ,
Subscribe to: Posts ( Atom )