Backbone and Handlebars : “Backbone is not defined”

It’s ages since I’ve used Backbone but I’ve needed to in the past few days. I also needed to use Handlebars templates with it.

I’ve spent far too long today trying to understand why when I tried to do that I kept getting an error message “Backbone is not defined”.

You see my previous ‘old-style’ use of Backbone was a plain old ‘HTML with script tags’ page which did it’s templates the Underscore way and  which included the relevant scripts something like this :

<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/underscore.js/1.8.3/underscore-min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/backbone.js/1.3.3/backbone-min.js"></script>

So when I wanted to switch from Underscore templates to Handlebar templates I just dropped the reference to Underscore and include a reference to Handlebars like this

<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/handlebars.js/4.0.6/handlebars.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/backbone.js/1.3.3/backbone-min.js"></script>

And that’s when I got the error message “Backbone is not defined”  … for a painfully long time.

Well long story short it seems that in order to use Handlebars in that way you have to include Underscore as well like this :

<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/handlebars.js/4.0.6/handlebars.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/underscore.js/1.8.3/underscore-min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/backbone.js/1.3.3/backbone-min.js"></script>

I still have yet to see it written down that you need to do this but it works ! I was put on the right path by this gist https://gist.github.com/kyleondata/3440492 .

Of course if I’d done something contemporary like use browserify in the first place I suspect I wouldn’t have has this problem but if you’re doing it old school then this is what I suggest you do.

Hope it helps.

ASP.NET ModelState – keys are changed behind your back

Summary

Today I came across a strange behaviour within the ModelState property of the APIController .

In summary I discovered that the keys used by the developer get changed by ASP.Net … but only sometimes !

Background

To paraphrase the doco the ModelState dictionary is used to “represent the state of an attempt to bind a posted form to an action method, which includes validation information”.

One straightforward place this is seen is where an API request has provided a value which can’t be used to populate the relevant property of the underlying model. The resultant error is reflected in the contents of the ModelState property.

As well as this type of ‘automatic’ population of ModelState it’s possible use the AddModelError method to add your own errors to the dictionary like this :

ModelState.AddModelError("inputSample.StartDate", "Some sort of validation error");

Weirdness (or should I say ‘WeIrdNeSs’ ?)

The behaviour I saw today concerns the way the key used as the first argument to AddModelError gets manipulated in what I consider to be a rather confusing manner.

In the example above the underlying model had a property of ‘StartDate’ and so everything worked as you might have expected.

If however you use a key which is unrelated to any property of the model, eg ‘Foo’, you can still add the item to ModelState and it will exist in the underlying dictionary under the key used, like this :

ModelState.AddModelError("inputSample.Foo", "Some sort of validation error");

Where things get strange, at least in my view, is if you use a string which happens to match a property name if both your proposed key and the a Model property match when both are lower cased. For instance ‘HairColour’ might be a Model property name and a key might be used of ‘HaIrCoLoUr’ .

In that situation something in the ASP.NET plumbing decides that you didn’t really mean to use a key of ‘HaIrCoLoUr’ and instead meant to use ‘HairColour’.

I don’t like this behaviour because I really don’t like ‘magic’ going on in the background – others may feel differently.

Using Python’s argparse for a “turn on”/”turn off” argument

What’s argparse ?

Argparse is a Python standard module and “makes it easy to write user-friendly command-line interfaces”. The 2.x doco is here, the 3.x doco is here. Before 2.7 there was the optparse module supplied as part of Python but that’s been deprecated and replaced with argparse.

“turn off” / “turn off” type arguments

I was working on some code yesterday and I wanted an argument of the “turn on” / “turn off” type. So for instance you might want the output to be verbose or not, it’s not uncommon to see this implemented by means of a

--verbose

argument. When ‘–verbose’ is present the programmer provides verbose output, when it’s absent they don’t.

How then ?

A nice neat way to do this is to make use of the `action` (2.x and 3.x) argument of the `add_argument` method and to combine that with use of the `set_defaults` method so that a value is set in the case when the argument is not used by the user.

Here’s an example taken from my django-row-count project :

parser.add_argument('--echotostdout', dest='echotostdout', action='store_true')
parser.set_defaults(echotostdout=False)
args = parser.parse_args()

In this case a command line argument …

--echotostdout

… sets an attribute

args.echotostdout

… to True if it’s present as an argument on the command line and to False if it’s absent.

Django and Heroku – getting it working

Django and Heroku – getting it working

What follows is based on a short talk I gave to the New Zealand Python User Group in Feb 2015. This blog post provides some specifics on areas I was only able to hand wave over during the talk.

Motivation

I recently tried to deploy a Django side project to Heroku.

I’d previously used Heroku for a Ruby on Rails project and remembered it being very straightforward so I was surprised to find it wasn’t that great an experience. The documentation is fragmentary and seems to have been only partially updated to reflect changes in Django and the Heroku environment.

“Simplest Possible”

In the end I decided to suspend my original project and try to make the simplest possible Django project work on Heroku. For “simplest possible” I chose the “Polls” project from the Django Tutorial . I got it working and the code is available in my github account:  https://github.com/shearichard/polls17/tree/v2.0 . If you’re interested the version of the Project which works locally and before I made any changes to support use on Heroku is here : https://github.com/shearichard/polls17/tree/v1.0 .

What needed to be done

To complement the Heroku documentation I’m going to record here the changes that were made to the Project between v1.0 (working locally) and v2.0 (working on Heroku).

The files to which changes were applied to support use in Heroku are as follows :

mysite/mysite/settings.py (before and after)
mysite/mysite/settings_heroku.py (after – there was no ‘before’ for this file !)
mysite/mysite/wsgi.py (before and after)
requirements.txt (before and after)

settings.py

diff --git a/mysite/mysite/settings.py b/mysite/mysite/settings.py
index cb992c1..b2082ba 100644
--- a/mysite/mysite/settings.py
+++ b/mysite/mysite/settings.py
@@ -87,4 +87,5 @@ USE_TZ = True
# https://docs.djangoproject.com/en/1.7/howto/static-files/

STATIC_URL = '/static/'
STATIC_ROOT = 'staticfiles'
TEMPLATE_DIRS = [os.path.join(BASE_DIR, 'templates')]

 

settings_heroku.py

The settings_heroku.py file was completely new for use within the Heroku environment and we can see it referenced below from within wsgi.py when a test is made to see if the code is running within Heroku.

The final form of settings_heroku.py is as follows :

from .settings import *

import dj_database_url
DATABASES['default'] =  dj_database_url.config()

BASE_DIR = os.path.dirname(os.path.abspath(__file__))
STATIC_ROOT = 'staticfiles'
STATIC_URL = '/static/'

STATICFILES_DIRS = (
    os.path.join(BASE_DIR, 'static'),
)
# Simplified static file serving.
# https://warehouse.python.org/project/whitenoise/
STATICFILES_STORAGE = 'whitenoise.django.GzipManifestStaticFilesStorage'

Things worthy of note here are :

  • we import the whole of the local settings file (referenced here as ‘.settings’) and then change or add to it as necessary.
  • we make use of the dj-database-url to pick up the database configuration to be used in the Heroku environment
  • `STATIC_ROOT` and `STATICFILES_DIRS` are not needed in the standard version of the ‘Polls’ project but they are needed when we move to Heroku so they’re added here.
  • `STATIC_URL` is already defined in the standard settings file and so doesn’t actually need to be in settings_heroku.py at all.
  • STATICFILES_STORAGE allow for the use of Whitenoise a module which allows wsgi apps (such as this one) to serve their own static files, something which hadn’t previously been possible. There’s other good reasons to use Whitenoise in the areas of file compression and cache-header handling

wsgi.py

The version of wsgi.py before the changes for Heroku is very straightforward and can be seen below.

"""
WSGI config for mysite project.

It exposes the WSGI callable as a module-level variable named ``application``.

For more information on this file, see
https://docs.djangoproject.com/en/1.7/howto/deployment/wsgi/
"""

import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.settings")

from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()

To make wsgi.py work for Heroku there are essentially three changes:

  • Make the settings file used dependent on the existence of an environmental variable, ‘DYNO’. If it’s present then the code is running on Heroku and the server is started  using the the settings_heroku.py file shown above, otherwise we continue to use the settings.py file.
  • To make use of Whitenoise we take the the output of `get_wsgi_application` and use it as an argument when instantiating a `DjangoWhiteNoise` object.
  • Lastly, and least important, we redirect standard output so to standard error. This isn’t necessary at all and is something I did to make for easier diagnosis of issues while getting the Heroku specific version working.
diff --git a/mysite/mysite/wsgi.py b/mysite/mysite/wsgi.py
index 15c7d49..e5e1e5c 100644
--- a/mysite/mysite/wsgi.py
+++ b/mysite/mysite/wsgi.py
@@ -8,7 +8,20 @@ https://docs.djangoproject.com/en/1.7/howto/deployment/wsgi/
"""

import os
import sys

#Allows us to see useful stuff in Gunicorn output
sys.stdout = sys.stderr

#Rely upon env var 'DYNO` to determine if we are
#running within Heroku
if 'DYNO' in os.environ:
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.settings_heroku")
else:
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.settings")

from django.core.wsgi import get_wsgi_application
from whitenoise.django import DjangoWhiteNoise

application = get_wsgi_application()
application = DjangoWhiteNoise(application)

 

requirements.txt

The requirements.txt (created as the output from a `pip freeze` command) reflects the libraries installed at any given point.

Here’s the diff of requirements.txt between the local installation and the ‘Heroku’ ready installation.

As can be seen the extra libraries required by the migration to Heroku were :

  • dj-database-url
  • gunicorn
  • whitenoise
diff --git a/requirements.txt b/requirements.txt
index 98b2fd1..4e189d2 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,6 +1,7 @@
Django==1.7.4
Pygments==2.0.2
argparse==1.2.1
dj-database-url==0.3.0
django-extensions==1.5.0
django-pdb==0.4.1
fancycompleter==0.4
gunicorn==19.2.1
@@ -11,5 +12,6 @@ psycopg2==2.6
pyflakes==0.8.1
pyrepl==0.8.4
six==1.9.0
whitenoise==1.0.6
wmctrl==0.1
wsgiref==0.1.2

A general point about Project structure

A good deal of the Heroku documention assumes that your project directory (the one that contains manage.py) is also your root directory . This isn’t how I do things. I prefer my root directory to contain stuff like .gitignore, requirements.txt, README.md etc and to have a directory within the root which is my project directory.

If your project is similarly structured it’s worth bearing in mind that the Procfile required by Heroku should include ” –pythonpath ./mysite” (where ‘mysite’ is the name of your project directory) as an argument to the gunicorn invocation … I had a number of issues before I did this . Here’s an example of the argument in use.

A general point about the Heroku CLI

The Heroku Toolbelt includes the Heroku CLI which allows you to manage Heroku apps from the command line. For instance this :

heroku ps --app foo

Provides a list of running dynos in your ‘foo’ application.

Anyway the strange thing is that it seems to me that almost every command you issue via the Heroku CLI requires the

--app foo

argument, where ‘foo’ is the name of your application, and yet the documentation never mentions that ! You work it out pretty quickly because you don’t do much without without it but it’s strange all the same.

 In conclusion

Using the free levels of Heroku for running a Django project gives you access to a really high quality hosting environment at a very attractive price (free as long as you don’t get too much traffic or data). Once you’ve got over the bumps it works really well and for many people will be a good solution for hobby projects.

Openshift client tools install instructions are wrong for Ubuntu 14.x

Openshift client tools install instructions are wrong for Ubuntu 14.x

Summary

This is a short post to provide a correction to the Openshift getting started instructions for Ubuntu for those using Ubuntu 14.x

Caveat

What follows concerns Ruby and what I know about Ruby could be written on the back of an envelope.

Background

I have an Ubuntu 14.x headless box and I want to make use of the Openshift client tools from it. The Ubuntu machine had never had Ruby installed on it previously and the Openshirt client tools makes use of Ruby so I had to install that.

What do they say ?

Under the heading “Setting up the OpenShift Environment on Ubuntu” you’re told to install Ruby from scratch like this :

$ sudo apt-get install ruby-full rubygems git-core

When I tried that I got :

$ sudo gem install rhc
[sudo] password for rshea:
ERROR:  While executing gem ... (Zlib::DataError)
incorrect header check

So what do you do ?

With help from this Stackoverflow question I adapted the initial command to read

$ sudo apt-get install ruby-full rubygems-integration git-core

Then a bit more

The documentation does tell you to do the following to update your Ruby stack:

$ sudo gem install rubygems-update
$ sudo update_rubygems

It’s a bit odd the way this is tucked away at the bottom as it seems to have been necessary since Ubuntu 11.10. After that I was able to run

$ sudo gem install rhc

followed by

$ rhc setup

After that everything worked as it should.

Dominate: Manipulate HTML DOM using Python

Dominate: Manipulate HTML DOM using Python

This is a talk I gave in March 2014 which I never got around to doing a blog post for.

Dominate is : “a Python library for creating and manipulating HTML documents using an elegant DOM API” .

There’s a part of me which deep down feels that using templates is “wrong” and that procedural processing is the way to go … it might be a deluded part of me but it is a part of me ! Anyway as a result when I read about Dominate I had to give it a spin.

I gave a talk to the Wellington branch of the New Zealand Python User group and the slides are here : https://s3.amazonaws.com/shearichard/dominate-demo-NZPUG-2014-March.pdf . They are really very simple examples but, I hope, instructive.

As part of that I put my examples up on Bitbucket and they’re available here : https://bitbucket.org/rshea/nzpug201403/src

 

Charting with Django : three approaches

Charting with Django : three approaches

This is a belated (and hasty) post about a talk I gave in October 2013 at the Wellington branch of the New Zealand Python User Group.

Comparing three different charting libraries

In the talk I compared three different approaches to providing charts within a Django project.

The three different approaches used were :

Chartit
Django-Graphos
Chartkick

Sample code and slides

I built a Django project and an application for each of the three approaches and that code is available here : https://bitbucket.org/rshea/django-charts-demo .

The slides for my talk are available here as a PDF : https://s3.amazonaws.com/shearichard/django-with-charts.pdf

Conclusion in brief

If you’re only interested in my conclusion I would suggest Django-Graphos – read the slides for why.

Python you want a string you get a tuple – howzat ?

Python you want a string you get a tuple – howzat ?

Summary

How come you’re getting a tuple when you passed a string ?

Don’t do this at home

This is something that happened to me today. It really perplexed me so maybe this post will help someone else.

My class

I’d got a class a bit like the one below:

class cat(object):
    def __init__(self, name, colour, weight):
        self.name = name
        self.colour = colour,
        self.weight = weight
    def report(self):
        print self.name
        print self.colour
        print self.weight

Using it

But when I tried to use it like this:

mycat = cat('Garfield', 'Marmalade', 10)
mycat.report()

the output looked like this :

Garfield
('Marmalade',)
10

The problem being the attribute `colour` was being stored as a tuple.

The Answer

Looking back on it the problem is quite obvious but I was so busy looking at other parts of the situation (which was significantly more complex than the my cat example I missed it for quite a while.

Within the __init__ method I had inadvertently appended a comma onto the end of the self.colour assignment and Python takes that to mean, in our example, colour is the first element of a tuple.

repoze.catalog and ZODB beginners example – part 2

repoze.catalog and ZODB beginners example – part 2

Summary

The second of two posts which illustrate how to use repoze.catalog alongside ZODB. The first post can be seen at: “repoze.catalog and ZODB beginners example – part 1” .

Where we’re up to

In the first post I explained how you can have objects stored within a ZODB database indexed by repoze.catalog and why that was sometimes a good idea. In this post I’m going to demonstrate searching for the previously stored objects using repoze.catalog’s search facilities. If you haven’t read the first post I suggest you read that now because what follows assumes you have.

Finding ZODB objects with repoze.catalog

As discussed in the first post repoze.catalog allows you to index arbitrary properties of the objects you save into a ZODB database and then do complex searches on those properties to extract only the objects you’re interested in.

The example I’m showing here demonstrates how we can search through those objects we added in the example of the last post using a number of criteria.

Example Code

Here’s my example code and underneath I’ll expand a little more on what each part does:

'''
Demonstrates how to use repoze.catalog to find objects
being stored in ZODB. This example has the catalog and ZODB
within the same repository
'''
from myzodb import MyZODB
from persistent import Persistent

from repoze.catalog.catalog import FileStorageCatalogFactory
from repoze.catalog.catalog import ConnectionManager
from repoze.catalog.query import InRange, Lt

class City(Persistent):
    '''Represents a City by name and population'''
    def __init__(self, cityname, citypop):
        self.name = cityname
        self.population = citypop
    def __str__(self):
        return "%s  (Pop: %s)" % \
                (self.name, \
                str(self.population))

def print_all_city_instances(myzodbinst):
    '''
    Pull everything keyed under 'cities' out of the
    ZODB instance (without any regard to the
    repoze.catalog cataloguing and print them
    '''
    print ""
    print "About to dump all City Instances:"
    for acity in myzodbinst.dbroot['cities'].itervalues():
        print acity
    print ""

def print_city_query_results(myzodbinst, res):
    '''
    Use the list of integers returned by a
    repoze.catalog query to pull elements
    keyed underneath 'cities' in the ZODB
    instance which we are using repoze.catalog
    to catalogue
    '''
    print ""
    print "Objects stored in ZODB corresponding"
    print "to the repoze.catalog resultset:"
    for idx in res:
        print myzodbinst.dbroot['cities'][idx]
    print ""

if __name__ == '__main__':
    #Setup access to the repoze.catalog instance
    factory = FileStorageCatalogFactory('../data/mdcatalog.db',
                                        'mycatalog')
    manager = ConnectionManager()
    catalog = factory(manager)
    #Setup access to the ZODB instance containing data
    #catalogued by the repoze.catalog instance
    myzodbinstance = MyZODB('../data/mdzdb.fs')
    #Demonstrate we really have all the Cities
    print_all_city_instances(myzodbinstance)
    #Demonstrate use of `Lt` on the `population` index
    print ""
    print "*" * 60
    print "Looking for 'less than' value on the `population` index"
    print "Populations less than 1,000,000"
    numdocs, results = catalog.query(Lt('populations', 1000000))
    print "Raw Result: "
    print (numdocs, [ x for x in results ])
    print_city_query_results(myzodbinstance, results)

    #Demonstrate use of `InRange` on the `population` index
    print ""
    print "*" * 60
    print "Looking for 'InRange' values on the `population` index"
    print "Populations between 1,000,000 and 4,000,000"
    numdocs, results = catalog.query(InRange('populations',
                                              1000000, 4000000))
    print "Raw Result: "
    print (numdocs, [ x for x in results ])
    print_city_query_results(myzodbinstance, results)

Example Step by Step

Here’s a breakdown on what’s happening in the above example

Initialize repoze.catalog

factory = FileStorageCatalogFactory('../data/mdcatalog.db', 'mycatalog')
manager = ConnectionManager()
catalog = factory(manager)

Here we connect to our repoze.catalog repository and instantiate a `catalog` object

Make our ZODB database ready for use

myzodbinstance = MyZODB('../data/mdzdb.fs')

`MyZODB` is a convenience class which wraps up the instantiation of a ZODB database instance and provides : `storage`; `db`;`connection`; and `dbroot` properties to help the programmer interact with the ZODB database, connection, storage objects. `MyZODB` also provides a close method to cleanly close the ZODB database, connection and storage.

`MyZODB` is not explicitly included in the above example but it looks like this :

from ZODB import FileStorage, DB
class MyZODB(object):
    '''Manage the state of a ZODB FileStorage connection'''
    def __init__(self, path):
        self.storage = FileStorage.FileStorage(path)
        self.db = DB(self.storage)
        self.connection = self.db.open()
        self.dbroot = self.connection.root()
    def close(self):
        self.connection.close()
        self.db.close()
        self.storage.close()</pre>

Dump contents of ZODB without using repoze.catalog

The first data access we do in the above example is just a simple dump of every object, held under the key ‘cities’, in our ZODB database. Notice we are not using repoze.catalog at all at this point. By viewing this data we can be sure that the subsequent queries using repoze.catalog do what we think they do.

So we call the function `print_all_city_instances`

print_all_city_instances(myzodbinstance)

which iterates over the ‘cities’ element of the `dbroot` property of the ZODB `connection` to allow us to see everything that’s in the ZODB database.

for acity in myzodbinst.dbroot['cities'].itervalues():
    print acity

Our output looks like this :

About to dump all City Instances:
Windhoek  (Pop: 322500)
Pretoria  (Pop: 525387)
Nairobi  (Pop: 3138295)
Maputo  (Pop: 1244227)
Jakarta  (Pop: 10187595)
Canberra  (Pop: 358222)
Wellington  (Pop: 393400)
Santiago  (Pop: 5428590)
Buenos Aires  (Pop: 2891082)

Demonstrating the `Lt` function of repoze.catalog

The next thing that happens in the sample is to make use of the `Lt` function offered by repoze.catalog

numdocs, results = catalog.query(Lt('populations', 1000000))

In the previous post when we initialized our repoze.catalog we created a `populations` index which was associated with the `population` property of our `City` class (take a look at the previous post if you’ve forgotten the details).

Our use of the `Lt` method asks repoze.catalog to find all `City` instances stored in our ZODB database with a population of less than 1,000,000. As you can see we get two objects returned which I’ve named `numdocs` and `results`.

`numdocs` is an integer showing how many instances have been found which meet the criteria.

`results` is a list of integers which are keys used when storing into ZODB those objects which satisfy the search criteria.

We then use our function

print_city_query_results(myzodbinstance, results)

to output the objects found. The resulting output looks like this :

Objects stored in ZODB corresponding
to the repoze.catalog resultset:
Windhoek  (Pop: 322500)
Pretoria  (Pop: 525387)
Canberra  (Pop: 358222)
Wellington  (Pop: 393400)

It’s worth mentioning that whilst there are many comporator methods offered by repoze.catalog.query not all of them are applicable to all index types. In this example of the `Lt` method we are searching on an index, ‘populations’ of type CatalogTextIndex which does offer the `Lt` method but not all do.

Demonstrating the `InRange` function of repoze.catalog

Finally in the sample we show off the `InRange` function offered by repoze.catalog

 numdocs, results = catalog.query(InRange('populations',
                                          1000000, 4000000))

As with the previous example we utilise the previously created catalog index ‘populations’ to find instances of `City` – in this case those instances that have their `population` property set to a value between 1,000,000 and 4,000,000.

We do this by using the  `InRange` method offered by repoze.catalog. As with the `Lt` example above we get two objects returned which I’ve named `numdocs` and `results`.

`numdocs` is an integer showing how many instances have been found which meet the criteria.

`results` is a list of integers which are keys used when storing into ZODB those objects which satisfy the search critiera.

We then use our function

print_city_query_results(myzodbinstance, results)

to output the objects found. The resulting output looks like this :

Objects stored in ZODB corresponding
to the repoze.catalog resultset:
Nairobi  (Pop: 3138295)
Maputo  (Pop: 1244227)
Buenos Aires  (Pop: 2891082)

Credit where credits due

As with part one of this two part post the example I’ve shown here owes some parts to one of the examples on the repoze.catalog website and the structure of the `myZODB` was taken from the article cited above,  ‘Example Driven ZODB‘ .