Tokenizer - customizable tokenizer class for use with the intrinsic class 'grammar-production' parser.

This tokenizer implementation is parameterized with a set of rules (see below); a basic set of rules is provided, but users can customize the tokenizer quite extensively simply by subclassing the Tokenizer class and overriding the 'rules_' property with a new set of rules declarations.