English ▼
Skip to page content Skip to main navigation
e.g. "templates" or "datastore"

# Writing Property Subclasses

#### Experimental!

NDB is an experimental, innovative, and rapidly changing new feature for App Engine. Unfortunately, being on the bleeding edge means that we may make backwards-incompatible changes to NDB. We will inform the community when this feature is no longer experimental.

The `Property` class is designed to be subclassed. However, it is normally easier to subclass an existing `Property` subclass.

All special `Property` attributes, even those considered 'public', have names starting with an underscore. This is because `StructuredProperty` uses the non-underscore attribute namespace to refer to nested `Property` names; this is essential for specifying queries on subproperties.

The `Property` class and its predefined subclasses allow subclassing using composable (or stackable) validation and conversion APIs. These require some terminology definitions:

- A user value is a value such as would be set and accessed by the application code using standard attributes on the entity.
- A base value is a value such as would be serialized to and deserialized from the Datastore.

A `Property` subclass that implements a specific transformation between user values and serialiazble values should implement two methods, `_to_base_type()` and `_from_base_type()`. These should not call their `super()` method. This is what is meant by composable (or stackable) APIs.

The API supports stacking classes with ever more sophisticated user-base conversions: the user-to-base conversion goes from more sophisticated to less sophisticated, while the base-to-user conversion goes from less sophisticated to more sophisticated. For example, see the relationship between `BlobProperty`, `TextProperty`, and `StringProperty`. For example, `TextProperty` inherits from `BlobProperty`; its code is pretty simple because it inherits most of the behavior it needs.

In addition to `_to_base_type()` and `_from_base_type()`, the `_validate()` method is also a composable API.

The validation API distinguishes between lax and strict user values. The set of lax values is a superset of the set of strict values. The `_validate()` method takes a lax value and if necessary converts it to a strict value. This means that when setting the property value, lax values are accepted, while when getting the property value, only strict values will be returned. If no conversion is needed, `_validate()` may return None. If the argument is outside the set of accepted lax values, `_validate()` should raise an exception, preferably `TypeError` or `datastore_errors.BadValueError`.

The `_validate()`, `_to_base_type()`, and `_from_base_type()` do not need to handle:

- `None`: They will not be called with `None` (and if they return None, this means that the value does not need conversion).
- Repeated values: The infrastructure takes care of calling `_from_base_type()` or `_to_base_type()` for each list item in a repeated value.
- Distinguishing user values from base values: The infrastructure handles this by calling the composable APIs.
- Comparisons: The comparison operations call `_to_base_type()` on their operand.
- Distinguishing between user and base values: the infrastructure guarantees that `_from_base_type()` will be called with an (unwrapped) base value, and that `_to_base_type()` will be called with a user value.

For example, suppose you need to store really long integers. The standard `IntegerProperty` only supports (signed) 64-bit integers. Your property might store a longer integer as a string; it would be good to have the property class handle the conversion. An application using your property class might look something like

    # Imports.
    from google.appengine.ext import ndb
    from somewhere import LongIntegerProperty

    # Define an entity class holding some long integers.
    class MyModel(ndb.Model):
      name = ndb.StringProperty()
      abc = LongIntegerProperty(default=0)
      xyz = LongIntegerProperty(repeated=True)

    # Create an entity and write it to the Datastore.
    ent = MyModel(name='booh', xyz=[10**100, 6**666])
    assert ent.abc == 0
    key = ent.put()

    # Read an entity back from the Datastore and update it.
    ent = key.get()
    ent.abc += 1
    ent.xyz.append(ent.abc//3)
    ent.put()

    # Query for a MyModel entity whose xyz contains 6**666.
    # (NOTE: using ordering operations don't work, but == does.)
    results = MyModel.query(MyModel.xyz == 6**666).fetch(10)

This looks simple and straightforward. It also demonstrates the use of some standard property options (default, repeated); as the author of `LongIntegerProperty`, you will be glad to hear you don't have to write any "boilerplate" to get those working. It's easier to define a subclass of another property, for example:

    class LongIntegerProperty(ndb.StringProperty):
      def _validate(self, value):
        if not isinstance(value, (int, long)):
          raise TypeError('expected an integer, got %s' % repr(value))

      def _to_base_type(self, value):
        return str(value) # Doesn't matter if it's an int or a long

      def _from_base_type(self, value):
        return long(value)  # Always return a long

When you set a property value on an entity, e.g. `ent.abc = 42`, your `_validate()` method is called, and (if it doesn't raise an exception) the value is stored on the entity. When you write the entity to the Datastore, your `_to_base_type()` method is called, converting the value to the string. Then that value is serialized by the base class, `StringProperty`. The inverse chain of events happens when the entity is read back from the Datastore. The `StringProperty` and `Property` classes together take care of the other details, such as serializing the and deserializing the string, setting the default, and handling repeated property values.

In this example, supporting inequalities (i.e. queries using <, <=, >, >=) requires more work. The following example implementation imposes a maximum size of integer and stores values as fixed-length strings:

    class BoundedLongIntegerProperty(ndb.StringProperty):
      def __init__(self, bits, **kwds):
        assert isinstance(bits, int)
        assert bits > 0 and bits % 4 == 0 # Make it simple to use hex
        super(BoundedLongIntegerProperty, self).__init__(**kwds)
        self._bits = bits

      def _validate(self, value):
        assert -(2 ** (self.bits - 1)) <= value < 2 ** (self.bits - 1)

      def _to_base_type(self, value):
        # convert from signed -> unsigned
        if value < 0:
          value += 2 ** self._bits
        assert 0 <= value < 2 ** self.bits
        # Return number as a zero-padded hex string with correct number of digits:
        return '%0*x' % (self._bits // 4, value)

      def _from_base_type(self, value):
        value = int(value, 16)
        if value >= 2 ** (self._bits - 1):
          value -= 2 ** self._bits
        return value

This can be used in the same way as `LongIntegerProperty` except that you must pass the number of bits to the property constructor, e.g. `BoundedLongIntegerProperty(1024)`.

You can subclass other property types in similar ways.

This approach also works for storing structured data. Suppose you have a `FuzzyDate` Python class that represents a date range; it uses fields `first` and `last` to store the date range's beginning and end:

    from datetime import date

    class FuzzyDate(object):
      def __init__(self, first, last=None):
        assert isinstance(first, date)
        assert last is None or isinstance(last, date)
        self.first = first
        self.last = last or first

You can create a `FuzzyDateProperty` that derives from `StructuredProperty`. Unfortunately, the latter doesn't work with plain old Python classes; it needs a `Model` subclass. So define a Model subclass as an intermediate representation;

    class FuzzyDateModel(ndb.Model):
      first = ndb.DateProperty()
      last = ndb.DateProperty()

Next, construct a subclass of `StructuredProperty` that hardcodes the modelclass argument to be `FuzzyDateModel`, and defines `_to_base_type()` and `_from_base_type()` methods to convert between `FuzzyDate` and `FuzzyDateModel`:

    class FuzzyDateProperty(StructuredProperty):

      def __init__(self, **kwds):
        super(FuzzyDateProperty, self).__init__(FuzzyDateModel, **kwds)

      def _validate(self, value):
        assert isinstance(value, FuzzyDate)

      def _to_base_type(self, value):
        return FuzzyDateModel(first=value.first, last=value.last)

      def _from_base_type(self, value):
        return FuzzyDate(value.first, value.last)

An application might use this class like so:

    # Class to record historic people and events in their life.
    class HistoricPerson(ndb.Model):
      name = ndb.StringProperty()
      birth = FuzzyDateProperty()
      death = FuzzyDateProperty()
      # Parallel lists:
      event_dates = FuzzyDateProperty(repeated=True)
      event_names = ndb.StringProperty(repeated=True)

    columbus = HistoricPerson(
      name='Christopher Columbus',
      birth=FuzzyDate(date(1451, 8, 22), date(1451, 10, 31)),
      death=FuzzyDate(date(1506, 5, 20)),
      event_dates=[FuzzyDate(date(1492, 1, 1), date(1492, 12, 31))],
      event_names=['Discovery of America'])
    columbus.put()

    # Query for historic people born no later than 1451.
    results = HistoricPerson.query(HistoricPerson.birth.last <=
                                   date(1451, 12, 31)).fetch()

Suppose you want to accept plain `date` objects in addition to `FuzzyDate` objects as the values for `FuzzyDateProperty`. To do this, modify the `_validate()` method as follows:

    def _validate(self, value):
      if isinstance(value, date):
        value = FuzzyDate(value)
      assert isinstance(value, FuzzyDate)
      return value  # Must return the converted value!

You could instead subclass `FuzzyDateProperty` as follows (assuming `FuzzyDateProperty._validate()` is as shown above).

    class MaybeFuzzyDateProperty(FuzzyDateProperty):
      def _validate(self, value):
        if isinstance(value, date):
          return FuzzyDate(value)
        # Otherwise, return None and leave validation to the base class

When you assign a value to a `MaybeFuzzyDateProperty` field, both `MaybeFuzzyDateProperty._validate()` and `FuzzyDateProperty._validate()` are invoked, in that order. The same applies to `_to_base_type()` and `_from_base_type()`: the methods in in superclass and subclass are implicitly combined. (Don't use `super` to control inherited behavior for this. For these three methods, the interaction is subtle and `super` doesn't do what you want.)