English ▼
Skip to page content Skip to main navigation
e.g. "templates" or "datastore"

# Entities, Properties, and Keys

- Overview
- Kinds and Identifiers
- Ancestor Paths
- Transactions and Entity Groups
- Properties and Value Types
- Working with Entities
- Understanding Write Costs

## Overview

Objects in the App Engine Datastore are known as entities. An entity has one or more named properties, each of which can have one or more values. Entities of the same kind need not have the same properties, and an entity's values for a given property need not all be of the same data type. (If appropriate, an application can establish and enforce such restrictions in its own data model.)

The Datastore supports a variety of data types for property values. These include, among others:

- Integers
- Floating-point numbers
- Strings
- Dates
- Binary data

Each entity in the Datastore has a key that uniquely identifies it. The key consists of the following components:

- The kind of the entity, which categorizes it for the purpose of Datastore queries
- An identifier for the individual entity, which can be either
  - a key name string
  - an integer numeric ID
- An optional ancestor path locating the entity within the Datastore hierarchy

An application has access only to entities it has created itself; it can't access data belonging to other applications. It can fetch an individual entity from the Datastore using the entity's key, or it can retrieve one or more entities by issuing a query based on the entities' keys or property values.

The Python App Engine SDK includes a data modeling library for representing Datastore entities as instances of Python classes, and for storing and retrieving those instances in the Datastore. The Datastore itself does not enforce any restrictions on the structure of entities, such as whether a given property has a value of a particular type; this task is left to the application and the data modeling library.

## Kinds and Identifiers

Each Datastore entity is of a particular kind, which categorizes the entity for the purpose of queries: for instance, a human resources application might represent each employee at a company with an entity of kind `Employee`. In the Python Datastore API, an entity's kind is determined by its model class, which you define in your application as a subclass of the data modeling library class `db.Model`. The name of the model class becomes the kind of the entities belonging to it. The following example creates an entity of kind `Employee`, populates its property values, and saves it to the Datastore:

    import datetime
    from google.appengine.ext import db

    class Employee(db.Model):
      first_name = db.StringProperty()
      last_name = db.StringProperty()
      hire_date = db.DateProperty()
      attended_hr_training = db.BooleanProperty()

    employee = Employee(first_name='Antonio',
                        last_name='Salieri')

    employee.hire_date = datetime.datetime.now().date()
    employee.attended_hr_training = True

    employee.put()

The `Employee` class declares four properties for the data model: `first_name`, `last_name`, `hire_date`, and `attended_hr_training`. The `Model` superclass ensures that the attributes of `Employee` objects conform to this model: for example, an attempt to assign a string value to the `hire_date` attribute would result in a runtime error, since the data model for `hire_date` was declared as `db.DateProperty`.

In addition to a kind, each entity has an identifier, assigned when the entity is created. Because the identifier is part of the entity's key, it is associated permanently with the entity and cannot be changed. It can be assigned in either of two ways:

- Your application can specify its own identifier string for the entity (called the key name).
- You can have the Datastore automatically assign the entity an integer numeric ID.

Note: Instead of using key name strings or generating numeric IDs automatically, advanced applications may sometimes wish to assign their own numeric IDs manually to the entities they create. Be aware, however, that there is nothing to prevent the Datastore from assigning one of your manual numeric IDs to another entity. The only way to avoid such conflicts is to have your application obtain a block of IDs with the methods `allocate_ids()` or `allocate_ids_async()`. The Datastore's automatic ID generator will keep track of IDs that have been allocated with these methods and will avoid reusing them for another entity, so you can safely use such IDs without conflict.

To assign an entity a key name, provide the named argument `key_name` to the model class constructor when you create the entity:

    # Create an entity with the key Employee:'asalieri'.
    employee = Employee(key_name='asalieri')

To have the Datastore assign a numeric ID automatically, omit the `key_name` argument:

    # Create an entity with a key such as Employee:8261.
    employee = Employee()

Note: In addition to arguments such as `key_name`, the `Model` constructor accepts initial values for properties as keyword arguments. This makes it inconvenient to have a property named `key_name`. For more information, see Disallowed Property Names on the `Model` class page.

## Ancestor Paths

Entities in the Datastore form a hierarchically structured space similar to the directory structure of a file system. When you create an entity, you can optionally designate another entity as its parent; the new entity is a child of the parent entity. This association between an entity and its parent is permanent, and cannot be changed once the entity is created. An entity without a parent is a root entity. The Datastore will never assign the same numeric ID to two entities with the same parent, or to two root entities (those without a parent).

To designate an entity's parent, use the `parent` argument to the model class constructor when creating the child entity. The value of this argument can be the parent entity itself or its key; you can get the key by calling the parent entity's `key()` method. The following example creates an entity of kind `Address` and shows two ways of designating an `Employee` entity as its parent:

    # Create Employee entity
    employee = Employee()
    employee.put()

    # Set Employee as Address entity's parent directly...
    address = Address(parent=employee)

    # ...or using its key
    e_key = employee.key()
    address = Address(parent=e_key)

    # Save Address entity to datastore
    address.put()

An entity's parent, parent's parent, and so on recursively, are its ancestors; its children, children's children, and so on, are its descendants. The sequence of entities beginning with a root entity and proceeding from parent to child, leading to a given entity, constitute that entity's ancestor path. The complete key identifying the entity consists of a sequence of kind-identifier pairs specifying its ancestor path and terminating with those of the entity itself:

    Person:GreatGrandpa / Person:Grandpa / Person:Dad / Person:Me

For a root entity, the ancestor path is empty and the key consists solely of the entity's own kind and identifier:

    Person:GreatGrandpa

## Transactions and Entity Groups

Every attempt to create, update, or delete an entity takes place in the context of a transaction. A single transaction can include any number of such operations. To maintain the consistency of the data, the transaction ensures that all of the operations it contains are applied to the Datastore as a unit or, if any of the operations fails, that none of them are applied.

Note: If your application receives an exception when submitting a transaction, it does not necessarily mean that the transaction has failed. It is possible to receive a `Timeout`, `TransactionFailedError`, or `InternalError` exception even when a transaction has been committed and will eventually be applied successfully. Whenever possible, structure your Datastore transactions so that the end result will be unaffected if the same transaction is applied more than once.

A single transaction can apply to multiple entities, so long as the entities are descended from a common ancestor. Such entities are said to belong to the same entity group. In designing your data model, you should determine which entities you need to be able to process in the same transaction. Then, when you create those entities, place them in the same entity group by declaring them with a common ancestor. This tells App Engine that the entities will be updated together, so it can store them in a way that supports transactions.

## Properties and Value Types

The data values associated with an entity consist of one or more properties. Each property has a name and one or more values. A property can have values of more than one type, and two entities can have values of different types for the same property.

Tip: Properties with multiple values can be useful, for instance, when performing queries with equality filters: an entity satisfies the query if any of its values for a property matches the value specified in the filter. For more details on multiple-valued properties, including issues you should be aware of, see the Queries and Indexes page.

The following value types are supported:

Value type | Python type(s) | Sort order | Notes
--- | --- | --- | ---
Integer | `int` `long` | Numeric | 64-bit integer, signed
Floating-point number | `float` | Numeric | 64-bit double precision, IEEE 754
Boolean | `bool` | `False` < `True` |
Text string (short) | `str` `unicode` | Unicode (`str` treated as ASCII) | Up to 500 Unicode characters
Text string (long) | `db.Text` | None | Up to 1 megabyte Not indexed
Byte string (short) | `db.ByteString` | Byte order | Up to 500 bytes
Byte string (long) | `db.Blob` | None | Up to 1 megabyte Not indexed
Date and time | `datetime.date` `datetime.time` `datetime.datetime` | Chronological |
Geographical point | `db.GeoPt` | By latitude, then longitude |
Postal address | `db.PostalAddress` | Unicode |
Telephone number | `db.PhoneNumber` | Unicode |
Email address | `db.Email` | Unicode |
Google Accounts user | `users.User` | Email address in Unicode order |
Instant messaging handle | `db.IM` | Unicode |
Link | `db.Link` | Unicode |
Category | `db.Category` | Unicode |
Rating | `db.Rating` | Numeric |
Datastore key | `db.Key` | By path elements (kind, identifier, kind, identifier...) |
Blobstore key | `blobstore.BlobKey` | Byte order |
Null | `NoneType` | None |

For text strings and unencoded binary data (byte strings), the Datastore supports two value types:

- Short strings (up to 500 Unicode characters or bytes) are indexed and can be used in query filter conditions and sort orders.
- Long strings (up to 1 megabyte) are not indexed and cannot be used in query filters and sort orders.

Note: The long byte string type is named `Blob` in the Datastore API. This type is unrelated to blobs as used in the Blobstore API.

For values of mixed types, the Datastore uses a deterministic ordering based on the internal representations:

1. Null values
2. Fixed-point numbers
   - Integers
   - Dates and times
   - Ratings
3. Boolean values
4. Byte strings (short)
5. Unicode strings
   - Text strings (short)
   - Postal addresses
   - Telephone numbers
   - Email addresses
   - IM handles
   - Links
   - Categories
6. Floating-point numbers
7. Geographical points
8. Google Accounts users
9. Datastore keys
10. Blobstore keys

Note: Integers and floating-point numbers are considered separate types in the Datastore. If an entity uses a mix of integers and floats for the same property, all integers will be sorted before all floats: for example,
`7` < `3.2`
Because long text strings and long byte strings are not indexed, they have no ordering defined.

## Working with Entities

Applications can use the Datastore API to create, retrieve, update, and delete entities. If the application knows the complete key for an entity (or can derive it from its parent key, kind, and identifier), it can use the key to operate directly on the entity. An application can also obtain an entity's key as a result of a Datastore query; see the Queries and Indexes page for more information.

### Creating an Entity

In Python, you create a new entity by constructing an instance of a model class, populating its properties if necessary, and calling its `put()` method to save it to the Datastore. You can specify the entity's key name by passing a `key_name` argument to the constructor:

    employee = Employee(key_name='asalieri',
                        first_name='Antonio',
                        last_name='Salieri')

    employee.hire_date = datetime.datetime.now().date()
    employee.attended_hr_training = True

    employee.put()

If you don't provide a key name, the Datastore will automatically generate a numeric ID for the entity's key:

    employee = Employee(first_name='Antonio',
                        last_name='Salieri')

    employee.hire_date = datetime.datetime.now().date()
    employee.attended_hr_training = True

    employee.put()

### Retrieving an Entity

To retrieve an entity identified by a given key, pass the `Key` object as an argument to the `db.get()` function. You can generate the `Key` object using the class method `Key.from_path()`. The complete path is a sequence of entities in the ancestor path, with each entity represented by its kind (a string) followed by its identifier (key name or numeric ID):

    address_k = db.Key.from_path('Employee', 'asalieri', 'Address', 1)
    address = db.get(address_k)

`db.get()` returns an instance of the appropriate model class. Be sure that you have imported the model class for the entity being retrieved.

### Updating an Entity

To update an existing entity, modify the attributes of the object, then call its `put()` method. The object data overwrites the existing entity. The entire object is sent to the Datastore with every call to `put()`.

Note: The Datastore API does not distinguish between creating a new entity and updating an existing one. If the object's key represents an entity that already exists, the `put()` method overwrites the existing entity. You can use a transaction to test whether an entity with a given key exists before creating one. See also the `Model.get_or_insert()` method.

Tip: To delete a property, delete the attribute from the Python object, then save the object:
`del address.postal_code`

### Deleting an Entity

Given an entity's key, you can delete the entity with the `db.delete()` function

    address_k = db.Key.from_path('Employee', 'asalieri', 'Address', 1)
    db.delete(address_k)

or by calling the entity's own `delete()` method:

    employee_k = db.Key.from_path('Employee', 'asalieri')
    employee = db.get(employee_k)

    # ...

    employee.delete()

### Batch Operations

The `db.put()`, `db.get()`, and `db.delete()` functions (and their asynchronous counterparts `db.put_async()`, `db.get_async()`, and `db.delete_async()`) can accept a list argument to act on multiple entities in a single Datastore call:

    # A batch put.
    db.put([e1, e2, e3])

    # A batch get.
    entities = db.get([k1, k2, k3])

    # A batch delete.
    db.delete([k1, k2, k3])

Performing operations in batches does not affect the cost, regardless of the entity's size. A batch operation for two keys costs two reads, even if one of the keys did not exist. For example, it is more economical to do a keys-only query that retrieves 1000 keys, and then do a fetch on 500 of them, than to do a regular (not keys-only) query for all 1000 directly:

Note: A batch call to `db.put()` or `db.delete()` may succeed for some entities but not others. If it is important that the call succeed completely or fail completely, you must use a transaction, and all affected entities must be in the same entity group.

### Deleting Entities in Bulk via the Administration Console

You can use the Datastore Admin tab of the Administration Console to delete all entities of a given kind, or all entities of all kinds, in the default namespace. To enable this feature, include the builtin handler `datastore_admin` in your `app.yaml` file:

    builtins:
    - datastore_admin: on

This enables the Datastore Admin screen in the Data section of the Administration Console. From this screen, you can select the entity kind(s) to delete individually or in bulk, and delete them using the Delete Entities button. Note that bulk deletion takes place within your application, and thus counts against your quota.

Warning: This feature is currently experimental. We believe it is the fastest way to bulk-delete data, but it is not yet stable and you may encounter occasional bugs.

## Understanding Write Costs

When your application executes a Datastore `put()` operation, the Datastore must perform a number of writes to store the entity. Your application is charged for each of these writes. You can see how many writes will be required to store an entity by looking at the data viewer in the SDK Development Console. This section explains how App Engine calculates these values.

Every entity requires a minimum of two writes to store: one for the entity itself and another for the built-in `EntitiesByKind` index, which is used by the query planner to service a variety of queries. In addition, the Datastore maintains two other built-in indexes, `EntitiesByProperty` and `EntitiesByPropertyDesc`, which provide efficient scans of entities by single property values in ascending and descending order, respectively. Each of an entity's indexed property values must be written to each of these indexes.

As an example, consider an entity with properties A, B, and C:

    Key: 'Foo:1' (kind = 'Foo', id = 1, no parent)
    A: 1, 2
    B: null
    C: 'this', 'that', 'theOther'

Assuming there are no composite indexes (see below) for entities of this kind, this entity requires 14 writes to store:

- 1 for the entity itself
- 1 for the `EntitiesByKind` index
- 4 for property A (2 for each of two values)
- 2 for property B (a null value still needs to be written)
- 6 for property C (2 for each of three values)

Composite indexes (those referring to multiple properties) require additional writes to maintain. Suppose you define the following composite index:

    Kind: 'Foo'
    A ▲, B ▼

where the triangles indicate the sort order for the specified properties: ascending for property A and descending for property B. Storing the entity defined above now takes an additional write to the composite index for every combination of A and B values:

- (`1`, `null`)
- (`2`, `null`)

This adds 2 writes for the composite index, for a total of 1 + 1 + 4 + 2 + 6 + 2 = 16. Now add property C to the index:

    Kind: 'Foo'
    A ▲, B ▼, C ▼

Storing the same entity now requires a write to the composite index for each possible combination of A, B, and C values:

- (`1`, `null`, `'this'`)
- (`1`, `null`, `'that'`)
- (`1`, `null`, `'theOther'`)
- (`2`, `null`, `'this'`)
- (`2`, `null`, `'that'`)
- (`2`, `null`, `'theOther'`)

This brings the total number of writes to 1 + 1 + 4 + 2 + 6 + 6 = 20.

If a Datastore contains many multiple-valued properties, or if a single such property is referenced many times, the number of writes required to maintain the index can explode combinatorially. Such exploding indexes can be very expensive to maintain. For example, consider a composite index that includes ancestors:

    Kind: 'Foo'
    A ▲, B ▼, C ▼
    Ancestor: True

Storing a simple entity with this index present takes the same number of writes as before. However, if the entity has ancestors, it requires a write for each possible combination of property values and ancestors, in addition to those for the entity itself. Thus an entity defined as

    Key: 'GreatGrandpa:1/Grandpa:1/Dad:1/Foo:1' (kind = 'Foo', id = 1, parent = 'GreatGrandpa:1/Grandpa:1/Dad:1')
    A: 1, 2
    B: null
    C: 'this', 'that', 'theOther'

would require a write to the composite index for each of the following combinations of properties and ancestors:

- (`1`, `null`, `'this'`, `'GreatGrandpa'`)
- (`1`, `null`, `'this'`, `'Grandpa'`)
- (`1`, `null`, `'this'`, `'Dad'`)
- (`1`, `null`, `'this'`, `'Foo'`)
- (`1`, `null`, `'that'`, `'GreatGrandpa'`)
- (`1`, `null`, `'that'`, `'Grandpa'`)
- (`1`, `null`, `'that'`, `'Dad'`)
- (`1`, `null`, `'that'`, `'Foo'`)
- (`1`, `null`, `'theOther'`, `'GreatGrandpa'`)
- (`1`, `null`, `'theOther'`, `'Grandpa'`)
- (`1`, `null`, `'theOther'`, `'Dad'`)
- (`1`, `null`, `'theOther'`, `'Foo'`)
- (`2`, `null`, `'this'`, `'GreatGrandpa'`)
- (`2`, `null`, `'this'`, `'Grandpa'`)
- (`2`, `null`, `'this'`, `'Dad'`)
- (`2`, `null`, `'this'`, `'Foo'`)
- (`2`, `null`, `'that'`, `'GreatGrandpa'`)
- (`2`, `null`, `'that'`, `'Grandpa'`)
- (`2`, `null`, `'that'`, `'Dad'`)
- (`2`, `null`, `'that'`, `'Foo'`)
- (`2`, `null`, `'theOther'`, `'GreatGrandpa'`)
- (`2`, `null`, `'theOther'`, `'Grandpa'`)
- (`2`, `null`, `'theOther'`, `'Dad'`)
- (`2`, `null`, `'theOther'`, `'Foo'`)

Storing this entity in the Datastore now requires 1 + 1 + 4 + 2 + 6 + 24 = 38 writes.