English ▼
Skip to page content Skip to main navigation
e.g. "templates" or "datastore"

# Queries and Indexes

- Overview
- Restrictions on Queries
- Fetching Results
- Introduction to Indexes
- Big Entities and Exploding Indexes
- Queries on Keys
- Query Cursors
- Setting the Read Policy and Datastore Call Deadline

## Overview

A query retrieves entities from the datastore that meet a specified set of conditions. The query specifies an entity kind, zero or more filters based on entity property values, and zero or more sort order descriptions. When the query is executed, it retrieves all entities of the given kind that satisfy all of the given filters, sorted in the specified order.

The Master/Slave Datastore and the High Replication Datastore have different guarantees when it comes to query consistency. By default:

- The Master/Slave datastore is strongly consistent for all queries.
- The High Replication datastore is strongly consistent by default for queries within an entity group. With the High Replication Datastore, non-ancestor queries are always eventually consistent.

For more information, see Setting the Read Policy and Datastore Call Deadline.

The low-level Java datastore API provides a Query class for constructing queries and a PreparedQuery class for fetching and returning entities from the datastore:

    import com.google.appengine.api.datastore.DatastoreService;
    import com.google.appengine.api.datastore.DatastoreServiceFactory;
    import com.google.appengine.api.datastore.Entity;
    import com.google.appengine.api.datastore.PreparedQuery;
    import com.google.appengine.api.datastore.Query;

    // ...
    // Get the Datastore Service
    DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();

    // The Query interface assembles a query
    Query q = new Query("Person");
    q.addFilter("lastName", Query.FilterOperator.EQUAL, lastNameParam);
    q.addFilter("height", Query.FilterOperator.LESS_THAN, maxHeightParam);

    // PreparedQuery contains the methods for fetching query results
    // from the datastore
    PreparedQuery pq = datastore.prepare(q);

    for (Entity result : pq.asIterable()) {
      String firstName = (String) result.getProperty("firstName");
      String lastName = (String) result.getProperty("lastName");
      Long height = (Long) result.getProperty("height");
      System.out.println(lastName + " " + firstName + ", " + height.toString() + " inches tall");
    }

A filter includes a property name, a comparison operator, and a value. An entity satisfies the filter if it has a property of the given name and the property value compares to the value specified in the filter in the manner described by the comparison operator. The entity is a result for a query if it satisfies all of the query's filters.

The comparison operator can be any of the following (defined as enumerated constants in the nested class Query.FilterOperator):

Operator | Meaning
--- | ---
`EQUAL` | Equal to
`LESS_THAN` | Less than
`LESS_THAN_OR_EQUAL` | Less than or equal to
`GREATER_THAN` | Greater than
`GREATER_THAN_OR_EQUAL` | Greater than or equal to
`NOT_EQUAL` | Not equal to
`IN` | Member of (equal to any of the values in a specified list)

The `NOT_EQUAL` operator actually performs two queries: one in which all other filters are the same and the `NOT_EQUAL` filter is replaced with a `LESS_THAN` filter, and one where it is replaced with a `GREATER_THAN` filter. The results are merged, in order. As described below under Restrictions on Queries, a query can have no more than one `NOT_EQUAL` filter, and a query that has one cannot have any other inequality filters.

The `IN` operator also performs multiple queries, one for each item in the specified list, with all other filters the same and the `IN` filter replaced with an `EQUAL` filter. The results are merged, in the order of the items in the list. If a query has more than one `IN` filter, it is performed as multiple queries, one for each possible combination of values in the `IN` lists.

A single query containing `NOT_EQUAL` or `IN` operators is limited to no more than 30 sub-queries.

For more information about how `NOT_EQUAL` and `IN` queries translate to multiple queries in a JDO/JPA framework, see the article Queries with != and IN filters.

A query can also return just the keys of the result entities instead of the entities themselves:

    Query q = new Query("Person").setKeysOnly();

### Kindless Queries

A query with no kind and no ancestor or key filters returns every entity in the datastore for the application. This includes entities created and managed by other App Engine features, including statistics entities (which exist for all apps) and Blobstore metadata entities (if any). Kindless queries cannot include filters on properties. They can, however, filter by Entity Key by passing Entity.KEY_RESERVED_PROPERTY as the property name for the filter. Ascending sorts on Entity.KEY_RESERVED_PROPERTY are also supported.

### Ancestor Queries

You can filter your datastore queries to a specified ancestor, so that the result set only includes entities descended from the specified ancestor. In other words, all of the results will have the ancestor as their parent, or parent's parent, or etc.

Passing null as a parameter to Query.setAncestor(String ancestor) does not query for entities without ancestors (this type of query is not currently supported).

    DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
    Entity person = new Entity("Person", "tom");

    Entity weddingPhoto = new Entity("Photo", person.getKey());
    weddingPhoto.setProperty("imageUrl",
                             "http://domain.com/some/path/to/wedding_photo.jpg");

    Entity babyPhoto = new Entity("Photo", person.getKey());
    babyPhoto.setProperty("imageUrl",
                          "http://domain.com/some/path/to/baby_photo.jpg");

    Entity dancePhoto = new Entity("Photo", person.getKey());
    dancePhoto.setProperty("imageUrl",
                           "http://domain.com/some/path/to/dance_photo.jpg");

    Entity campingPhoto = new Entity("Photo");
    campingPhoto.setProperty("imageUrl",
                             "http://domain.com/some/path/to/camping_photo.jpg");

    datastore.put(Arrays.asList(person, weddingPhoto, babyPhoto, dancePhoto, campingPhoto));

    Query userPhotosQuery = new Query("Photo");
    userPhotosQuery.setAncestor(person.getKey());

    // This returns weddingPhoto, babyPhoto and dancePhoto, but
    // not campingPhoto because tom is not an ancestor.
    List<Entity> results = datastore.prepare(userPhotosQuery).asList(
        FetchOptions.Builder.withDefaults());

### Kindless Ancestor Queries

You can also perform queries for entities with a given ancestor regardless of kind. Such queries can also include filters, equality or inequality, on Entity.KEY_RESERVED_PROPERTY. Kindless queries cannot include filters on properties. They can, however, filter by Entity Key by passing Entity.KEY_RESERVED_PROPERTY as the property name for the filter. Ascending sorts on Entity.KEY_RESERVED_PROPERTY are also supported.

Notably, a kindless ancestor query without any additional filters always returns the entity uniquely identified by the ancestor (assuming that entity exists). The name "ancestor" implies that queries return only children of an entity; but in fact, ancestor queries can return the entity itself. The following code sample demonstrates this point.

Kindless ancestor queries do not require custom indexes.

To perform a kindless ancestor query using the `Query` class, call the constructor without a kind:

    Query q =  new Query();

    q.setAncestor(ancestorKey);
    q.addFilter(Entity.KEY_RESERVED_PROPERTY, Query.FilterOperator.GREATER_THAN, lastKeyParam)

The following code sample demonstrates how to iterate over all Entities belonging to a given User:

    DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
    Entity person = new Entity("Person", "tom");

    Entity weddingPhoto = new Entity("Photo", person.getKey());
    weddingPhoto.setProperty("imageUrl",
                             "http://domain.com/some/path/to/wedding_photo.jpg");

    Entity weddingVideo = new Entity("Video", person.getKey());
    weddingVideo.setProperty("videoUrl",
                             "http://domain.com/some/path/to/wedding.avi");

    datastore.put(Arrays.asList(person, weddingPhoto, weddingVideo));

    Query userMediaQuery = new Query();
    userMediaQuery.setAncestor(person.getKey());

    // Ancestor queries return ancestors by default. This filter
    // excludes the ancestor from query results.
    userMediaQuery.addFilter(Entity.KEY_RESERVED_PROPERTY,
                             Query.FilterOperator.GREATER_THAN,
                             person.getKey());

    // Returns both weddingPhoto and weddingVideo even though they are
    // different entity kinds.
    List<Entity> results = datastore.prepare(userMediaQuery).asList(
        FetchOptions.Builder.withDefaults());

## Restrictions on Queries

The nature of the index query mechanism imposes a few restrictions on what a query can do. These restrictions are described in this section.

### Filtering or Sorting on a Property Requires That the Property Exist

If a property has a query filter condition or sort order, the query returns only those datastore entities that have a value (including null) for that property.

Entities of a kind need not have the same properties. A filter on a property can only match an entity with a value for that property. If an entity has no value for a property used in a filter or sort order, that entity is omitted from the index built for the query.

### No Use of Filters That Match Entities Missing a Property

It is not possible to query for entities that are missing a given property. One alternative is to create a fixed (modeled) property with a default value of `null`, then create a filter for entities with `null` as the property value.

### Inequality Filters Are Allowed on One Property Only

A query may only use inequality filters (`<`, `<=`, `>=`, `>`, `!=`) on one property across all of its filters.

For example, this query is allowed:

    import com.google.appengine.api.datastore.Query;
    import com.google.appengine.api.datastore.Query.FilterOperator;
    import com.google.appengine.api.datastore.Query.SortDirection;

    Query q = new Query("Person");
    q.addFilter("birthYear", FilterOperator.GREATER_THAN_OR_EQUAL, minBirthYearParam);
    q.addFilter("birthYear", FilterOperator.LESS_THAN_OR_EQUAL, maxBirthYearParam);

However, the following query is not allowed, because it uses inequality filters on two different properties in the same query:

    Query q = new Query("Person");
    q.addFilter("birthYear", FilterOperator.GREATER_THAN_OR_EQUAL, minBirthYearParam);
    // Error
    q.addFilter("height", FilterOperator.GREATER_THAN_OR_EQUAL, minHeightParam);

Filters can combine equal (`==`) comparisons for different properties in the same query, including queries with one or more inequality conditions on a property. This is allowed:

    Query q = new Query("Person");
    q.addFilter("lastName", FilterOperator.EQUAL, lastNameParam);
    q.addFilter("city", FilterOperator.EQUAL, cityParam);
    q.addFilter("birthYear", FilterOperator.GREATER_THAN_OR_EQUAL, minBirthYearParam);

The query mechanism relies on all results for a query to be adjacent to one another in the index table, to avoid having to scan the entire table for results. A single index table cannot represent multiple inequality filters on multiple properties while maintaining that all results are consecutive in the table.

### Properties in Inequality Filters Must Be Sorted before Other Sort Orders

If a query has both a filter with an inequality comparison and one or more sort orders, the query must include a sort order for the property used in the inequality, and the sort order must appear before sort orders on other properties.

This query is not valid, because it uses an inequality filter and does not order by the filtered property:

    Query q = new Query("Person");
    q.addFilter("birthYear", FilterOperator.GREATER_THAN_OR_EQUAL, minBirthYearParam);
    q.addSort("lastName", SortDirection.ASCENDING); // ERROR

Similarly, the following query is not valid because it does not order by the filtered property before ordering by other properties:

    Query q = new Query("Person");
    q.addFilter("birthYear", FilterOperator.GREATER_THAN_OR_EQUAL, minBirthYearParam);
    q.addSort("lastName", SortDirection.ASCENDING); // ERROR
    q.addSort("birthYear", SortDirection.ASCENDING);

This query is valid:

    Query q = new Query("Person");
    q.addFilter("birthYear", FilterOperator.GREATER_THAN_OR_EQUAL, minBirthYearParam);
    q.addSort("birthYear", SortDirection.ASCENDING);
    q.addSort("lastName", SortDirection.ASCENDING);

To get all results that match an inequality filter, a query scans the index table for the first matching row, then returns all consecutive results until it finds a row that doesn't match. For the consecutive rows to represent the complete result set, the rows must be ordered by the inequality filter before other sort orders.

### Properties with Multiple Values Can Have Surprising Behaviors

Due to the way they're indexed, properties with multiple values interact with query filters and sort orders in specific and sometimes surprising ways.

First, if a query has multiple inequality filters on a given property, an entity matches the query only if it has an individual value for that property that matches all of the inequality filters. For example, if an entity has the values `[1, 2]` for property `x`, it will not match the query `WHERE x > 1 AND x < 2`. Each filter does match one of `x`'s values, but no single value matches both filters.

(Note that this does not apply to `=` filters. For example, the query `WHERE x = 1 AND x = 2` will return the above entity.)

The `!=` operator works as "value is other than" test. So, for example, the filter `x != 1` matches an entity with value greater than or less than 1. In Java only, you can use a filter like `x != 1 AND x != 2`, which acts like "a < 1 OR (a > 1 and a < 2) OR a > 2", so `[1, 2]` matches, but `[1, 2, 3]` does not.

Similarly, the sort order for multiply valued properties is unusual:

- If the entities are sorted by a multi-valued property in ascending order, the smallest value is used for ordering.
- If the entities are sorted by a multi-valued property in descending order, the greatest value is used for ordering.
- Other values do not affect the sort order, nor does the number of values.

This ordering takes place because multivalued properties appear in an index once for each unique value they contain, but the datastore removes duplicate values from the index, so the first value seen in the index determines its sort order.

This sort order has the unusual consequence that `[1, 9]` comes before `[4, 5, 6, 7]` in both ascending and descending order.

### Filtering on Unindexed Properties Returns No Results

Queries can't find property values that aren't indexed. (You also cannot sort on unindexed properties.) For a detailed discussion, see Unindexed Properties.

### Sort Orders Are Ignored on Properties with Equality Filters

One important caveat is queries with both an equality filter and a sort order on a multi-valued property. In those queries, the sort order is disregarded. For single-valued properties, this is a simple optimization. Every result would have the same value for the property, so the results do not need to be sorted further.

However, multi-valued properties may have additional values. Since the sort order is disregarded, the query results may be returned in a different order than if the sort order were applied. (Restoring the dropped sort order would be expensive and require extra indices, and this use case is rare, so the query planner leaves it off.)

### Query Ordering When Sort Order Is Unspecified

When a query does not specify an order, the datastore returns the results in the order of retrieval. As we make changes to the datastore implementation, the order of results in an unordered query may change as well. Therefore, if you require a specific sort ordering for your query results, be sure to specify it in the query. If you don't specify a sort order, the ordering of results is undefined and may change over time.

### Queries Inside Transactions Must Include Ancestor Filters

Queries are only supported inside transactions if they include an ancestor filter. The query's ancestor must be in the same entity group as the other operations in the transaction. This preserves the restriction that a transaction can only operate on entities in a single entity group.

## Fetching Results

After constructing your query, you can specify a number of fetch options to control which results get returned for your query.

If you want to return only a single entity matching your query, you can use the `PreparedQuery.asSingleEntity()` method. This will return the first result that matches the query. If there is more than one matching result, the system throws a TooManyResultsException exception.

Key-only queries return just the keys to the entities that match your query from the datastore. Key-only queries run faster than queries that return complete entities. To return only the keys, use the `Query.setKeysOnly()` method.

You can specify a limit and offset with your query to control the number and range of results returned in one batch. Specifying an integer limit returns up to that number of results that match the query. Using an integer offset will skip that number of results and return the rest, up to the limit specified. For example, to fetch the tallest five people from your datastore, you would construct your query as:

    // Construct then prepare your query
    List<Entity> get5TallestPeople() {
      DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
      Query q = new Query("Person");
      q.addSort("height", SortDirection.DESCENDING);
      PreparedQuery pq = ds.prepare(q);
      return pq.asList(FetchOptions.Builder.withLimit(5));
    }

If you wanted the 6th through 10th tallest people, you would instead use:

    return pq.asList(FetchOptions.Builder.withLimit(5).offset(5)));

When iterating through the results of your query using `PreparedQuery.asIterable()` and `PreparedQuery.asIterator()`, the datastore fetches the results in batches. By default, each batch contains 20 results, but you can change this value using FetchOptions.chunkSize(int). You can continue iterating through query results until all are returned or the request times out.

## Introduction to Indexes

Note: For an in-depth discussion of indexes and querying, see the article Index Selection and Advanced Search.

Every datastore query uses an index, a table containing the results for the query in the desired order. An App Engine application defines its indexes in a configuration file named `datastore-indexes.xml`. The development web server automatically adds suggestions to this file as it encounters queries that do not yet have indexes configured. You can tune indexes manually by editing the file before uploading the application.

Note: You can get the list of indexes for your application at runtime by calling the `getIndexes` method of class DataStoreService or AsyncDataStoreService .

The index-based query mechanism supports most common kinds of queries, but it does not support some queries common in other database technologies. Restrictions on queries, and their explanations, are described below.

The datastore maintains an index for every query an application intends to make. As the application makes changes to datastore entities, the datastore updates the indexes with the correct results. When the application executes a query, the datastore fetches the results directly from the corresponding index.

The datastore executes a query using the following steps:

1. The datastore identifies the index that corresponds with the query's kind, filter properties, filter operators, and sort orders.
2. The datastore starts scanning the index at the first entity that meets all of the filter conditions using the query's filter values.
3. The datastore continues to scan the index, returning each entity, until it finds the next entity that does not meet the filter conditions, reaches the end of the index, or has collected the maximum number of results requested by the query.

An index table contains columns for every property used in a filter or sort order. The rows are sorted by the following aspects, in order:

- ancestors
- property values used in equality filters
- property values used in inequality filters
- property values used in sort orders

This puts all results for every possible query that uses this index in consecutive rows in the table.

An index contains an entity only if the index refers to every property in the entity. If the index does not reference a property of the entity, that entity will not appear in the index, and will never be a result for the query that uses the index.

Note that the App Engine datastore makes a distinction between an entity that does not possess a property and an entity that possesses a property with a value of `null`. If you want every entity of a kind to be a potential result for a query, you can use a JDO or JPA data class, which always assigns a value to every property that corresponds to a field in the class.

### Unindexed Properties

In some cases, you will never have to filter or sort on a particular property. When this happens, you can make the property an unindexed property. This type of property includes those that are explicitly marked as unindexed, as well as those with values of the long text value type (Text ) or the long binary value type (Blob ). Using unindexed properties allows you to reduce the cost of your application by decreasing the number of datastore writes needed, because the datastore will not need to maintain index entries for the property.

A query with a filter or sort order on a property will never match an entity with that property unindexed. Properties with such values behave as if the property is not set with regard to query filters and sort orders.

To specify that a property not be indexed for the low level API, set the property value using Entity.setUnindexedProperty(), as shown here:

    DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
    Key acmeKey = KeyFactory.createKey("Company", "acme");
    Entity tom = new Entity("Person", "tom", acme);
    tom.setProperty("age", 32);

    Entity lucy = new Entity("Person", "lucy", acmeKey);
    lucy.setUnindexedProperty("age", 29);

    datastore.put(Arrays.asList(tom, lucy));

    Query query = new Query("Person");
    query.setAncestor(acmeKey);
    query.addFilter("age", FilterOperator.GREATER_THAN, 25);

    // Returns tom but not lucy because her age is unindexed
    List<Entity> results = datastore.prepare(query).asList(FetchOptions.Builder.withDefaults());

For JDO or JPA, you must change the class definition for the property, adding the line `@Extension(vendorName="datanucleus", key="gae.unindexed", value="true")` above the property:

    JPA:
    @Entity
    public class Book {
      @Id
      @GeneratedValue(strategy=GenerationType.IDENTITY)
      private Long id;

      // The @Extension line below causes coverImageUrl to be unindexed. Remove this line to make the property indexed.
      @Extension(vendorName="datanucleus", key="gae.unindexed", value="true")
      private Link coverImageUrl;

      private String title;

      // getters and setters
    }

#### Indexing Formerly Unindexed Properties

For JDO or JPA, if you have existing records created with an unindexed property, that property continues to be unindexed for those records even after you change the entity (class) definition to make the property indexed again. Consequently, those records will not be returned by queries filtering on that property. To change this, you must rewrite those records, that is, do a fetch then a put on each one. The act of rewriting creates new index rows for those formerly unindexed properties. You can use task queues and query cursors to simplify this work.

Note: For the datastore low-level API, you don't change any entity definition as you do with JDO/JPA. Instead, you specify whether a property is indexed or not on a per-record basis by the method you use when setting the property value. Unindexed properties are specified when you set the property value using Entity.setUnindexedProperty(). To make that property value indexed again, you must reset the property value using Entity.setProperty().

To make a formerly unindexed property be indexed for JDO and JPA,

1. Remove the @Extension tag above the unindexed property that you want to be indexed. The tag line you must remove is `@Extension(vendorName="datanucleus", key="gae.unindexed", value="true")`, as shown below:

       JPA:
       @Entity
       public class Book {
         @Id
         @GeneratedValue(strategy=GenerationType.IDENTITY)
         private Long id;

         // The @Extension line below causes coverImageUrl to be unindexed. Remove this line to make the property indexed.
         @Extension(vendorName="datanucleus", key="gae.unindexed", value="true")
         private Link coverImageUrl;

         private String title;

         // getters and setters
       }

2. After you change the entity definition to make the property indexed, you need to process all of the existing records with the unindexed property as follows:
   1. Fetch a record.
   2. Using a PrePut callback, change the timestamp. (If you don't actually change something in the record, it won't get re-written and the required new property indexes won't be created.)
   3. Put the record.

#### Unindexing Formerly Indexed Properties

If you change a property from indexed to unindexed, the index records for all existing entities with that property will continue to exist until you update (or delete) the entity associated with those index records. Any new entities are created without the index records for the now unindexed property. So you need to purge your code of all queries that filter or sort by the property that is now unindexed.

### Mixed Types

When two entities have properties of the same name but of different value types, an index of the property sorts the entities first by value type, then by an order appropriate to the type. For example, if two entities each have a property named "age," one with an integer value and one with a string value, the entity with the integer value always appears before the entity with the string value when sorted by the "Age" property, regardless of the values themselves.

This is especially worth noting in the case of integers and floating point numbers, which are treated as separate types by the datastore. A property with the integer value `38` is sorted before a property with the floating point value `37.5`, because all integers are sorted before floats.

### Which Queries Need Indexes

App Engine builds indexes for several simple queries by default. For other queries, the application must specify the indexes it needs in a configuration file named `datastore-indexes.xml`. If the application running under App Engine tries to perform a query for which there is no corresponding index (either provided by default or described in `datastore-indexes.xml`), the query will fail with a DatastoreNeedIndexException.

App Engine suggests automatic indexes for the following forms of queries:

- queries using only equality and ancestor filters
- queries using only inequality filters (which can only be on a single property)
- kindless queries using only ancestor/key filters
- queries with no filters and only one sort order on a property, either ascending or descending
- queries using only equality filters on properties, and inequality filters on keys and ancestor filters

Other forms of queries require their indexes to be specified in `datastore-indexes.xml`, including:

- queries with multiple sort orders
- queries with a sort order on keys in descending order
- queries with one or more inequality filters on a property and one or more equality filters over other properties
- queries with inequality filters and ancestor filters

Note: The App Engine SDK suggests indexes that are appropriate for most applications. Depending on your application's use of the datastore and the size and shape of your application's data, manual adjustments to your indexes may be warranted. For example, writing entities with multiple property values may result in an exploding index with high write costs. For more information, see Big Entities and Exploding Indexes.

The development web server makes managing index configuration easy: Instead of failing to execute a query that does not have an index and requires it, the development web server can generate configuration for an index that would allow the query to succeed. If your local testing of an application calls every possible query the application will make, using every combination of filter and sort order, the generated entries will represent a complete set of indexes. If your testing does not exercise every possible query form, you can review and adjust the index configuration before uploading the application.

## Big Entities and Exploding Indexes

Note: One way to avoid big entities and exploding indexes is described in detail in the article Index Selection and Advanced Search.

As described above, every property (that doesn't have a `Text` or `Blob` value) of every entity is added to at least one index table, including a simple index provided by default, and any indexes described in the application's `datastore-indexes.xml` file that refer to the property. For an entity that has one value for each property, App Engine stores a property value once in its simple index, and once for each time the property is referred to in a custom index. Each of these index entries must be updated every time the value of the property changes.

The datastore limits the number of index entries that a single entity can have. The index entry limit is large, and most applications are not affected. However, there are some circumstances where you might encounter the limit. For example, an entity with a large number of single-value properties can exceed the index entry limit.

Properties with multiple values store each value as a separate entry in an index. An entity with a single property with a large number of values can exceed the index entry limit.

Custom indexes that refer to multiple properties with multiple values can get very large with only a few values. To completely record such properties, the index table must include a row for every permutation of the values of every property for the index.

For example, consider the following query:

    SELECT * FROM MyModel WHERE x=1 AND y=2 ORDER BY date

This query causes the SDK to suggest the following index:

    <?xml version="1.0" encoding="utf-8"?>
    <datastore-indexes>
      <datastore-index kind="MyModel">
        <property name="x" direction="asc" />
        <property name="y" direction="asc" />
        <property name="date" direction="asc" />
      </datastore-index>
    </datastore-indexes>

The following code creates an entity with four values for the property `x`, three values for the property `y`, and sets the `date` to the current date:

    Entity myModel = new Entity("Model");
    myModel.setProperty("x", Arrays.asList("one", "two", "three", "four"));
    myModel.setProperty("y", Arrays.asList("red", "green", "blue"));
    myModel.setProperty("date", new Date());
    ds.put(myModel);

To accurately represent multiple properties with multiple values, the index must store a value for each permutation of values contained in the properties `x`, `y`, and `date`, as modeled by the following equation:

    index_value_count = |x| * |y| * |date|

These indexes are called "exploding indexes" because they can become very large with just a few values. Exploding indexes cause entity writes to cost more because you're writing more data. They can also very easily cause an entity to exceed the per-entity index value limit.

For example, the code snippet above generates the following values for `x`, `y`, and `date`:

    x = [1, 2, 3, 4]
    y = ['red', 'green', 'blue']
    date = <now>

This index stores the following permutations:

`x` value | `y` value | `date` value
--- | --- | ---
`1` | `'blue'` | `<now>`
`1` | `'green'` | `<now>`
`1` | `'red'` | `<now>`
`2` | ``'blue'`` | `<now>`
`2` | `'green'` | `<now>`
`2` | `'red'` | `<now>`
`3` | ``'blue'`` | `<now>`
`3` | `'green'` | `<now>`
`3` | `'red'` | `<now>`
`4` | ``'blue'`` | `<now>`
`4` | `'green'` | `<now>`
`4` | `'red'` | `<now>`
Total: 12 values saved to the datastore.

You can avoid exploding indexes by defining these indexes instead in `datastore-indexes.xml`:

    <?xml version="1.0" encoding="utf-8"?>
    <datastore-indexes>
      <datastore-index kind="MyModel">
        <property name="x" direction="asc" />
        <property name="date" direction="asc" />
      </datastore-index>
      <datastore-index kind="MyModel">
        <property name="y" direction="asc" />
        <property name="date" direction="asc" />
      </datastore-index>
    </datastore-indexes>

With the index above, the number of values stored for the same query is additive:

    index_value_count = |x| * |date| + |y| * |date|

`x` value | `date` value | `y` value | `date` value
--- | --- | --- | ---
`1` | `<now>` | `'red'` | `<now>`
`2` | `<now>` | `'green'` | `<now>`
`3` | `<now>` | `'blue'` | `<now>`
`4` | `<now>` |  |
Total: 7 values saved to the datastore.

The App Engine SDK can detect exploding indexes only when the same property is repeated multiple times. In this case, the SDK suggests an alternative index. However, in all other circumstances (such as indexes generated for queries like the one in this example), the SDK generates an exploding index. In this circumstance, you can manually configure an index in `datastore-indexes.xml` to circumvent the exploding index.

If a `put()` would result in a number of index entries that exceeds the limit, the call fails with an exception. If you create a new index that would contain a number of index entries that exceeds the limit for any entity when built, queries against the index fail, and the index appears in the "Error" state in the Admin Console.

You can avoid exploding indexes by avoiding queries that would require a custom index using a list property. As described above, this includes queries with multiple sort orders or queries with a mix of equality and inequality filters.

## Queries on Keys

Keys are ordered first by parent path, then by kind, then by key name or ID. Kinds and key names are strings and are ordered by byte value. IDs are integers and are ordered numerically. If entities of the same parent and kind use a mix of key name strings and numeric IDs, entities with numeric IDs are considered to be less than entities with key name strings. Elements of the parent path are compared similarly: by kind (string), then by key name (string), or ID (number).

Queries involving keys use indexes just like queries involving properties. Queries on keys require custom indexes in the same cases as with properties, with a couple of exceptions: inequality filters or an ascending sort order on `Entity.KEY_RESERVED_PROPERTY` do not require a custom index, but a descending sort order on `Entity.KEY_RESERVED_PROPERTY` does. As with all queries, the development server creates appropriate configuration entries in this file when a query that needs a custom index is tested.

## Query Cursors

Query cursors allow an app to perform a query and retrieve a batch of results, then fetch additional results for the same query in a subsequent web request without the overhead of a query offset. After the app fetches some results for a query, it can ask for an encoded string that represents the location in the result set after the last result fetched (the "cursor"). The app can use the cursor to fetch additional results starting from that point at a later time.

A cursor is an opaque base64-encoded string that represents the next starting position of a query after a fetch operation. The app can embed the cursor in web pages as HTTP GET or POST parameters, or it can store the cursor in the datastore or memcache, or in a task queue task payload. A future request handler can perform the same query and include the cursor with the query to tell the datastore to start returning results from the location represented by the cursor. A cursor can only be used by the app that performed the original query, and can only be used to continue the same query.

Tip: It is generally safe to pass a datastore cursor to a client, such as in a web form, and accept a cursor value from a client. A client cannot change the cursor value to access results outside of the original query. However, the base64-encoded value can be decoded to expose information about result entities, such as the key (app ID, kind, key name or ID, and all ancestor keys) and properties used in the query (including filters and sort orders). If you don't want users to have access to that information, you can encrypt the cursor, or store it and provide the user with an opaque key.

### Cursors and Data Updates

The cursor's position is defined as the location in the result list after the last result returned. A cursor is not a relative position in the list (it's not an offset); it's a marker to which the datastore can jump when starting an index scan for results. If the results for a query change between uses of a cursor, the query notices only changes that occur in results after the cursor. If a new result appears before the cursor's position for the query, it will not be returned when the results after the cursor are fetched. Similarly, if an entity is no longer a result for a query but had appeared before the cursor, the results that appear after the cursor do not change. If the last result returned is removed from the result set, the cursor still knows how to locate the next result.

An interesting application of cursors is to monitor entities for unseen changes. If the app sets a timestamp property with the current date and time every time an entity changes, the app can use a query sorted by the timestamp property, ascending, with a datastore cursor to check when entities are moved to the end of the result list. If an entity's timestamp is updated, the query with the cursor returns the updated entity. If no entities were updated since the last time the query was performed, no results are returned, and the cursor does not move.

When retrieving query results, you can use both a start cursor and an end cursor to return a continuous group of results from the datastore. When using a start and end cursor to retrieve the results, you are not guaranteed that the size of the results will be the same as when you generated the cursors. Entities may be added or deleted from the datastore between the time the cursors are generated and when they are used in a query.

In the low-level Java API, the app can use cursors via the QueryResultList, QueryResultIterable, and QueryResultIterator interfaces, which are returned by methods of a PreparedQuery (`asQueryResultList()`, `asQueryResultIterable()`, and `asQueryResultIterator()`, respectively). These result objects provide a `getCursor()` method, which returns Cursor object. The app can get a web-safe string by calling the `Cursor` object's `toWebSafeString()` method, and can create a `Cursor` object from a web-safe string using the `Cursor.fromWebSafeString()` static method. The following example demonstrates the use of cursors for pagination.

    import java.io.IOException;

    import javax.servlet.ServletException;
    import javax.servlet.http.HttpServlet;
    import javax.servlet.http.HttpServletRequest;
    import javax.servlet.http.HttpServletResponse;

    import com.google.appengine.api.datastore.Cursor;
    import com.google.appengine.api.datastore.DatastoreService;
    import com.google.appengine.api.datastore.DatastoreServiceFactory;
    import com.google.appengine.api.datastore.Entity;
    import com.google.appengine.api.datastore.FetchOptions;
    import com.google.appengine.api.datastore.PreparedQuery;
    import com.google.appengine.api.datastore.Query;
    import com.google.appengine.api.datastore.QueryResultList;

    public class ListPeopleServlet extends HttpServlet {

      @Override
      protected void doGet(HttpServletRequest req, HttpServletResponse resp)
                       throws ServletException, IOException {

        DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
        Query q = new Query("Person");
        PreparedQuery pq = datastore.prepare(q);
        int pageSize = 15;

        resp.setContentType("text/html");
        resp.getWriter().println("<ul>");

        FetchOptions fetchOptions = FetchOptions.Builder.withLimit(pageSize);
        String startCursor = req.getParameter("cursor");

        // If this servlet is passed a cursor parameter, let's use it
        if (startCursor != null) {
          fetchOptions.startCursor(Cursor.fromWebSafeString(startCursor));
        }

        QueryResultList<Entity> results = pq.asQueryResultList(fetchOptions);
        for (Entity entity : results) {
          resp.getWriter().println("<li>" + entity.getProperty("name") + "</li>");
        }
        resp.getWriter().println("</ul>");

        String cursor = results.getCursor().toWebSafeString();

        // Assuming this servlet lives at '/people'
        resp.getWriter().println(
          "<a href='/people?cursor=" + cursor + "'>Next page</a>");
      }
    }

### Limitations of Cursors

A few things to know about cursors:

- You cannot use cursors with queries that use the `IN` or `!=` filter operators.
- To use a cursor, the app must perform the same query that provided the cursor, including the same kind, filters and filter values, ancestor filter, and sort orders. There is no way to fetch results using a cursor without setting up the same query.
- A query with a cursor does not always work as expected when the query uses an inequality filter or a sort order on a property with multiple values. Logic that de-duplicates the multivalued property for the query does not persist between queries, and may return a result more than once.
- Cursors depend on the index configuration that supports the query. In cases where multiple index configurations can support the same query, changing the index configuration for the query will invalidate cursors for that query. (For example, a query of only equality filters and no sort orders can be supported by the built-in indexes in most cases, and can also use a custom index if one is provided.) Cursors also depend on implementation details that may change with a new App Engine release that invalidates them. If an app attempts to use a cursor that is no longer valid, the datastore raises an exception. In Java, it raises an `IllegalArgumentException` (low-level API), a `JDOFatalUserException` (JDO), or a `PersistenceException` (JPA).

## Setting the Read Policy and Datastore Call Deadline

In order to increase data availability, you can set the datastore read policy so that all reads and queries are eventually consistent. While the API also allows you to explicitly set a strong consistency policy, in the High Replication Datastore non-ancestor queries are always eventually consistent. Setting the read policy to strong consistency for a non-ancestor query in the High Replication Datastore will have no effect.

When you select eventual consistency for a datastore query, the indexes used by the query to gather results are also accessed with eventual consistency. Eventual consistency queries occasionally return entities that don't match the query criteria, while strong consistency queries are always transactionally consistent. (You can use transactions to ensure a consistent result set if the query uses an ancestor filter.) See Transaction Isolation in App Engine for more information on how entities and indexes are updated.

Specifying a DatastoreServiceConfig allows you to choose between two options for read consistency. The choices are `"EVENTUAL"` (for reads with eventual consistency) and `"STRONG"` (for reads with strong consistency). The default is `"STRONG"`. Choosing strong consistency selects for freshness of data, but will timeout more frequently than eventually consistent reads. Eventually consistent reads timeout less frequently, but occasionally return stale results. You can construct a DatastoreServiceConfig using the DatastoreServiceConfig.Builder class. The following code snippet demonstrates how to set the read policy, how to set the call deadline, and how to set both:

    double deadline = 5.0;

    // Set the read policy with eventual consistency.
    ReadPolicy policy = new ReadPolicy(ReadPolicy.Consistency.EVENTUAL);

    DatastoreServiceConfig eventuallyConsistentConfig = DatastoreServiceConfig.Builder.withReadPolicy(policy);

    // Set the call deadline.
    DatastoreServiceConfig deadlineConfig = DatastoreServiceConfig.Builder.withDeadline(5);

    // Set both the read policy and the call deadline.
    DatastoreServiceConfig datastoreConfig = DatastoreServiceConfig.Builder.withDeadline(deadline).readPolicy(policy);

    DatastoreService datastore = DatastoreServiceFactory.getDatastoreService(datastoreConfig);

The default datastore call deadline is 30 seconds. It is not currently possible to set the deadline to be higher, but you can adjust it downward if you want to ensure a particular operation fails fast.