English ▼
Skip to page content Skip to main navigation
e.g. "templates" or "datastore"

# The datastore package

## Introduction

Package datastore provides a client for App Engine's datastore service.

Basic Operations

Entities are the unit of storage and are associated with a key. A key consists of an optional parent key, a string application ID, a string kind (also known as an entity type), and either a StringID or an IntID. A StringID is also known as an entity name or key name.

It is valid to create a key with a zero StringID and a zero IntID; this is called an incomplete key, and does not refer to any saved entity. Putting an entity into the datastore under an incomplete key will cause a unique key to be generated for that entity, with a non-zero IntID.

An entity's contents are a mapping from case-sensitive field names to values. Valid value types are:

    - signed integers (int, int8, int16, int32 and int64),
    - bool,
    - string,
    - float32 and float64,
    - any type whose underlying type is one of the above predeclared types,
    - *Key,
    - Time,
    - appengine.BlobKey,
    - []byte (up to 1 megabyte in length),
    - slices of any of the above.

The Get and Put functions load and save an entity's contents. An entity's contents are typically represented by a struct pointer.

Example code:

    type Entity struct {
        Value string
    }

    func handle(w http.ResponseWriter, r *http.Request) {
        c := appengine.NewContext(r)

        k := datastore.NewKey(c, "Entity", "stringID", 0, nil)
        e := new(Entity)
        if err := datastore.Get(c, k, e); err != nil {
        	serveError(c, w, err)
        	return
        }

        old := e.Value
        e.Value = r.URL.Path

        if _, err := datastore.Put(c, k, e); err != nil {
        	serveError(c, w, err)
        	return
        }

        w.Header().Set("Content-Type", "text/plain; charset=utf-8")
        fmt.Fprintf(w, "old=%q\nnew=%q\n", old, e.Value)
    }

GetMulti, PutMulti and DeleteMulti are batch versions of the Get, Put and Delete functions. They take a []*Key instead of a *Key, and may return an ErrMulti when encountering partial failure.

Properties

An entity's contents can be represented by a variety of types. These are typically struct pointers, but can also be any type that implements the PropertyLoadSaver interface, or a Map (although Maps are deprecated). If using a struct pointer, you do not have to explicitly implement the PropertyLoadSaver interface; the datastore will automatically convert via reflection. Struct pointers are more strongly typed and are easier to use; PropertyLoadSavers and Maps are more flexible.

The actual types passed do not have to match between Get and Put calls or even across different App Engine requests. It is valid to put a *PropertyList and get that same entity as a *myStruct, or put a *myStruct0 and get a *myStruct1. Conceptually, any entity is saved as a sequence of properties, and is loaded into the destination value on a property-by-property basis. When loading into a struct pointer, an entity that cannot be completely represented (such as a missing field) will result in an error but it is up to the caller whether this error is fatal, recoverable or ignorable.

By default, for struct pointers, all properties are potentially indexed, and the property name is the same as the field name (and hence must start with an upper case letter). Fields may have a `datastore:"name,options"` tag. The tag name is the property name, which may start with a lower case letter. An empty tag name means to just use the field name. A "-" tag name means that the datastore will ignore that field. If options is "noindex" then the field will not be indexed. If the options is "" then the comma may be omitted. There are no other recognized options.

Example code:

    // A and B are renamed to a and b.
    // A, C and J are not indexed.
    // D's tag is equivalent to having no tag at all (E).
    // I is ignored entirely by the datastore.
    // J has tag information for both the datastore and json packages.
    type TaggedStructExample struct {
        A int `datastore:"a,noindex"`
        B int `datastore:"b"`
        C int `datastore:",noindex"`
        D int `datastore:""`
        E int
        I int `datastore:"-"`
        J int `datastore:",noindex" json:"j"`
    }

An entity's contents can also be represented by any type that implements the PropertyLoadSaver interface. This type may be a struct pointer, but it does not have to be. The datastore package will call LoadProperties when getting the entity's contents, and SaveProperties when putting the entity's contents. Possible uses include deriving non-stored fields, verifying fields, or indexing a field only if its value is positive.

Example code:

    type CustomPropsExample struct {
        I, J int
        // Sum is not stored, but should always be equal to I + J.
        Sum int `datastore:"-"`
    }

    func (x *CustomPropsExample) Load(c <-chan Property) os.Error {
        // Load I and J as usual.
        if err := datastore.LoadStruct(x, c); err != nil {
        	return err
        }
        // Derive the Sum field.
        x.Sum = x.I + x.J
        return nil
    }

    func (x *CustomPropsExample) Save(c chan<- Property) os.Error {
        // Validate the Sum field.
        if x.Sum != x.I + x.J {
        	return os.NewError("CustomPropsExample has inconsistent sum")
        }
        // Save I and J as usual. The code below is equivalent to calling
        // "return datastore.SaveStruct(x, c)", but is done manually for
        // demonstration purposes.
        c <- datastore.Property{
        	Name:  "I",
        	Value: int64(x.I),
        }
        c <- datastore.Property{
        	Name:  "J",
        	Value: int64(x.J),
        }
        close(c)
        return nil
    }

The *PropertyList type implements PropertyLoadSaver, and can therefore hold an arbitrary entity's contents.

Queries

A query is created using datastore.NewQuery and is configured by calling its methods. Running a query yields an iterator of results: either an iterator of keys or of (key, entity) pairs. Once initialized, a query can be re-used, and it is safe to call Query.Run from concurrent goroutines.

Example code:

    type Widget struct {
        Description string
        Price       int
    }

    func handle(w http.ResponseWriter, r *http.Request) {
        c := appengine.NewContext(r)
        q := datastore.NewQuery("Widget").
        	Filter("Price <", 1000).
        	Order("-Price")
        b := bytes.NewBuffer(nil)
        for t := q.Run(c); ; {
        	var x Widget
        	key, err := t.Next(&x)
        	if err == datastore.Done {
        		break
        	}
        	if err != nil {
        		serveError(c, w, err)
        		return
        	}
        	fmt.Fprintf(b, "Key=%v\nWidget=%#v\n\n", x, key)
        }
        w.Header().Set("Content-Type", "text/plain; charset=utf-8")
        io.Copy(w, b)
    }

Transactions

RunInTransaction runs a function in a transaction.

Example code:

    type Counter struct {
        Count int
    }

    func inc(c appengine.Context, key *datastore.Key) (int, os.Error) {
        var x Counter
        if err := datastore.Get(c, key, &x); err != nil && err != datastore.ErrNoSuchEntity {
        	return 0, err
        }
        x.Count++
        if _, err := datastore.Put(c, key, &x); err != nil {
        	return 0, err
        }
        return x.Count, nil
    }

    func handle(w http.ResponseWriter, r *http.Request) {
        c := appengine.NewContext(r)
        var count int
        err := datastore.RunInTransaction(c, func(c appengine.Context) os.Error {
        	var err1 os.Error
        	count, err1 = inc(c, datastore.NewKey(c, "Counter", "singleton", 0, nil))
        	return err1
        }, nil)
        if err != nil {
        	serveError(c, w, err)
        	return
        }
        w.Header().Set("Content-Type", "text/plain; charset=utf-8")
        fmt.Fprintf(w, "Count=%d", count)
    }

## Variables

    var (
        // ErrInvalidEntityType is returned when an invalid destination entity type
        // is passed to Get, GetAll, GetMulti or Next.
        ErrInvalidEntityType = os.NewError("datastore: invalid entity type")
        // ErrInvalidKey is returned when an invalid key is presented.
        ErrInvalidKey = os.NewError("datastore: invalid key")
        // ErrNoSuchEntity is returned when no entity was found for a given key.
        ErrNoSuchEntity = os.NewError("datastore: no such entity")
    )

Done is returned when a query iteration has completed.

    var Done = os.NewError("datastore: query has no more results")

ErrConcurrentTransaction is returned when a transaction is rolled back due to a conflict with a concurrent transaction.

    var ErrConcurrentTransaction = os.NewError("datastore: concurrent transaction")

## Functions

func Delete

  `func Delete(c appengine.Context, key *Key) os.Error`

  Delete deletes the entity for the given key.

func DeleteMulti

  `func DeleteMulti(c appengine.Context, key []*Key) os.Error`

  DeleteMulti is a batch version of Delete.

func Get

  `func Get(c appengine.Context, key *Key, dst interface{}) os.Error`

  Get loads the entity stored for k into dst, which may be either a struct pointer, a PropertyLoadSaver or a Map (although Maps are deprecated). If there is no such entity for the key, Get returns ErrNoSuchEntity.

  The values of dst's unmatched struct fields or Map entries are not modified. In particular, it is recommended to pass either a pointer to a zero valued struct or an empty Map on each Get call.

  ErrFieldMismatch is returned when a field is to be loaded into a different type than the one it was stored from, or when a field is missing or unexported in the destination struct. ErrFieldMismatch is only returned if dst is a struct pointer.

func GetMulti

  `func GetMulti(c appengine.Context, key []*Key, dst []interface{}) os.Error`

  GetMulti is a batch version of Get.

func LoadStruct

  `func LoadStruct(dst interface{}, c <-chan Property) os.Error`

  LoadStruct loads the properties from c to dst, reading from c until closed. dst must be a struct pointer.

func PutMulti

  `func PutMulti(c appengine.Context, key []*Key, src []interface{}) ([]*Key, os.Error)`

  PutMulti is a batch version of Put.

func RunInTransaction

  `func RunInTransaction(c appengine.Context, f func(tc appengine.Context) os.Error, opts *TransactionOptions) os.Error`

  RunInTransaction runs f in a transaction. It calls f with a transaction context tc that f should use for all App Engine operations.

  If f returns nil, RunInTransaction attempts to commit the transaction, returning nil if it succeeds. If the commit fails due to a conflicting transaction, RunInTransaction retries f, each time with a new transaction context. It gives up and returns ErrConcurrentTransaction after three failed attempts.

  If f returns non-nil, then any datastore changes will not be applied and RunInTransaction returns that same error. The function f is not retried.

  Note that when f returns, the transaction is not yet committed. Calling code must be careful not to assume that any of f's changes have been committed until RunInTransaction returns nil.

  Nested transactions are not supported; c may not be a transaction context.

func SaveStruct

  `func SaveStruct(src interface{}, c chan<- Property) os.Error`

  SaveStruct saves the properties from src to c, closing c when done. src must be a struct pointer.

## Types

type ErrFieldMismatch

ErrFieldMismatch is returned when a field is to be loaded into a different type than the one it was stored from, or when a field is missing or unexported in the destination struct. StructType is the type of the struct pointed to by the destination argument passed to Get or to Iterator.Next.

      type ErrFieldMismatch struct {
          StructType reflect.Type
          FieldName  string
          Reason     string
      }

  func (*ErrFieldMismatch) String

    `func (e *ErrFieldMismatch) String() string`

    String returns a string representation of the error.

type ErrMulti

ErrMulti indicates that a batch operation failed on at least one element.

      type ErrMulti []os.Error

  func (ErrMulti) String

    `func (m ErrMulti) String() string`

    String returns a string representation of the error.

type Iterator

Iterator is the result of running a query.

      type Iterator struct {
          // contains filtered or unexported fields
      }

  func (*Iterator) Next

    `func (t *Iterator) Next(dst interface{}) (*Key, os.Error)`

    Next returns the key of the next result. When there are no more results, Done is returned as the error. If the query is not keys only, it also loads the entity stored for that key into the struct pointer or Map dst, with the same semantics and possible errors as for the Get function. If the query is keys only, it is valid to pass a nil interface{} for dst.

type Key

Key represents the datastore key for a stored entity, and is immutable.

      type Key struct {
          // contains filtered or unexported fields
      }

  func DecodeKey

    `func DecodeKey(encoded string) (*Key, os.Error)`

    DecodeKey decodes a key from the opaque representation returned by Encode.

  func NewIncompleteKey

    `func NewIncompleteKey(c appengine.Context, kind string, parent *Key) *Key`

    NewIncompleteKey creates a new incomplete key. kind cannot be empty.

  func NewKey

    `func NewKey(c appengine.Context, kind, stringID string, intID int64, parent *Key) *Key`

    NewKey creates a new key. kind cannot be empty. Either one or both of stringID and intID must be zero. If both are zero, the key returned is incomplete. parent must either be a complete key or nil.

  func Put

    `func Put(c appengine.Context, key *Key, src interface{}) (*Key, os.Error)`

    Put saves the entity src into the datastore with key k. src may be either a struct pointer, a PropertyLoadSaver or a Map (although Maps are deprecated); if the former then any unexported fields of that struct will be skipped. If k is an incomplete key, the returned key will be a unique key generated by the datastore.

  func (*Key) AppID

    `func (k *Key) AppID() string`

    AppID returns the key's application ID.

  func (*Key) Encode

    `func (k *Key) Encode() string`

    Encode returns an opaque representation of the key suitable for use in HTML and URLs. This is compatible with the Python and Java runtimes.

  func (*Key) Eq

    `func (k *Key) Eq(o *Key) bool`

    Eq returns whether two keys are equal.

  func (*Key) GobDecode

    `func (k *Key) GobDecode(buf []byte) os.Error`

  func (*Key) GobEncode

    `func (k *Key) GobEncode() ([]byte, os.Error)`

  func (*Key) Incomplete

    `func (k *Key) Incomplete() bool`

    Incomplete returns whether the key does not refer to a stored entity. In particular, whether the key has a zero StringID and a zero IntID.

  func (*Key) IntID

    `func (k *Key) IntID() int64`

    IntID returns the key's integer ID, which may be 0.

  func (*Key) Kind

    `func (k *Key) Kind() string`

    Kind returns the key's kind (also known as entity type).

  func (*Key) MarshalJSON

    `func (k *Key) MarshalJSON() ([]byte, os.Error)`

  func (*Key) Parent

    `func (k *Key) Parent() *Key`

    Parent returns the key's parent key, which may be nil.

  func (*Key) String

    `func (k *Key) String() string`

    String returns a string representation of the key.

  func (*Key) StringID

    `func (k *Key) StringID() string`

    StringID returns the key's string ID (also known as an entity name or key name), which may be "".

  func (*Key) UnmarshalJSON

    `func (k *Key) UnmarshalJSON(buf []byte) os.Error`

type Map

Map is a map representation of an entity's fields. It is more flexible than but not as strongly typed as a struct representation.

Map is deprecated. It cannot represent whether a property is indexed. Use a PropertyList instead.

      type Map map[string]interface{}

type Property

Property is a name/value pair plus some metadata. A datastore entity's contents are loaded and saved as a sequence of Properties. An entity can have multiple Properties with the same name, provided that p.Multiple is true on all of that entity's Properties with that name.

      type Property struct {
          // Name is the property name.
          Name string
          // Value is the property value. The valid types are:
          //    - int64
          //    - bool
          //    - string
          //    - float64
          //    - *Key
          //    - Time
          //    - appengine.BlobKey
          //    - []byte (up to 1 megabyte in length)
          // This set is smaller than the set of valid struct field types that the
          // datastore can load and save. A Property Value cannot be a slice (apart
          // from []byte); use multiple Properties instead. Also, a Value's type
          // must be explicitly on the list above; it is not sufficient for the
          // underlying type to be on that list. For example, a Value of "type
          // myInt64 int64" is invalid. Smaller-width integers and floats are also
          // invalid. Again, this is more restrictive than the set of valid struct
          // field types.
          Value interface{}
          // NoIndex is whether the datastore cannot index this property.
          NoIndex bool
          // Multiple is whether the entity can have multiple properties with
          // the same name. Even if a particular instance only has one property with
          // a certain name, Multiple should be true if a struct would best represent
          // it as a field of type []T instead of type T.
          Multiple bool
      }

type PropertyList

PropertyList converts a []Property to implement PropertyLoadSaver.

      type PropertyList []Property

  func (*PropertyList) Load

    `func (l *PropertyList) Load(c <-chan Property) os.Error`

    Load loads all of c's properties into l. It does not first reset *l to an empty slice.

  func (*PropertyList) Save

    `func (l *PropertyList) Save(c chan<- Property) os.Error`

    Save saves all of l's properties to c.

type PropertyLoadSaver

PropertyLoadSaver can be converted from and to a sequence of Properties. Load should drain the channel until closed, even if an error occurred. Save should close the channel when done, even if an error occurred.

      type PropertyLoadSaver interface {
          Load(<-chan Property) os.Error
          Save(chan<- Property) os.Error
      }

type Query

Query represents a datastore query.

      type Query struct {
          // contains filtered or unexported fields
      }

  func NewQuery

    `func NewQuery(kind string) *Query`

    NewQuery creates a new Query for a specific entity kind. The kind must be non-empty.

  func (*Query) Ancestor

    `func (q *Query) Ancestor(ancestor *Key) *Query`

    Ancestor sets the ancestor filter for the Query. The ancestor should not be nil.

  func (*Query) Count

    `func (q *Query) Count(c appengine.Context) (int, os.Error)`

    Count returns the number of results for the query.

  func (*Query) Filter

    `func (q *Query) Filter(filterStr string, value interface{}) *Query`

    Filter adds a field-based filter to the Query. The filterStr argument must be a field name followed by optional space, followed by an operator, one of ">", "<", ">=", "<=", or "=". Fields are compared against the provided value using the operator. Multiple filters are AND'ed together. The Query is updated in place and returned for ease of chaining.

  func (*Query) GetAll

    `func (q *Query) GetAll(c appengine.Context, dst interface{}) ([]*Key, os.Error)`

    GetAll runs the query in the given context and returns all keys that match that query, as well as appending the values to dst. The dst must be a pointer to a slice of structs, struct pointers, or Maps. If q is a “keys-only” query, GetAll ignores dst and only returns the keys.

  func (*Query) KeysOnly

    `func (q *Query) KeysOnly() *Query`

    KeysOnly configures the query to return just keys, instead of keys and entities.

  func (*Query) Limit

    `func (q *Query) Limit(limit int) *Query`

    Limit sets the maximum number of keys/entities to return. A zero value means unlimited. A negative value is invalid.

  func (*Query) Offset

    `func (q *Query) Offset(offset int) *Query`

    Offset sets how many keys to skip over before returning results. A negative value is invalid.

  func (*Query) Order

    `func (q *Query) Order(fieldName string) *Query`

    Order adds a field-based sort to the query. Orders are applied in the order they are added. The default order is ascending; to sort in descending order prefix the fieldName with a minus sign (-).

  func (*Query) Run

    `func (q *Query) Run(c appengine.Context) *Iterator`

    Run runs the query in the given context.

type Time

Time is the number of microseconds since the Unix epoch, January 1, 1970 00:00:00 UTC.

It is a distinct type so that loading and saving fields of type Time are displayed correctly in App Engine tools like the Admin Console.

      type Time int64

  func SecondsToTime

    `func SecondsToTime(n int64) Time`

    SecondsToTime converts an int64 number of seconds since to Unix epoch to a Time value.

  func (Time) Time

    `func (t Time) Time() *time.Time`

    Time returns a *time.Time from a datastore time.

type TransactionOptions

TransactionOptions are the options for running a transaction.

      type TransactionOptions struct {
          // XG is whether the transaction can cross multiple entity groups. In
          // comparison, a single group transaction is one where all datastore keys
          // used have the same root key. Note that cross group transactions do not
          // have the same behavior as single group transactions. In particular, it
          // is much more likely to see partially applied transactions in different
          // entity groups, in global queries.
          // It is valid to set XG to true even if the transaction is within a
          // single entity group.
          XG bool
      }