English ▼
Skip to page content Skip to main navigation
e.g. "templates" or "datastore"

# Using the XMPP service

Nick Johnson
September 2009, updated October 2011

## Introduction

With the introduction of the XMPP service to App Engine, it's now possible to write an App Engine app that communicates with users - or even other applications - over XMPP. XMPP is an instant-messaging protocol, used by Google Talk, Jabber, and other IM networks. In this article, we're going to walk through an example that covers all the basic functionality of App Engine's XMPP API.

For our example app, we're going to write the Amazing Crowd Guru. The Amazing Crowd Guru is a veritable oracle, who can answer any question you might pose it over XMPP. Writing an omniscient computer program is no small task, but thanks to a little behind-the-scenes trickery, we're going to get our users to do all the work of answering questions for us.

The basic sequence of events will go like this:

1. A user adds `crowdguru@appspot.com` to their buddy list in Google Talk, or another XMPP client.
2. The user asks the Amazing Crowd Guru a question, by typing "/tellme Does a duck's quack echo?"
3. Our code receives the question, stores it in the datastore as an unanswered question, then looks in the datastore for another unanswered question. If it finds one, it sends it back to the user, saying "While I'm thinking, perhaps you can answer me this: If a mole can dig a mole of holes, how many moles of holes can a mole of moles dig?"
4. The user thinks a bit, and replies "A mole of moles can dig a mole of holes!"
5. Our code receives the user's answer, stores it in the datastore alongside the original question, and then sends it back to the user who originally asked that question.

In addition to the basic flow above, we'll add a couple of enhancements: A user can type "/help" to ask the Guru for a quick overview of what they can type, and they can type "/askme", to be asked a question by the Guru, without having to ask one of their own. We'll also suspend questions for users who are offline, to ensure that answers are delivered to users who are available.

## Getting Started

First, we need to set up a few basic settings. Create a directory for your app, and create your configuration file:

- Python
- Java

`app.yaml`

    application: crowdguru
    version: 1
    runtime: python
    api_version: 1

    handlers:
    - url: /.*
      script: guru.py

    inbound_services:
    - xmpp_message
    - xmpp_presence

`war/WEB-INF/appengine-web.xml`

    <?xml version="1.0" encoding="utf-8"?>
    <appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
      <application>crowdguru</application>
      <version>1</version>

      <system-properties>
        <property name="java.util.logging.config.file" value="WEB-INF/logging.properties" />
      </system-properties>

      <inbound-services>
        <service>xmpp_message</service>
        <service>xmpp_presence</service>
      </inbound-services>
    </appengine-web-app>

If you're planning to deploy your version of the app to App Engine, you'll want to replace 'crowdguru' with another app ID that you've created.

Next, if you're building your application in Python, create a handler script called `guru.py` and put the following in it:

- Python

`guru.py`

    from google.appengine.ext import db
    from google.appengine.ext import webapp
    from google.appengine.ext.webapp.util import run_wsgi_app
    from google.appengine.ext.webapp import template
    from google.appengine.ext.webapp import xmpp_handlers

    application = webapp.WSGIApplication([], debug=True)

    def main():
        run_wsgi_app(application)

    if __name__ == "__main__":
        main()

All of this should seem very familiar from writing previous apps, so we won't go into much detail describing it. Note that we haven't defined any models or handlers yet - we'll do that next.

## Creating a Model

Before we can do anything useful, we'll need a datastore model class to store questions and answers in. If you're building your application in Python, add the following to `guru.py` immediately after the import statements. If you're building in Java, create a new domain class, `Question.java`, with the following:

- Python
- Java (JDO)

`guru.py`

    class Question(db.Model):
      question = db.TextProperty(required=True)
      asker = db.IMProperty(required=True)
      asked = db.DateTimeProperty(required=True, auto_now_add=True)
      suspended = db.BooleanProperty(required=True)

      assignees = db.ListProperty(db.IM)
      last_assigned = db.DateTimeProperty()

      answer = db.TextProperty()
      answerer = db.IMProperty()
      answered = db.DateTimeProperty()

`src/Question.java`

    import com.google.appengine.api.datastore.Key;
    import com.google.appengine.api.datastore.Text;

    import java.util.Date;

    import javax.jdo.annotations.IdGeneratorStrategy;
    import javax.jdo.annotations.IdentityType;
    import javax.jdo.annotations.PersistenceCapable;
    import javax.jdo.annotations.Persistent;
    import javax.jdo.annotations.PrimaryKey;

    @PersistenceCapable(identityType = IdentityType.APPLICATION, detachable="true")
    public class Question {

      @PrimaryKey
      @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
      private Key key;

      @Persistent(defaultFetchGroup = "true")
      private Text question;

      @Persistent
      private String asker;

      @Persistent
      private Date asked;

      @Persistent(defaultFetchGroup = "true")
      private Text answer;

      @Persistent
      private String answerer;

      @Persistent
      private Date answered;

      @Persistent
      private Boolean suspended;

      public Key getKey() {
        return key;
      }

      // ...

We've omitted a couple of helper methods here to save space, including several Python functions dealing with assigning questions to a user in a transaction-safe manner. If you want to see all the gory details, the source is freely available and linked from here.

## Responding to XMPP messages

Finally, we're ready to tackle the meat of the problem: Receiving and responding to XMPP messages sent by users. As described in the introduction, we're using a pattern fairly common to 'bots': Users send us either plain text messages, or commands prefixed with a "/" character. Messages are interpreted based on the command specified and the (optional) argument.

The Python environment includes some convenience classes to make writing XMPP bots easier, namely `CommandHandler` in `google.appengine.ext.webapp`. This class implements a "command dispatch" pattern: messages prefixed with "/foo" are handled by a method named `foo_command`, messages without a prefix are handled by `text_message`, and any messages with unknown prefixes are handled by `unhandled_command` which sends a message back to the sender saying "Unknown command".

While the Java environment does not have a special handler for dispatching XMPP commands, its XMPP API includes a handy `parseMessage` method for creating a new `Message` object from HTTP request data, and from there it's straightforward to inspect the body of the message to determine how to handle the command.

With these techniques, we can write the code to handle incoming requests to the Guru and send responses fairly simply:

- Python
- Java

`guru.py`

    # ...

    class XmppHandler(xmpp_handlers.CommandHandler):
      """Handler class for all XMPP activity."""

      def tellme_command(self, message=None):
        """Handles /tellme requests, asking the Guru a question."""
        # Identify the user by bare jid
        # (see http://wiki.xmpp.org/web/Jabber_Resources)
        user = message.sender.split('/')[0]
        im_from = db.IM("xmpp", user)
        asked_question = self._GetAsked(im_from)
        currently_answering = self._GetAnswering(im_from)

        if asked_question:
          # Already have a question
          message.reply(WAIT_MSG)
        else:
          # Asking a question
          asked_question = Question(question=message.arg, asker=im_from,
                                    suspended=False)
          asked_question.put()

          if not currently_answering:
            # Try and find one for them to answer
            question = Question.assignQuestion(im_from)
            if question:
              message.reply(TELLME_MSG % (question.question,))
              return
          message.reply(PONDER_MSG)

`src/XmppReceiverServlet.java`

    import com.google.appengine.api.xmpp.JID;
    import com.google.appengine.api.xmpp.Message;
    import com.google.appengine.api.xmpp.MessageBuilder;
    import com.google.appengine.api.xmpp.MessageType;
    import com.google.appengine.api.xmpp.XMPPService;
    import com.google.appengine.api.xmpp.XMPPServiceFactory;

    import java.io.IOException;
    import java.util.Date;

    import javax.servlet.http.HttpServlet;
    import javax.servlet.http.HttpServletRequest;
    import javax.servlet.http.HttpServletResponse;

    /**
     * Handler class for all XMPP messages.
     */
    public class XmppReceiverServlet extends HttpServlet {

      private static final XMPPService xmppService =
          XMPPServiceFactory.getXMPPService();

      public void doPost(HttpServletRequest req, HttpServletResponse resp)
          throws IOException {
        Message message = xmppService.parseMessage(req);

        if (message.getBody().startsWith("/askme")) {
          handleAskMeCommand(message);
        } else if (message.getBody().startsWith("/tellme ")) {
          String questionText = message.getBody().replaceFirst("/tellme ", "");
          handleTellMeCommand(message, questionText);
        } else if (message.getBody().startsWith("/")) {
          handleUnrecognizedCommand(message);
        } else {
          handleAnswer(message);
        }
      }

      /**
       * Handles /tellme requests, asking the Guru a question.
       */
      private void handleTellMeCommand(Message message, String questionText) {
        QuestionService questionService = new QuestionService();

        JID sender = message.getFromJid();
        Question previouslyAsked = questionService.getAsked(sender);

        if (previouslyAsked != null) {
          // Already have a question, and they're not answering one
          replyToMessage(message, "Please! One question at a time! You can ask " +
              "me another once you have an answer to your current question.");
        } else {
          // Asking a question
          Question question = new Question();
          question.setQuestion(questionText);
          question.setAsked(new Date());
          question.setAsker(sender);

          questionService.storeQuestion(question);

          // Try and find one for them to answer
          Question assigned = questionService.assignQuestion(sender);

          if (assigned != null) {
            replyToMessage(message, "While I'm thinking, perhaps you can " +
                "answer me this: " + assigned.getQuestion());
          } else {
            replyToMessage(message, "Hmm. Let me think on that a bit.");
          }
        }
      }

      // ...

      private void replyToMessage(Message message, String body) {
        Message reply = new MessageBuilder()
            .withRecipientJids(message.getFromJid())
            .withMessageType(MessageType.NORMAL)
            .withBody(body)
            .build();

        xmppService.sendMessage(reply);
      }
    }

This is the core functionality: how users ask the Guru a question. We've left out the `_GetAsked()` (Python) and `getAsked` (Java) methods here for brevity; they are fairly straightforward methods that query the datastore for the question the user asked (if any). If you're interested, you can see the full, unmodified code, linked from here.

As you can see, the actual functionality of using XMPP is very straightforward. In Python, received messages result in a call to the appropriate function which is passed a message object. In Java, the message object is parsed directly from the posted request. The message object encapsulates information about the sender of the message, the JID (Jabber ID) it was sent to, and the body of the message. The Python implementation also provides convenience methods to parse the body of the message into command and argument portions (if it is of that form), and to reply to the message.

In this case, the only user we're sending messages to is the sender of the message we're handling, so everything is very straightforward - we just call `message.reply()` with the text we want to send back to the user.

Next, we need a way for the user to send us their answer to a question. They do this by simply typing the answer, with no /command prefix:

- Python
- Java

`guru.py`

      def text_message(self, message=None):
        user = message.sender.split('/')[0]
        im_from = db.IM("xmpp", user)
        question = self._GetAnswering(im_from)
        if question:
          other_assignees = question.assignees
          other_assignees.remove(im_from)

          # Answering a question
          question.answer = message.arg
          question.answerer = im_from
          question.assignees = []
          question.answered = datetime.datetime.now()
          question.put()

          # Send the answer to the asker
          xmpp.send_message(question.asker.address,
                            ANSWER_INTRO_MSG % (question.question,))
          xmpp.send_message(question.asker.address, ANSWER_MSG % (message.arg,))

          # Send acknowledgement to the answerer
          asked_question = self._GetAsked(im_from)
          if asked_question:
            message.reply(TELLME_THANKS_MSG)
          else:
            message.reply(THANKS_MSG)

          # Tell any other assignees their help is no longer required
          if other_assignees:
            xmpp.send_message([x.address for x in other_assignees],
                              SOMEONE_ANSWERED_MSG)
        else:
          self.unhandled_command(message)

`src/XmppReceiverServlet.java`

      // ...

      /**
       * Handles answers to questions we've asked the user.
       */
      private void handleAnswer(Message message) {
        QuestionService questionService = new QuestionService();

        JID sender = message.getFromJid();
        Question currentlyAnswering = questionService.getAnswering(sender);

        if (currentlyAnswering != null) {
          // Answering a question
          currentlyAnswering.setAnswer(message.getBody());
          currentlyAnswering.setAnswered(new Date());
          currentlyAnswering.setAnswerer(sender);

          questionService.storeQuestion(currentlyAnswering);

          // Send the answer to the asker
          sendMessage(currentlyAnswering.getAsker(), "I have thought long and " +
              "hard, and concluded: " + currentlyAnswering.getAnswer());

          // Send acknowledgment to the answerer
          Question previouslyAsked = questionService.getAsked(sender);

          if (previouslyAsked != null) {
            replyToMessage(message, "Thank you for your wisdom. I'm still " +
                "thinking about your question.");
          } else {
            replyToMessage(message, "Thank you for your wisdom.");
          }
        } else {
          handleUnrecognizedCommand(message);
        }
      }

      // ...

      private void sendMessage(String recipient, String body) {
        Message message = new MessageBuilder()
            .withRecipientJids(new JID(recipient))
            .withMessageType(MessageType.NORMAL)
            .withBody(body)
            .build();

        xmppService.sendMessage(message);
      }

This is a little longer than tellme_command, but still fairly easy to follow. First we look up the question the user is answering (again, we've omitted the `_GetAnswering` (Python) and `getAnswering` (Java) methods for brevity). If they're not answering a question, we call `unhandled_command()` (Python) or `handleUnrecognizedCommand` (Java), which prints help text. Otherwise, we record the answer, send it back to the person who originally asked the question, and send an acknowledgement to the answerer. Note the bit where we send the answer to the original asker: This demonstrates using `xmpp.send_message` (Python) to send a message to a JID other than the sender of the message we're currently handling. In Java, we've added a custom method for sending a message to an arbitrary sender.

If a user asks a question and then goes offline, we shouldn't bother answering their question until they're around to hear the answer. We can track this using presence handlers.

- Python
- Java

`guru.py`

        class XmppPresenceHandler(webapp.RequestHandler):
          def post(self, available):
            user = self.request.get('from').split('/')[0]
            im_from = db.IM("xmpp", user)
            suspend = available == 'unavailable'
            q = Question.all()
            q.filter('asker =', im_from)
            q.filter('answer =', None)
            q.filter('suspended =', not suspend)
            question = q.get()
            if question:
              question.suspended = suspend
              question.put()

      public void doPost(HttpServletRequest request, HttpServletResponse response)
          throws IOException {
        Presence presence = xmppService.parsePresence(request);
        JID sender = presence.getFromJid();

        QuestionService questionService = new QuestionService();
        if (presence.getPresenceType() == PresenceType.AVAILABLE) {
          questionService.setSuspended(sender, true);
        } else if (presence.getPresenceType() == PresenceType.UNAVAILABLE) {
          questionService.setSuspended(sender, false);
        }
      }

When we are notified that a user has gone offline, we search our database for an unanswered question that they have asked. If one exists, we mark it as "suspended" and don't give it to other users to answer. When the user comes back online, we un-suspend the question so it's available to be answered again.

Finally, it would be nice if users have a way to answer questions without having to ask one first; this way, dedicated users can help clear out any backlog of unanswered questions:

- Python
- Java

`guru.py`

        def askme_command(self, message=None):
            """Handles requests to be asked a question without providing one."""
            im_from = db.IM("xmpp", message.sender)
            question = Question.assignQuestion(im_from)
            if question:
                message.reply(TELLME_MSG % (question.question,))
            else:
                message.reply(EMPTYQ_MSG)

`src/XmppReceiverServlet.java`

      // ...

      /**
       *  Handles requests to be asked a question without providing one.
       */
      private void handleAskMeCommand(Message message) {
        QuestionService questionService = new QuestionService();

        JID sender = message.getFromJid();
        Question newlyAssigned = questionService.assignQuestion(sender);

        if (newlyAssigned != null) {
          replyToMessage(message, "While I'm thinking, perhaps you can answer " +
              "me this: " + newlyAssigned.getQuestion());
        } else {
          replyToMessage(message, "Sorry, I don't have anything to ask you at " +
              "the moment.");
        }
      }

      // ...

This method is even simpler - grab a question and send it back to the user. In fact, it's pretty much just a subset of the code we saw in the `tellme_command()` method.

There's one last thing we need to do to get this all working, of course - hook it up to the serving infrastructure so it can serve requests. In Python, a CommandHandler is a standard webapp RequestHandler subclass, so we can set it up as we would any other handler. In Java, you can wire up the XMPP receiving servlet in your deployment descriptor.

- Python
- Java

`guru.py`

    application = webapp.WSGIApplication([
        ('/_ah/xmpp/message/chat/', XmppHandler),
        ('/_ah/xmpp/presence/(available|unavailable)/', XmppPresenceHandler),
        ], debug=True)

`war/WEB-INF/web.xml`

    <?xml version="1.0" encoding="utf-8"?>
    <web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
      xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
    http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
      version="2.5">
      <servlet>
        <servlet-name>xmppreceiver</servlet-name>
        <servlet-class>com.google.appengine.demos.crowdguru.web.XmppReceiverServlet</servlet-class>
      </servlet>
      <servlet-mapping>
        <servlet-name>xmppreceiver</servlet-name>
        <url-pattern>/_ah/xmpp/message/chat/</url-pattern>
      </servlet-mapping>
    </web-app>

The URL path `/_ah/xmpp/message/chat` is reserved for XMPP messages to be sent to. Similarly, the paths `/_ah/xmpp/presence/available/` and `/_ah/xmpp/presence/unavailable` are reserved for XMPP presence notifications.

Presto! We have a working (and amusing) XMPP bot! For the full source code, including the bits we left out for space reasons, see the next section. If you just want to test the wisdom of the Crowd Guru, there's a slightly enhanced version of this demo running on http://crowdguru.appspot.com/ - just add crowdguru@appspot.com to your Google Talk or Jabber buddy list to try it out!

## Source

The Python source for the full implementation of the crowdguru sample application is available in the Google App Engine Samples project as crowdguru. The Java source is available in the java/demos directory of the Google App Engine SDK project as crowdguru.