whyfore chat, with django, twisted and websockets?

Aside

upon seeing the work i’ve put into writing tutorials, showing how to get realtime chat working in django + twisted/websockets, you might make the assumption that i consider this architecture to be, in general, a good idea.

A.

twisted’s implementation of websockets is, as of this writing, not integrated into the main branch.

don’t use code that isn’t considered, by its authors, to be reliable enough to merge into and release as part of their application distribution.

B.

twisted is an event-driven networking engine
django is a solid, easy to use web framework
websockets, a tcp based protocol, is usually implemented as a strange mix between the tcp and http protocols

it is, generally speaking, not a good idea to mix abstraction levels; adding event-driven components to your application by combining twisted and django is a bad architectural decision. I strongly suggest you consider using twisted.web instead of mixing django and twisted.

websockets are a strange mix of protocols, and can be difficult to work with unless you are very careful with your choice of libraries and application design, scope and implementation. at the time of this post, i would recommend against using websockets, in production, with the standard deployment of twisted. i strongly urge you to consider the following alternatives, in rough order of likelihood to work for you:

 


TUTORIAL: Real-time chat with Django, Twisted and WebSockets – Part 2

[Part 1] - [Part 2] - [Part 3] - [Addendums] - [Source]
[Table of Contents]

Thus far, we have a very basic UI framework for a non-functional chat system, served by Django. We’ll now implement a chat server, chat client, and api, with the chat functionality being managed and served by Twisted via websockets.

Install Twisted with Websockets 

I used this git branch of the Twisted project, as it’s the most up-to-date as of the time of this writing. The specific steps you’ll have to take to install it depend on your architecture, but you’ll probably want to run commands similar to these ones:

#checkout the twisted project
git clone https://github.com/twisted/twisted.git twisted-websocket

#switch to the most up to date websocket branch
cd twisted-websocket
git fetch
git checkout websocket-4173-4

#install it - preferably do this in a virtualenv
python setup.py install

Write a web socket chat server for twisted

Create a file to store the chat server code, say, chatserver.py, and write the following code into it:

from twisted.protocols import basic
from twisted.web.websockets import WebSocketsResource, WebSocketsProtocol, lookupProtocolForFactory

#basic protocol/api for handling realtime chat
class MyChat(basic.LineReceiver):
    def connectionMade(self):
        print "Got new client!"
        self.transport.write('connected ....\n')
        self.factory.clients.append(self)

    def connectionLost(self, reason):
        print "Lost a client!"
        self.factory.clients.remove(self)

    def dataReceived(self, data):
        print "received", repr(data)
        for c in self.factory.clients:
            c.message(data)

    def message(self, message):
        self.transport.write(message + '\n')

from twisted.web.resource import Resource
from twisted.web.server import Site
from twisted.internet import protocol
from twisted.application import service, internet

#Create a protocol factory
#The factory is usually a singleton, and
#all instantiated protocols should have a reference to it,
#so we'll use it to store shared state
#(the list of currently connected clients)
from twisted.internet.protocol import Factory
class ChatFactory(Factory):
protocol = MyChat
clients = []

resource = WebSocketsResource(lookupProtocolForFactory(ChatFactory()))
root = Resource()
#serve chat protocol on /ws
root.putChild("ws",resource)

application = service.Application("chatserver")
#run a TCP server on port 1025, serving the chat protocol.
internet.TCPServer(1025, Site(root)).setServiceParent(application)

The code above follows one of the twisted code samples/tutorials, implementing a basic telnet chat server, and modifies it to use the web socket protocol and related infrastructure, instead of the basic tcpip one. Note that this example is still using a wrapper around the basic.lineReceiver protocol – nothing too fancy.

Since, a few steps from now, we’re going to want to serve more than one protocol (one for web sockets, and the other for http/long poll requests), the code is explicitly creating a Site(), and building a WebSocketResource/ChatFactory to deploy (as opposed to relying on ServerFactory() to build one for us). The websockets api is being served at http://127.0.0.1:8000/ws/

This is the extent of work required to get a very basic twisted server working. Now, try running it:

bash: twistd -n -y chatserver.py

If the Django server is running, also, you should now be able to connect to

http://127.0.0.1:8000/chats/1

see the chat room, and, well, still not be able to properly send messages back and forth, since we don’t yet have a client-side component connecting to our server over websockets, and handling that side of the communication. So let’s do that next:

Download and install a javascript web socket client library

Not all browsers implement websockets, and not all websockets implementations are equivalent, so it’s difficult to write code which is portable. It’s easier to use a library for handling this kind of work – I used this one: https://jquery-graceful-websocket.googlecode.com/files/jquery.gracefulWebSocket.js

Download the file, and place it under chat/static/. (You might have to create the directory):

bash: mkdir chat/static
bash: cp ~/Downloads/jquery.gracefulWebSocket.js chat/static/

Implement client-side web socket logic:

We’re already serving the chat-room scaffolding through a template. One way to provide the clientside components for a websockets connection is through javascript embedded directly in the source html file. So let’s do that – edit chat/templates/chats/chat_room.html, and add the following to the header section:

<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script type="text/javascript" src="{% static "jquery.gracefulWebSocket.js" %}"></script>
<script>
    $(document).ready( function() {
    window.chat = {};
    //Instantiate a websocket client connected to our server
    chat.ws = $.gracefulWebSocket("ws://127.0.0.1:1025/ws");

    //Basic message send
    chat.send = function (message) {
      chat.ws.send(message);
    }

    //Basic message receive
    chat.ws.onmessage = function (event) {
      var messageFromServer = event.data;
      var list_element = document.createElement('li');
      list_element.innerHTML = messageFromServer;
      $("#message_list ul").prepend(list_element);
    };

    var inputBox = document.getElementById("inputbox");

    inputbox.addEventListener("keydown", function(e) {
      if (!e) { var e = window.event; }

      //keyCode 13 is the enter/return button keyCode
      if (e.keyCode == 13) {
        // enter/return probably starts a new line by default
        e.preventDefault();
        chat.send(inputbox.value);
        inputbox.value="";
      }
    }, false); });
</script>

The script first includes jQuery, which the gracefulWebsocket library relies on. The order of inclusion matters, so don’t switch those lines around. It then defines a namespace for our chat functionality (window.chat), and instantiates a gracefulWebsocket in the namespace, configuring it to connect to our web socket chat server:

chat.ws = $.gracefulWebSocket("ws://127.0.0.1:1025/ws");

We need two functions, let’s call them chat.send(), for sending messages out, and chat.onmessage() for handling messages received from the server – the code is fairly readable. We’re also adding an event listener to our input text box, sending out a message each time the user presses return.

Make sure you’ve saved the file, and, voila, you should now have a working, very basic, web-socket based chat server. The static/html content served by Django, and a websockets based chat api being served/managed by twisted.

Django is smart enough to know to pick up changes to its templates, so all of this should now work, without requiring a restart of the server. Just reload the two windows you have opened on http://127.0.0.1/chats/1 (or go back to http://127.0.0.1/chats/, and open two browser windows on the same chat room) – and enjoy sending messages, in real time, from one window to the next.

Next: add a http api for your chat rooms, and a long-polling based client to connect to them.

TUTORIAL: Real-time chat with Django, Twisted and WebSockets – Part 1

[Part 1] - [Part 2] - [Part 3] - [Addendums] - [Source]
[Table of Contents]

(I make some basic assumptions about the readers of this tutorial).

Overview

We’re going to be building a very basic chat server on top of Django 1.5, and Twisted 13.1. For this implementation, we’ll be relying on Django, and Django’s html templating facility, to design, build  and to serve the ui components for the application.

Twisted will provide the actual chat functionality, over two separate channels:

  • first, we’ll have twisted serving chat to websocket connections
  • we’ll then add an http api, to demonstrate what a long-polling chat client might look like

if you haven’t already, it’s probably worth spending a few minutes making sure you have gitpipvirtualenv and virtualenvwrapper installed and working properly, and then set up an environment using Django 1.5.

Django scaffolding

create a new Django project:

bash: django-admin.py startproject django_twisted_chat

test that the project is up and running properly by starting the server:

bash: cd django_twisted_chat
bash: python manage.py runserver

and then connecting to http://127.0.0.1:8000/. You can check that the server is running – you’ll see a “Welcome to Django” page if it’s running properly.

Configure Django Project

edit django_twisted_chat/settings.py and add the following lines to the top:

#convenience variable, to refer to the directory the application is currently running in:
import os
BASE_DIR = os.path.dirname(os.path.abspath(__file__))

configure a database; since this is a demo, and is unlikely to ever make it to production, we’ll use sqllite and store our data to disk:

DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
        'NAME': './sqlite_database_file',
         # The following settings are not used with sqlite3:

we’re going to be serving some static files in this demo, javascript files only for now (a more advanced version may serve images, and other static elements too), so we should tell Django where we’re storing these files. I used the default – a  “static” directory, off the base of the server and/or individual applications:

# Additional locations of static files
STATICFILES_DIRS = (
# Put strings here, like "/home/html/static" or "C:/www/django/static".
# Always use forward slashes, even on Windows.
# Don't forget to use absolute paths, not relative paths.
    os.path.join(BASE_DIR, "static"),
)

We’ll also use Django’s templating facilities, so configure the server to use them:

TEMPLATE_DIRS = (
# Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
# Always use forward slashes, even on Windows.
# Don't forget to use absolute paths, not relative paths.
os.path.join(BASE_DIR, 'templates'),
)

In a few minutes we’ll make our chat application, and we can reuse Django’s admin interface as a shortcut interface, for manually adding a few required elements to the database (ie., chat room definitions). So, let’s configure the server to serve both of those:

INSTALLED_APPS = (
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
'django.contrib.staticfiles',
'django.contrib.admin',
'chat',
)

Make sure to save your settings file.

Set up url paths

Now, well want the Django project to properly direct http requests to our chat application, as well as to the built in admin components. To do that, edit the django_twisted_chat/urls.py file, like so:

from django.conf.urls import patterns, include, url
from django.views.generic import RedirectView
from django.contrib import admin
admin.autodiscover()

urlpatterns = patterns('',
     url(r'^(/)?$', RedirectView.as_view(url='/chats/')),
     url(r'^chats/', include('chat.urls')),
     url(r'^admin/', include(admin.site.urls)),
)

Most of this should be familiar from the Django tutorial. The only potential addition is the use of the RedirectView.as_view shorthand. This is a built in shorthand tool for redirecting requests from one url to the other; we’re using it to redirect all requests for http://127.0.0.1 and/or http://127.0.0.1/ to http://127.0.0.1/chats/. We’ll using it here for convenience’s sake – since we’re not going to be serving anything on http://127.0.0.1/ for now, have the server automatically redirect requests to the /chats/ application instead of returning a 404 error.

Create and configure Django Chat application

Create a Django application to serve the chat specific components:

bash: python manage.py startapp chat

for now, we’ll only worry about adding the basic components for a web-sockets based chat client. Django will serve the ui and javascript source portions, letting Twisted handle the chat api and server components. First, construct the model for a chat room -edit chat/models.py:

from django.db import models

class ChatRoom(models.Model):
    name = models.CharField(max_length=200)

    def __unicode__(self):
	return self.name

We're building a model for storing a ChatRoom. Note that this specific tutorial doesn't go so far as to store actual chat messages to disk, so we won't need a ChatMessage model (though that model is probably present in the sample code provided at the end of this sequence). That's work for a future followup.

Make sure that we have a nice UI for editing the room of existing chat rooms. For now, we'll rely on the existing admin console for Django, so, create and edit the chat/admin.py file:

from django.contrib import admin
from chat.models import ChatRoom

admin.site.register(ChatRoom)

Next, we'll define the application specific urls, in chat/urls.py:

from django.conf.urls import patterns, url

from chat import views

urlpatterns = patterns('',
        url(r'^$', views.index, name='index'),
        url(r'^(?P\d+)/$', views.chat_room, name='chat_room'),
)

This is where the project urls file (django_twisted_chat/urls.py) is going to look for urls (a result of the "url(r'^chats/', include('chat.urls'))" line we wrote in there earlier).

Now, we'll need a couple of html templates for displaying chat room related information. The templates need somewhere to sit on disk, so create an app specific templates directory:

bash: mkdir chat/templates

then, create a templates directory for the url we'll be using (we called it chats in the django_twisted_chat/urls.py file, so this directory structure is what Django will, by default, expect):

bash: mkdir chat/templates/chats

We'll have our index list all available chat rooms, so create a template for that. Edit chat/templates/chats/index.html:

{% if chat_list %}</pre>
<ul>
<ul>{% for chat in chat_list %}</ul>
</ul>
<ul>
<ul>
	<li><a id="" href="{% url 'chat_room' chat.id %}"> {{ chat.name }}</a></li>
</ul>
</ul>
<ul>{% endfor %}</ul>
<pre>



{% else %}

No chats are available.

{% endif %}

Then create a page to serve as ui for an individual chat room. Edit chat/templates/chats/chat_room.html:

{% load staticfiles %}</pre>
<h1>{{ chat.name }}</h1>
<div id="message_list"></div>
<pre>



Next, we can construct the relevant views. Edit the chat/views.py file:

from django.http import HttpResponse
from django.shortcuts import render, get_object_or_404

from chat.models import ChatRoom

def index(request):
  chat_rooms = ChatRoom.objects.order_by('name')[:5]
  context = {
    'chat_list': chat_rooms,
  }
  return render(request,'chats/index.html', context)

def chat_room(request, chat_room_id):
  chat = get_object_or_404(ChatRoom, pk=chat_room_id)
  return render(request, 'chats/chat_room.html', {'chat': chat})

Again, this follows the basic pattern taught in the Django tutorial. The convenience method get_object_or_404 may be unfamiliar - it's a shorthand for searching for an element in the database, and automatically returning a 404 not found error if the object is missing (in this case, if someone accidentally tries to access a chat room that doesn't yet exist).

This should now give us a functional Django application. Let's make sure that it works properly, and that we don't have any typos. Create and define the structure of the database (make a super user for it when you create it), then start Django:

bash: python manage.py syncdb
bash: python manage.py runserver

Next, log in to the admin console and create a chat room. Go to:

http://127.0.0.1:8000/admin/

log in with the super user that you created, and add a ChatRoom (give it a name and save it).

To test that the chat room exists and everything we've done so far is working properly, go to:

http://127.0.0.1:8000/

You should automatically be redirected to http://127.0.0.1:8000/chats/, and see a list of existing chat rooms.

Chat Room List

Click on one of them, and you should now see an empty chat room on your screen:

Empty chat room

For fun, try opening the chat room url in multiple browser windows, and then typing in it. Unfortunately, nothing interesting happens yet. That's where the Twisted chat server comes in - see Part 2

TUTORIAL: Real-time chat with Django, Twisted and WebSockets – Prerequisites

If you came directly here, you can probably skip to Part 1.

I’m assuming a basic competency writing Python code, as well as access to a development environment, editor, and debug/test environment configured for work with Python (with possible extensions for working with Django).

If new at working with Python, you can do worse than using Eclipse’s pydev setup, though you’ll be fine working with any text editor.

This is not the place to learn about Django. Completing and understanding the Django tutorial is worthwhile, since I write code that’s very close to the code written there, in this tutorial. (If you’ve completed it before but are still shaky on Django architecture and terminology, it’s worth your rereading the tutorial, and re-completing the first 4 stages before continuing)

Twisted is a bit harder to get into. For the minimalist, read twisted in 60 seconds, and then take a look at relevant example 1 and example 2, and understand how they work.

Continue to Part 1.