WebSockets for Real-Time and Interactive Interfaces

Jason Ronallo
NCSU Libraries
@ronallo

Participate! Go to this address now:
http://go.ncsu.edu/websocket

Before I start please go to this URL. If someone could please paste this into IRC as well. Hi. I'm Jason Ronallo at NCSU Libraries and I'm going to be talking about WebSockets.

Audience Participation

Go to this address: http://go.ncsu.edu/websocket

Number of Audience Connections

Please go to this address. It ought to work on mobile devices as well. I'll wait a second. And then either: 1. It looks like people are connecting, or 2. This is a complete failure, why did I try to do this?!

Let's Try WebSockets

Or Will the Wireless Fail Us Now?

Gratuitous Chat Application

Every article or presentation about WebSockets has got to mention chat applications, so I thought we'd get an example out of the way from the start. If you went to that link, then you ought see a form field where you can send a message up here onto the screen. Be nice.

Now that that is out of the way.

What just happened?

Or should have happened. ;-)

Presentation Slides

When the presentation slides changed a message gets sent to this big black cloud that then broadcasts it to your browser.

Presentation Chat

Then you sent message to the big black cloud and it forwarded them all on to my slides and displayed them on the big screen. So what's in the big black cloud?

A WebSocket serving running node.js and socket.io. I'll show you how simple that is in a bit. Let's talk about WebSockets.

But before we talk about the problem that WebSockets tries to solve, we really need to talk about HTTP.

HTTP is a request-response protocol. The communication always starts on the client side. Traffic can only flow in one direction. The server cannot send a message to the client without the client making the request.

Solutions

So what are some solutions to these limitations?

There's polling, where the browser client makes request after request just in case there might be something on the server that's new. With basic polling this can mean opening up one connection after another until there's new data. But there's that polling interval.

With Long Polling the client makes a request and the server just holds open the connection open for as long as it can. If new data becomes available then the server responds. When the client gets some data, then the client fires off a new request. And the process continues over and over.

Queuing Latency

The time a message has to wait in the server before it can be delivered to the client.

The problem with both of these approaches is queuing latency. Even though this latency might not be much it is still present and depending on the application can really matter.

There's a lesser known API known as Server Sent Events where after the initial handshake creates a persistent connection, the server can continue to stream text to the client whenever it wants. The client can't use the same channel, but it can always just use regular AJAX. Some of what I've done here can probably be done with Server Sent Events which might be easier for you to get started with in some environments.

WebSockets are a standardised cross browser solution to real-time bi-directional communication between a server and a client.

After the initial handshake where the connection is upgraded, the client or server and send data whenever they want over a persistent connection. They can even send data at the same time.

This ends any latency that happens because of having to reopen a connection.

So it is a real full-duplex method of communication. Doge-approved.

WebSockets Message Payload

It is possible to implement subprotocols over WebSockets if you need to.

You can send text or binary. So anything you want.

WebSockets sound great!
Why not use WebSockets for everything?!

When you use WebSockets you lose caching, state management, and other services that XHR and HTTP in the browser give you for free. WebSockets force you to make decisions about how to handle these things.

Use the right protocol for the job.

What are WebSockets good for?

Anything like collaborative editing and HTML5 games where everything needs to be real-time.

My Use Cases

This is the NCSU Libraries Rare and Unique Digital Collections site. And it has a lot of great images.

Google Analytics Real-time

I got into real-time stuff because the Google Analytics Realtime was so addictive. But then--Oh, ten people are viewing this image at the same time, which image is that? And I'd have to copy and paste or do a search for it. So tedious.

Digital Collections Now!

I just wanted to see the images now.

Vanity Metrics

Let's be honest: This is vanity metrics. You're not going to make any business decisions from it, but this kind of tracking can still be useful. I don't know that we do enough vanity metrics to be able to tell the story about how what we do is popular and useful.

Real-time Digital Collections

So this is what it looks like. As people view a resource it shows up here. If it is viewed multiple times then it gets a badge. And they all just scroll down the page. People have called this view addictive. Which I guess is good?

Basketball Now

And when the page looks like this, I know it is basketball season.

Architecture of Now

Visitors request a resource page from the Rails app. Rails POSTs a message with that resource's id to the WebSocket server. And then node broadcasts that message out via WebSockets to clients on the real-time page. Finally the real-time client requests the image partial to display it on the page.

The realtime viewers subscribe via websockets to the updates.

Code

So let's say we have an architecture like this, what would some of the code look like?

Rails + Blacklight Server

# naive implementation in catalog_controller.rb
after_filter :websocket_update, only: [:show]
# POST id to 
# http://lib.example.edu:1337/message/room-name-development
def websocket_update
  if !robot_request?
    begin
      client = HTTPClient.new
      body = { id: params[:id] }
      post_url = "http://lib.example.edu:1337/message/room-name"
      res = client.post(post_url, body)
    rescue
    end
  end
end
This is a naive implementation, but after any Blacklight show view is rendered we post the id of the resource to the node server.

Node + Express + socket.io WebSocket Server

// some other requires for Express etal.
var io = require('socket.io').listen(server);

// Route for POSTs
app.post('/message/:room', function(req, res){
  io.sockets.in(req.params.room)
    .emit('update', req.body);
  res.send(200); // success!
});
Here we have an Express route for POST requests. All it does is broadcast the message containing the resource id to all of the clients that are in the given room.

WebSocket Client Using socket.io

// set up the socket.io connection
var host = location.protocol + '//' + 
           location.host;
var socket = io.connect(host);

socket.emit('subscribe', 
            {room: 'digital_collections-staging'});

socket.on('update', function(data){
  console.log(data);
  // Do something like make a request for the 
  // partial of the most recently viewed image.
});
On the client we connect to the WebSocket server. Immediately we send a message asking to be subscribed to a room. One trick here to allow us to reuse the same WebSocket server across applications and environments is to use the environment as part of the room name. And finally any time we receive a message we do something. Easy stuff.

Raw WebSockets on the Client

// Create a new WebSocket
var socket = 
  new WebSocket("ws://echo.websocket.org");
// receive a message from the server
socket.onmessage = function(event) {
  console.log(event.data);
}
// Send a message to the server.
// See it echo back in this example.
socket.send("Sending a message to the server!");
// close the connection
socket.close();
Even the JavaScript for working with WebSockets directly is really simple. Just open a socket, set some callbacks for onmessage and other events. And sending a message is easy.

If it is that easy why use a client library like socket.io provides?

Big
Video
Walls

Let's look at some of the big video walls in Hunt Library.

Staircase Wall

Gaming Lab

 

Gaming Lab

Art Wall

Immersion Theater (6824 x 2240)

Immersion Theater (6824 x 2240)
The immersion theater is this big curved wall. And it has seating and sound.

Listen to Wikipedia

You might have seen this before.

Listen to Wikipedia is an award winning web-based visualization and audiation of realtime wikipedia edits. When I saw it, I thought it would be perfect on the Immersion Theater wall.

Listen to Wikipedia

How's It Work?

Listen to Wikipedia Uses WebSockets

  1. Wikipedia uses IRC to broadcast changes for each language
  2. Wikimon listens to the IRC channels and broadcasts the data over WebSockets
  3. Listen to Wikipedia listens to wikimon for the changes and
  4. Creates a visualizition and audiation from the data

Also uses: D3, Web Audio API

Adapting to a Big Wall

Making this work on the big wall was relatively easy since it is Web-based.

Listen to Wikipedia Languages

  1. Wikidata
  2. English
  3. German
  4. Russian
  5. Japanese
  6. Spanish
  7. French
  8. Dutch
  9. Italian
  1. Swedish
  2. Arabic
  3. Farsi
  4. Hebrew
  5. Indonesian
  6. Assamese
  7. Hindi
  8. Bengali
  1. Punjabi
  2. Telugu
  3. Tamil
  4. Western Mari
  5. Kannada
  6. Oriya
  7. Sanskrit
  8. Gujarati

Turning a language off or on really just turns off or on a websocket connection listening for updates.

But one feature of that I really liked was the ability to turn off and on various languages. I really wanted to replicate this ability to change the languages on the wall.

Remote Controls

So I started thinking about remote controls.

Complicated Remote Controls

Not complicated remote controls like these.

Simple Remote Controls

More like this. Simple, but something that keeps you clicking.

Interact!

As you walk into Hunt you go by the Immersion Theater. Sometimes Listen to Wikipedia will be there advertising that you can interact with it. You can either enter a URL or scan a QR code.

Token-based Authentication

Short-lived 3-letter token: xyz

QR: 80%

 

URL: 20%

http://lib.example.edu/l2w/xyz
Future alternatives to investigate: geolocation, BLE, NFC
Before you snicker at the QR code, note that 80% of the interactions begin with the QR code. The purpose of the token is to keep folks in Australia from controlling the wall in Hunt Library. In the future I may investigate using location based authorization or look into how we might use Bluetooth Low Energy or Near Field Communication to make these kinds of interactions quicker and easier.

Remote Control Interface

On the right is the remote control interface. Touching one of the checkboxes activates or de-activates one of the languages on the wall. The feedback is very quick.

The communication between the remote control and the wall uses a combination of XHR and WebSockets.

Real-Time Experience

Goal: Low latency. Like pressing a button on the wall itself.

One advantage of WebSockets has to do with user experience. The low latency of WebSockets allows for more immediate feedback. If a user clicks a button they want something to happen immediately.

Keeping Clients in Sync

But since more than one person could walk into the immersion theater with their own remote control.

WebSockets were also used to help keep multiple clients in sync with the state of the wall.

Current Changes

I also wanted to provde some way for folks to actually get to the articles they discover and read them. This view uses WebSockets to show the same stream of articles as the wall. You can pause the stream so you can actually click on one of them before it scrolls away.

Wall Remote Architecture

And here's an architectural drawing that probably tells you nothing.

Reasons why I should come to the library more often. #huntlibrary

 

This is people editing--Oh, someone just edited Home Alone--editing Wikipedia in this exact moment.

And the reaction has been positive and enthusiastic.

Unintended Consequences

But there have also been some unintended consequences to this. One time when we were testing this on the wall a very enthusiastic student sat down with their laptop and deleted a large portion of an article just to see the large bubble on the screen. The edit was quickly reverted. Please don't do this.

Vizualization Wall
3840x1518
This is a rather unique array of MicroTiles. It has a picket fence effect.

Digital Collections Explorer

This is just a screencast. On the wall it doesn't loop the animations through so quickly!

The columns are filled with images and then over time scroll down through each of the images. Once the last image is reached it scrolls up to the top again.

Invitation to Take Control

Again a similar message to take control of the wall.

Vizualization Wall Controller

Folks basically get to design their own exhibit. On the left you can see that you can select which column you want new content to show up on and then you can select a topic. Random images and video are returned for the topic and displayed.

On the right you can see how you can also explore and find out more information about the images that are currently on the wall. Find the correct column and you'll see all of the images there right now. Then you can click through to see it.

Chrome Screen Emulation

Chrome screen emulation is great. Usually used for mobile testing the Chrome Developer Tools Screen Emulation panel is great for getting an initial look at what your work will look like on these big walls.

Hunt Tour

Please try these applications out while you're on the Hunt Library tour tonight. I'll be hanging around if you'd like to talk about them.

Conclusion

Now that I have some experience with WebSockets, I'm considering how to use them for things like notifications for when a long-running background job process has completed. Or generally how to make my Web apps faster.

Credits

Jason Ronallo

@ronallo                       

http://ronallo.com        http://github.com/jronallo

Please feel free to contact me if you want to talk more about any of this.

Questions?

If you've been following along you can enter a question there since we have some time.

Bonus Slides!

Places and Spaces

Realtime Searches

But there just aren't that many of them. People come to the digital collections site via Google and may click from there to other resources or into a faceted search, but they don't do a lot of keyword searching.

QuickSearch

QuickSearch does get a lot of search activity. I did one prototype of being able to see the current searches on the site that could have been extended to a dashboard of all the current activity on the site. But we've decided to pass on it for now.

OK, that's what I've done with realtime views of activity. Hopefully you can begin to see how this could be used for all sort of dashboards and activity feeds.

TODO