Guests, patchXHR, and other updates Grimwire/Local.js 0.5

@pfrazee on Nov 19, 2013 link

Local.js and Grimwire have been tagged "v0.5 beta" on GitHub today. They are (respectively) an Ajax messaging library and a node.js signalling relay. Together, they broadcast Web services from the page over WebRTC.

This version's updates include API refinements and additions, bugfixes, and some optimizations (including much faster async calls in promises and ajax dispatching).


Guest Users

The biggest addition with 0.5 is an experimental form of guest accounts, which enables users to get streams on the relay without registering first. Instead, an existing account-holder allocates some of their streams for guests to take under their name.

Other users can then specify that account as the host during login.

They'll be assigned ids based on the host user, eg "bob-guest123". Guests work exactly as normal users, except they can't update their accounts or persist past their 24-hour session. You can see an example of their usage in GrimChat, which provides a link for inviting guests when a room is created.

Currently there's no way to restrict guest access to specific users (for instance, by issuing "guest tokens" or setting passwords) so only allocate guest slots when you plan to use them.


patchXHR()

Another convenience update is the new local.patchXHR() function which updates XMLHttpRequest to support HTTPL. This is useful for libraries which rely on XHR and can't easily be ported to use Local.js' dispatcher.

local.patchXHR();
var xhr = new XMLHttpRequest();
xhr.open('GET', 'httpl://myserver');
xhr.onreadystatechange = function() {
    if (xhr.readyState == 4) {
        console.log(xhr.getResponseHeader('content-type')); // => 'application/json'
        console.log(xhr.response); // => { foo: 'bar' }
    }
};
xhr.send();

Other Updates

You can read the rest of the changes in the Local.js release and the Grimwire release. If this is your first time seeing Grimwire, check out this post on building applications and the Local.js documentation. Future beta versions will continue to focus on bugfixing and API improvements, and will particularly focus on limiting access to other users as those use-cases become clearer.





P2P Ajax and Server-Sent Events in practice with GrimChat

@pfrazee on Nov 19, 2013 link

This article is an introduction to Grimwire, a messaging system for WebRTC that uses Ajax and service-discovery to create a cloud of user-hosted P2P apps. Learn more at grimwire.com.


When evaluating a framework, one of the first questions I ask is, "how does it work in real applications?" As a developer, it helps me build a full picture of the toolset, and answers whether the framework really is cohesive and complete. So as you're evaluating Grimwire, you must be wondering, "how does it work in the real world?" Let's explore that with GrimChat.

GrimChat is a "moderator-hosts" program. Whoever creates the room becomes the server that everyone connects to: they accept all messages, then redistribute them in a star topology. The room members don't connect to each other – just to the room host. The full source is at github.com/grimwire/chat.


Step 1 - Getting Connected with GrimWidget

The nature of Grimwire is that there's no single network for arranging WebRTC connections. Instead, the "relay" servers can be downloaded and hosted by anybody, much like IRC. This means users will need to input their choice of network before getting started. GrimWidget.js is a simple login and configuration UI to do that.

// Create Grimwidget
ui.grimWidget = grimwidget.create({
    triggerEl: document.querySelector('#conn-status a'),
    halign: 'right',
    render: renderGrimWidget
});

Once the user logs in and a relay connection is made, the render function kicks in. The function is called when the GrimWidget is opened and is given a target element and an array of links from the relay. We'll use them here to render active rooms.

The links come from a registry on the relay which apps populate to find each other. We'll add our own links later in this article.

function renderGrimWidget(el, links) {
    if (CHATAPP.mode == CHATAPP.OFFLINE) {
        // Render room list from available links
        links = local.queryLinks(links, { rel: 'chat.grimwire.com/room' });
        if (links.length) {
            // Remove duplicates
            var added = {};
            links = links.filter(function(link) { var isfirst = !added[link.href]; added[link.href] = true; return isfirst; });
        }
        el.innerHTML = renderRoomList({ links: links });
    } else {
        // Render room view
        el.innerHTML = renderRoomView({ roomHost: CHATAPP.roomHost, roomName: CHATAPP.roomName, peers: CHATAPP.peersList });
    }
}

We look for links that include the chat.grimwire.com/room reltype because that indicates support for our chat protocol. Link reltypes are a standard feature of links – in fact, you should recognize them from stylesheet importing (<link rel="stylesheet" href="mystyles.css"/>).

This is how the links are rendered (using Handlebars):

<p><strong>Join a room:</strong></p>
<form action="httpl://app.local" method="JOIN">
  {{#each links}}
    <p>
      <button class="btn btn-primary btn-xs" name="link" value="{{json this}}" title="{{this.user}}/{{this.room}}">Join</button>
      {{this.host_user}}/{{this.room}}
    </p>
  {{else}}
    <p>No rooms online.</p>
  {{/each}}
  <p><button class="btn btn-success btn-xs" formmethod="HOST">Start a room</button></p>
</form>

The attributes on the links are either mandated by the reltype (as in the case of room) or extracted from the href (as in the case of host_* attributes). Notice that the form sends its request to a local server, httpl://app.local. Let's look at how that works next.


Step 2 - Handling UI Events (as Requests)

Local.js allows applications to register Javascript functions as "local" servers. This is the basis for most of Grimwire's application messaging, as the functions can then transport the requests to Web Workers and WebRTC peers. However, there are some cases where purely document-local servers (that is, servers that aren't exposed to peers or workers) can provide some use.

Local.js includes a utility function, bindRequestEvents, which captures unhandled "click" and "submit" events and emits "request" events in their place. This allows applications to host document-local servers for links/forms to target instead of binding individual event handlers.

// Setup DOM request-event handling
local.bindRequestEvents(document.body);
document.body.addEventListener('request', function(e) { local.dispatch(e.detail); });

Now, when the "Join" and "Start a room" buttons in GrimWidget are clicked, they'll emit requests which are dispatched. Let's look at how those requests are handled:

// UI behaviors
local.addServer('app.local', function(req, res) {
    switch (req.method) {
    case 'JOIN':
        // Finish stream
        req.on('end', function() {
            var link;
            try { link = JSON.parse(req.body.link); }
            catch (e) {
                console.error(e);
                return res.writeHead(422, 'bad ent').end();
            }
            // Connect
            net.joinRoom(link.room, link.host_user, link.href);
            ui.grimWidget.refresh();
            res.writeHead(204).end();
        });
        break;

    case 'HOST':
        // Get room name
        var roomName = prompt('Room name:');
        if (!roomName) {
            if (roomName === '')
                alert('You must specify a room name');
            res.writeHead(400).end();
        } else {
            // Start server
            net.startServer(roomName, relay.getUserId());
            ui.grimWidget.refresh();
            res.writeHead(204).end();
        }
        break;

    // ...
});

Node.js developers should find this fairly familiar. Requests and responses are streams, and so the request "end" event must fire before the request body is available. Unlike node, Local.js includes content-type parsing and encoding so that familiar media is immediately available in the body. By default it supports json, x-www-form-urlencoded, and event-streams, and you can add more types with local.contentTypes.register().

It may feel strange to do UI work inside the server, since separation from clients is a defining attribute of Web servers. If you want, you can move "client-side" tasks into a response handler and interpret the status codes to do UI updates. In GrimChat, however, that's not necessary. Not only is "app.local" unexposed to other clients, but conceptually it wraps the state of the app, making it an appropriate place for UI work. Of course, the decision in your apps is up to you.


Before we leave this section, let's look at GrimChat's one other event-handler, as it includes a 'gotcha' that's worth recognizing:

// User chat controls
$('#chatin').on('keypress', function(e) {
    if (!e.shiftKey && e.keyCode == 13) {
        // Trigger the submit event
        // :NOTE: directly calling the form's submit() function circumvents the 'submit' event, which local needs to intercept
        //        see http://jsfiddle.net/danmana/afznK/
        document.getElementById('chatsubmit').click();
        return false;
    }
});

If you were to call form.submit(), the browser would not emit the "submit" event. Instead, it would jump directly to the submission & page navigation logic. As a result, to trigger a "request" event from a form, you need to fire a click event on one of it's buttons.


Step 3 - Joining a Room

On room join, we have 3 tasks to handle: 1 Fetch the room state, 2 Acquire a stream of room events, and 3 Publish a link to the relay so other users can find us here. Here's how that's done:

net.joinRoom = function(roomName, roomHost, roomUri) {
    // Update app state
    CHATAPP.mode = CHATAPP.CLIENT;
    CHATAPP.roomName = roomName;
    CHATAPP.roomHost = roomHost;
    CHATAPP.roomUri  = roomUri;
    CHATAPP.roomApi  = local.agent(roomUri);
    ui.setModeLine(roomHost+'/'+roomName);

    // Get room info 1
    CHATAPP.roomApi.get({ Accept: 'application/json' })
        .then(function(res) {
            // Update state
            CHATAPP.peersList = res.body.users;
            ui.grimWidget.refresh();
        }, function(res) {
            ui.renderToChat('danger', '', 'Failed to get chat room info. Try refreshing and connecting again.');
        });

    // Connect to the room's event-stream 2
    ui.renderToChat('info', '', 'Connecting '+roomHost+'/'+roomName+'....');
    net.subscribeToRoom();

    // Publish room link 3
    grimwidget.getRelay().registerLinks([{
        href: roomUri,
        rel: 'service chat.grimwire.com/room',
        title: 'Chat: Connected to '+roomHost+'/'+roomName,
        room: roomName
    }]);
};

CHATAPP.roomApi will be our primary interface to the room. We create it at the top of the function by calling local.agent() with the room's URL. The .get() call results in a Ajax GET request to the room host. To send a chat statement, we use .post():

CHATAPP.roomApi.post('Hello, room') // defaults to text/plain
    .fail(function(res) {
        if (res.status == 429) {
            ui.renderToChat('danger', '', 'Anti-spamming rate limit hit. Please wait for your usage counter to reset.');
        }
    });

Take a moment to read about agents if you haven't already.

The call to relay.registerLinks() puts a link to the room on the relay's index. If you're logged into a network and using GrimChat, check your relay's dashboard to see your current links. The relay adds a relay_user attribute so that queries can filter by the registering user. For instance, to list links from "bob" I'd run:

var relayIndex = relay.agent(); // spawn an agent pointing to the relay's index
relayIndex.head().then(function() { // issue a HEAD request to get the latest links
    local.queryLinks(relayIndex.links, { relay_user: 'bob' }); // => [{ href: ... }]
});

The room's event-stream is acquired with the SSE protocol in subscribeToRoom():

net.subscribeToRoom = function() {
    var roomEvents = local.subscribe(CHATAPP.roomUri);
    roomEvents.on('chat', function(e) {
        // Render message
        var labelColor = (e.data.user == relay.getUserId()) ? 'success' : 'primary';
        ui.renderToChatSafe(labelColor, ''+e.data.user+'', e.data.statement);
    });
    roomEvents.on('join', function(e) {
        // Render event
        ui.renderToChat('default', ''+e.data.user+'', 'joined ('+e.data.app+')');

        // Add to users list
        if (CHATAPP.mode == CHATAPP.CLIENT && CHATAPP.peersList) {
            CHATAPP.peersList[e.data.domain] = e.data;
        }
        ui.grimWidget.refresh();
    });
    roomEvents.on('part', function(e) {
        // Render event
        ui.renderToChat('default', ''+e.data.user+'', 'left ('+e.data.app+')');

        // Remove from users list
        if (CHATAPP.mode == CHATAPP.CLIENT && CHATAPP.peersList) {
            delete CHATAPP.peersList[e.data.domain];
        }
        ui.grimWidget.refresh();
    });
    roomEvents.on('close', function(e) {
        if (CHATAPP.mode == CHATAPP.CLIENT) {
            // Update state
            CHATAPP.mode = CHATAPP.DEAD;
            CHATAPP.peersList = {};

            // Remove links
            relay.registerLinks([]);

            // Notify user
            ui.setModeLine('Connection Closed');
            ui.renderToChat('danger', '', 'Connection Closed. You are no longer connected to the room host.');
        }
    });
    roomEvents.on('error', function(e) {
        console.error('Chat Event Stream Error', e);
    });
};

The code above should be fairly intuitive. SSE Event-streams are responses to Ajax requests which are left open indefinitely. When the server wishes to broadcast, it writes a new chunk which the clients can decode and emit within their code.

Since the GrimChat is peer-to-peer, it is possible for the room host to send Ajax requests to its participants on new events. This would have been perfectly valid, though it would incur some extra overhead compared to an event-stream.


Step 4 - Hosting the Room

If the user were to choose to be a host, they would be prompted to select a name, then this function would be called:

net.startServer = function(roomName, roomHost) {
    // Update app state
    CHATAPP.mode = CHATAPP.SERVER;
    CHATAPP.roomName = roomName;
    CHATAPP.roomHost = roomHost;
    CHATAPP.roomUri  = 'httpl://'+relay.getAssignedDomain();
    CHATAPP.roomApi  = local.agent(CHATAPP.roomUri);
    ui.setModeLine('Hosting '+roomHost+'/'+roomName);

    // Publish room link
    relay.registerLinks([{
        href: '/',
        rel: 'service chat.grimwire.com/room',
        title: 'Chat: Hosting '+roomHost+'/'+roomName,
        room: roomName
    }]);

    // Connect to the room's event-stream
    net.subscribeToRoom();
};

This looks like a stripped-down version of joinRoom(), which it is: it goes through the same process, except that it doesn't have to fetch the room state. The host subscribes to its own event-stream to update it's UI, which Local.js handles by looping back to the server function. Notice also that the registered link uses a relative path – Grimwire automatically resolves that to the app's absolute URI before adding it to the registry.

The final piece of the puzzle is the peer-server function. This function is actually registered by both room hosts and room participants – both can receive Ajax requests – but only the room hosts make use of it. If an application will never handle incoming requests, it can choose not to set a peer server, and Local.js will automatically respond 501 Not Implemented.

var roomEvents = new local.EventHost();
grimwidget.getRelay().setServer(function (req, res, peer) {
    if (CHATAPP.mode != CHATAPP.SERVER) {
        // Not a room host - Just serve current links
        if (CHATAPP.mode == CHATAPP.CLIENT) {
            var title = 'Chat: Connected to '+CHATAPP.roomHost+'/'+CHATAPP.roomName;
            res.setHeader('link', [
                { href: '/', rel: 'self service chat.grimwire.com/member', title: title, room: CHATAPP.roomName },
                { href: roomUri, rel: 'service chat.grimwire.com/room', title: title, room: CHATAPP.roomName }
            ]);
        } else {
            res.setHeader('link', [{ href: '/', rel: 'self service', title: 'Chat: Not connected to any rooms' }]);
        }
        return res.writeHead(204, 'no content').end();
    }

    // Room host
    var title = 'Chat: Hosting '+CHATAPP.roomHost+'/'+CHATAPP.roomName;
    res.setHeader('link', [{ href: '/', rel: 'self service chat.grimwire.com/room', title: title, room: CHATAPP.roomName }]);
    switch (req.method) {
    case 'HEAD':
        res.writeHead(204).end();
        break;

    case 'GET':
        if (!local.preferredType(req, 'application/json')) {
            return res.writeHead(406, 'bad accept: only provides json').end();
        }
        res.writeHead(200, 'ok', { 'content-type': 'application/json' });
        res.end({ roomName: CHATAPP.roomName, roomHost: CHATAPP.roomHost, users: CHATAPP.peersList });
        break;

    case 'SUBSCRIBE':
        // Content negotiation
        if (!local.preferredType(req, 'text/event-stream')) {
            return res.writeHead(406, 'bad accept: only provides text/event-stream').end();
        }

        // Setup peer entry
        var peerInfo = JSON.parse(JSON.stringify(peer.getPeerInfo()));
        peerInfo.domain = peer.getDomain();
        peerInfo.ishost = (peerInfo.domain == local.parseUri(CHATAPP.roomUri).authority);
        if (peerInfo.domain in CHATAPP.peersList) {
            return res.writeHead(409, 'you may only open one event-stream per peer host').end();
        }

        // Setup the response stream
        res.writeHead(200, 'ok', { 'content-type': 'text/event-stream' });
        roomEvents.addStream(res);

        // Emit join
        roomEvents.emit('join', peerInfo);
        CHATAPP.peersList[peerInfo.domain] = peerInfo;
        res.on('end', function() {
            // Emit part
            roomEvents.emit('part', peerInfo);
            delete CHATAPP.peersList[peerInfo.domain];
        });
        break;

    case 'POST':
        if (req.headers['content-type'] != 'text/plain') {
            return res.writeHead(415, 'content-type must be text/plain').end();
        }

        if (!(peer.getDomain() in CHATAPP.peersList)) {
            return res.writeHead(403, 'forbidden: must hold a stream in the room to send messages').end();
        }

        req.on('end', function() {
            if (req.body.length > 255) {
                req.body = req.body.slice(0,255);
            }
            roomEvents.emit('chat', { user: peer.getPeerInfo().user, statement: req.body });
            res.writeHead(204).end();
        });
        break;

    default:
        res.writeHead(405, 'bad method').end();
    }
});

Local.js includes the EventHost to manage SSE broadcasts. It keeps a list of streams receiving events and automatically removes them as they close.

Peer server functions are given an extra third parameter, peer, which is the RTCBridgeServer instance that wraps the WebRTC connection. Because the RTC connection is arranged by the relay, and the relay requires authentication, applications can trust the identity information provided in getPeerInfo(). In GrimChat, this is used to conveniently assign usernames and know the source of a message.

The life-time of the event-stream is used to decide user presence, which makes sense because the user needs an active stream to see what's happening. The stream's initialization is used to trigger a "join" broadcast, and its end event is used to trigger a "part". To keep the server simple, each peer is only allowed to acquire a single event-stream. (They shouldn't need more.)

Responses include 'link' headers which mimic the links sent to the relay. This is so that agents can navigate GrimChat's Web API and reason about its state. For instance, if an agent were to navigate to a 'client-mode' GrimChat instance (identified by chat.grimwire.com/member) it could find the link to the room host and continue navigating.


Wrapping Up

Despite the P2P networking, GrimChat should be immediately familiar to Web developers because of its client/host architecture. In applications which create stable network configurations (like this chat room, or a document host, or many others) the model is convenient and easy to reason about. Unlike traditional Web services, however, the nodes acting as servers can easily change. Should the host drop, for instance, other users could connect horizontally to elect a new leader.

Ajax and Server-Sent Events make a strong toolset for passing messages and reacting to updates in real-time. These tools can be used for P2P, for leveraging multi-threading with Web Workers, or for simply modularizing your application as an SOA. Additionally, the peer discovery through programmatic agents and link queries makes configuration between unfamiliar nodes simple and painless.

I hope this has given you a good picture of how Grimwire and Local.js function together in applications. Continue reading the Local.js documentation or join us at #grimwire on freenode to learn more and find help on issues.





P2P Ajax over WebRTC Grimwire/Local.js 0.4

@pfrazee on Nov 6, 2013 link

Local.js and Grimwire have been tagged 0.4 on GitHub today. They are (respectively) an Ajax messaging library and a node.js signalling relay. Together, they broadcast Web services from the page over WebRTC.

Local.js includes third-party libraries by Steve Levithan (parseUri) Franz Antesberger (URI Templates) and Federico Romero (Negotiator).


How it works

The Local.js dispatcher adds a new scheme, httpl://, for requests to target javascript functions. It uses an in-process emulation of HTTP with the same semantics and a stream-based interface. The registered server functions then behave as Web hosts, and (with some utility functions) can even be targeted by links and forms.

local.addServer('myfunc', function(req, res) {
    res.writeHead(200, 'ok', { 'content-type': 'text/plain' });
    res.end('Hello, world!');
});
local.dispatch({ method: 'GET', url: 'httpl://myfunc' })
    .then(function(res) {
        console.log(res.body); // => "Hello, world!"
    });

Certain types of server functions, "bridge" servers, serialize the streams into JSON and transport them over reliable channels. This is how WebRTC peers host to each other, and is also used to run servers in Workers. After establishing a connection to a relay host, the namespace of peer URIs under the relay (httpl://user@relayDomain!appDomain:streamId/) are watched for requests. New messages to those peers cause the RTC bridge server to auto-create and establish a connection, then continue exchanging requests over the DataChannel.

Grimwire, the node.js relay, hosts event-streams to actively-broadcasting pages. The pages use Grimwire to bounce WebRTC session information to each other, and will also bounce HTTPL traffic if WebRTC fails. Once the DataChannel is open, all traffic travels directly betwen peers using encrypted SCTP streams. The connection and traffic fallback processes are automated by Local.js so applications can send requests to peer URIs and ignore the underlying transport.

Connecting over a relay looks like this:

// Get access to the relay
var relay = local.joinRelay('https://grimwire.net', peerServerFn);
relay.requestAccessToken(); // this will prompt the user to authorize the app
relay.on('accessGranted', function() {
    relay.startListening();
});

// Serve peers
function peerServerFn(req, res, peer) {
    res.writeHead(200, 'ok', { 'content-type': 'text/plain' })
    res.end('Hello, '+peer.getPeerInfo().user);
}

// Contact peers on the relay
local.dispatch({ method: 'GET', url: 'httpl://bob@grimwire.net!bobs-app.com' })
    .then(/* ... */);

Since the user should need to set their relay host, Grimwire includes GrimWidget.js, which changes the process to look like this:

// Add the widget to the page
grimwidget.create({
    triggerEl: document.querySelector('#widget-btn')
});

// Set the server function on the widget's relay instance
grimwidget.getRelay().setServer(function(req, res, peer) {
    res.writeHead(200, 'ok', { 'content-type': 'text/plain' })
    res.end('Hello, '+peer.getPeerInfo().user);
});

SOA in the Browser

Conceptually, Grimwire & Local.js are designed to unify Javascript components and remote hosts under a Service-Oriented Architecture. The purpose is dynamic reconfiguration: messaging divides the application into loosely-bound components which can be changed at runtime, particularly by the user. Using WebRTC (or SharedWorkers) as a transport, pages in the browser can co-depend for vital services as they would on traditional Web hosts. However, the SOA has to be configured for interoperation, which means the components need to find and reason about each other.

To help that process, Local.js implements a service-discovery protocol based on RFC 5988 – a standard for placing typed links in response headers. Local's Agent object searches response links with queries to programmatically navigate the SOA, using the types (and other attributes) to reason about the APIs. Well-documented relation types can then be used to guarantee behaviors; for example, see Grimwire's Web API documentation.

local.agent('http://myhost.com')
  .follow({ rel: 'gwr.io/user item', id: 'bob' })
  .patch({ avatar: 'cowboy' });

To discover services running on the users' browsers, Grimwire maintains a registry of links added by the peers. The agent and link-query helpers use the registry to find other active applications and automatically connect (or give the user some choices, as appropriate).


Beta Status

Grimwire and Local.js are not feature-complete. Grimwire is missing access controls for selecting who can view and signal to streams. It is also not able to connect to users at other relays; that will require inter-relay signalling or a means to grant temporary streams to guests. Local.js lacks binary support, needs smarter streams (backpressure, flood protection) and has lots of open opportunities to optimize. Both APIs should be considered unstable.

Please don't deploy anything critical with this software until it's published as "safe-to-deploy." That will be a specific release label. There is a lot of vulnerability safe-guarding to do, and it's trivial to open the console and flood a peer with Ajax calls.

File any issues at the Local.js tracker (client-side issues) or Grimwire tracker (relay issues).


Getting Started

Grimwire (repo) and Local.js (repo) are available under the MIT license. Grimwire is simple to administer and has no external dependencies other than node.js; start by reading the Download documentation. Local.js requires Grimwire for WebRTC connections, but can be used independently if WebRTC is not needed. A good place for developers to start is with the documentation and TodoSOA example. A chat application is also available to help familiarize with Grimwire applications.