Recently I’ve been to IndieWebCamp Berlin, where I spend the Hack Day on abusing the PushAPI to update ServiceWorker caches.

I would like to start with a small section on what and why, but while I was procrastinating on writing this blog post (the pressure is high), no one less than Jeremy Keith wrote a blog post about it. Since that’s a perfect what and why, there are just two things to do for me here: demo and how.

Demo

I did a demo in Berlin, but the demo-gods where unforgiving. It did not work at all, but when I got back to my seat, it started working again. What happened? My Mac tried to be nice and turned off notifications while I was presenting.

But, as to make up for it, the new macOS Mojave shipped with a screen capturing tool. So here is a retry of the demo in under 5 minutes:

The how

This might not be the most interesting part of it, but it’s nice to share work. It’s not a full comprehensive guide on how to do this stuff, because that would just take way too long. See it as a quick guide behind the different API’s involved.

I googled it all anyway. You can google along.

Oh and if you want to skip ahead: there are some use cases at the end.

Showing a local notification

Like with any Javascript, you should check support before you ask something. There is a list of things to ask in the below code example: we want Notifications, it should not be denied, there has to be ServiceWorker support, and for the part later on, there should be a PushManager too.

Once we prompted the user and got permission, it’s as simple as getting our ServiceWorker registration and ask it to show a notification. As you can see: this involves the ServiceWorker, but it does not involve any other servers.

function activateNotifications() {
    Notification.requestPermission()
        .then(status => this.status = status)
},

function supportsNotifications() {
    return ('Notification' in window) && (this.status !== 'denied') &&
        ('serviceWorker' in navigator) && ('PushManager' in window)
}

async function sendTestNotification() {
    const reg = await navigator.serviceWorker.getRegistration()

    return reg.showNotification('Hallo, test!')
}

Note: the demo code is using Vue, which I leave out in this blog post to simplify things. But that’s where this points to: a collection of variables on the Vue instance.

Subscribing for the PushAPI

Once the user clicks the button ‘Subscribe’, the following function gets triggered. In here, we again get the ServiceWorker registration, and then access the PushManager on it, which we tell to subscribe.

async function subscribe() {
    const reg = await navigator.serviceWorker.getRegistration()
    const sub = await reg.pushManager.subscribe({
        userVisibleOnly: true, // required for Chrome
        applicationServerKey: urlB64ToUint8Array(this.publicVapidKey)
    })

    this.notifications = true
    const key = sub.getKey('p256dh')
    const token = sub.getKey('auth')

    await this.$http.post('/subscriptions', {
        endpoint: sub.endpoint,
        key: key ? btoa(String.fromCharCode.apply(null, new Uint8Array(key))) : null,
        token: token ? btoa(String.fromCharCode.apply(null, new Uint8Array(token))) : null
    })

    this.subscribed = sub !== null
},

Some browsers have their own way of doing authentication, but the most universal is with a Vapid key pair. The package I use for the backend came with a way of creating them. We give the public key to the PushManager, which will give us a Subscription object.

In the end, we send the Subscription’s key, token and endpoint to the server via a POST request.

Note: my HTTP library of choice is axios and the urlB64ToUint8Array() function can be found here

Storing the Subscription

For the backend, I’m using a Laravel package for WebPush, which allows me to save the endpoint with very minimal code:

public function update(Request $request)
{
    $this->validate($request, ['endpoint' => 'required']);
    $request->user()->updatePushSubscription(
        $request->endpoint,
        $request->key,
        $request->token
    );
    return response()->json(null, 201);
}

As you can see, it is using the user to associate the data with. (I fake the auth in the demo, which I do not recommend.) It ends up in a database, with four main columns: user_id, endpoint, public_key, auth_token.

In theory, you can go without users, but you will need to store the other parts. The token and key look like random strings, but the endpoint is an actual URL, on a subdomain of either Mozilla or Google, depending on the browser. (No support on Safari yet, mind you.)

These endpoints and tokens can expire, so you will need to keep an eye on the table.

Sending the notification

I can be short about this part: I have no idea. The following code is all it takes to trigger it:

Notification::send(
    User::all(), 
    new NewBlogPostCreated($content, $notify)
);

... where $content is the content of the post, and $notify a boolean, telling my ServiceWorker whether or not to show a notification (we’ll get to that).

The NewBlogPostCreated class extends Laravel’s build-in Notification class and has these two methods:

public function via($notifiable)
{
    return [WebPushChannel::class];
}

public function toWebPush($notifiable, $notification)
{
    return (new WebPushMessage)
        ->title($this->notify ? 'notify' : 'update-cache')
        ->body($this->content);
}

There is a lot of magic behind the scenes here. I have no idea. In the end, they send a POST request to the endpoints of those users, after signing the right things with the right keys.

Receiving the notification and then don’t

Next, we’re back in Javascript-land, however, this is the ServiceWorker-province. The ServiceWorker, once installed, is a script, written in Javascript, but completely decoupled from any window. It lives in your browser and represents not one page, but your whole website.

It’s quite hard to wrap your head around at first, but, I think the PushAPI makes it easier: there is no window involved with a push message, and there is no page involved with a push message. There is only your ServiceWorker, which acts for your whole website.

The ServiceWorker script itself consists of a series of callbacks, that are executed whenever things happen. In the case of a push message, the 'push' event is triggered:

(function() {
    'use strict';

    self.addEventListener('push', function (e) {
        const data = e.data.json()

        self.caches.open('manual')
            .then(cache => cache.put('hello', new Response(data.body)))

        if (data.title == 'notify') {
            e.waitUntil(
                self.registration.showNotification(
                    'New content!', 
                        {body: data.body}
                )
            );
        }
    });

})();

That’s all I need for receiving push notifications. I first retrieve the data from the message. Then I open the cache named ‘manual’ and I put the body of the message in that cache as the content of a URL (in this case ‘offline.test/hello’). It is made for pages, but I use it as a key-value store here.

Then I check the title field, which I have abused for this purpose. If it is set to the magic string ‘notify’, I will trigger the notification. If it’s something else I will do nothing.

This shows that I don’t have to: I can leave the notification out, but I still get a ServiceWorker activation and I can do whatever I want with it.

Use cases

I think this can be used for creepy things (can I occasionally ping my ServiceWorkers and ask for data like ‘how many windows are open?’ and phone that home?), but I also think there are nice uses for this as well.

As Jeremy wrote: this can be used for magazines, podcasts and blogs to push new content to my phone, to read on a plane or in the subway when I’m offline. I see a nice feature for a web-based IndieWeb Reader too: it can push me copies of posts it collected.

I think the Reader is a nice place to use this. With great power comes great responsibility. Do I want to grand that great power to that weird magazine, that dubious podcast, that blog I visit once or twice a month? I might know you well, I might not. Do I trust you, pushing megabytes on my phone without me noticing?

Web apps like a Reader are easier to bond with. Plus: once I know my Reader supports reading offline, I might visit it in the subway. Will I remember the magazine?

The last bonus of the IndieWeb Reader specifically: it can send me posts from any magazine or podcast or blog, whether they support offline reading or not. But that’s more specific to the Reader than it is to Push.

I’m also very curious to know how things will evolve if ServiceWorkers get even more superpowers. How well will those pair with a free ServiceWorker activation? Lot’s of exploring to do!

One of the discussions this weekend in Berlin was on the topic of private feeds. Martijn and Sven made great progress by implemeting a flow to fetch private pages using various endpoints for tokens and authentication.

Apart from the question how to fetch private feeds, there is also the question how to present private feeds. The easiest way is probably to give every user their own feed, containing only the private posts for them. They can separately follow your public feed, and your queries are easier.

But in line with Silo’s like Twitter and Facebook, I think I would prefer presenting one feed, with both public and private posts, scoped for the authenticated user. When I described this to Aaron he said that he liked it, but that he didn’t know where to begin with writing code that does that. I didn’t either, but it made me want to explore the possibilities.

On a sidenote: this feed design also raises another problem, of how to signal to the user that they can see this post but no-one else. I leave that one for another time.

Drawing rough lines around boxes

Borrowing from Facebook, there are roughly four categories you can share content in:

1. public – These posts can be seen by anyone. This is the default on nearly all IndieWeb sites today.
2. authenticated – These posts can only been seen if you sign in, but, anyone can sign in. Facebook has this category and we can mimic that with IndieAuth, but it might not add that much value.
3. friends only – This is a big category on Facebook, and made possible by the friendslist, which is also a big feature on Facebook.
4. selected audience – Facebook also allows you to pick your audience on a per-post basis. This can be done by either selecting individual users, or selecting lists, which can contain users.

There is also the possibility of excluding specific people or lists from posts, but that one is even more advanced, so I put it out of scope for this exploration.

The first category is easy, for we already have it. The second category is harder, but once you got past the authentication it’s easy again. One could query a database for visibility = 'authenticated' OR visibility = 'public', that would work.

The third category would require us to keep a list of friends. The fourth category could also require us to keep lists of people, so it might be better to merge them.

Throw in some tables

This brings us to a simple database schema. I see three main tables: entries, people and groups, with a pivot table between all of them: entry_group, entry_person and group_person. I have chosen ‘people’ over ‘users’, because I might not want to give these people write access to anything, but they could be users as well.

It should work like this:

• Entries have a field for visibility, which can me marked public, authenticated or private.
• People can belong to groups, which have names. Think ‘Friends’, ‘Family’ and ‘Coworkers’.
• Entries can be opened up to individual people, or for a whole group.

There might be better ways of naming these, but I like the simplicity of this model. With private posts and audiences, I will always have to manage some form of lists, and this is the most simple way of doing it.

Enter the monster query

So, with some trial and error, PHPUnit tests, and a lot of Laravel magic I came to the following monster query for these tables:

(
  select `entries`.* 
  from `groups` 
  inner join `group_person` 
    on `groups`.`id` = `group_person`.`group_id` 
  inner join `entry_group` 
    on `groups`.`id` = `entry_group`.`group_id` 
  inner join `entries` 
    on `entries`.`id` = `entry_group`.`entry_id` 
  where `group_person`.`person_id` = ?
) 
union 
(
  select `entries`.* 
  from `entries` 
  inner join `entry_person` 
    on `entries`.`id` = `entry_person`.`entry_id` 
  where `entry_person`.`person_id` = ?
) 
union 
(
  select * 
  from `entries` 
  where `visibility` = 'public'
) 
order by `published_at` desc

... which is way shorter when expressed in with Laravel’s Eloquent:

class Person extends Model
{
    public function timeline()
    {
        return $this->groups()
            ->join('entry_group', 'groups.id', '=', 'entry_group.group_id')
            ->join('entries', 'entries.id', '=', 'entry_group.entry_id')
            ->select('entries.*')
            ->union($this->entries()->select('entries.*'))
            ->union(Entry::whereVisibility('public'))
            ->orderBy('published_at', 'desc');
    }

    public function groups()
    {
        return $this->belongsToMany(Group::class);
    }

    public function entries()
    {
        return $this->belongsToMany(Entry::class);
    }
}

Since the method timeline() returns the Query object, other where-clauses can be appended when needed.

I am in a bit of a fight with Laravel still, for it adds
'`group_person`.`person_id` as `pivot_person_id`, `group_person`.`group_id` as `pivot_group_id`' to the first query, which makes it blow up, but the raw query works!

There is possibly a better way of doing it, but this is a start! Feel free to steal or improve, but if you improve, let me know.