At IndieWebCamp Berlin this year, at the session about Workflow, we came up with an idea, how to enhance your blogposts with an external service using Micropub. I’ve thought of a few variants, and in spirit of the IndieWeb I should first build them and then show it, but I haven’t got around it yet.

So y’all will have to do with just a description. I might implement it at some point, if I have a real use case for it. I don’t actually want weather on my posts.

But let’s start at an idea I first had at IndieWebCamp Nürnberg.

The Syndication Button Hack

Micropub is an open API standard that allows clients to post to servers. In the spec, there is a mechanism for clients to show buttons for syndication targets. The client asks the server what targets there are, and the server responds with a list of names and UIDs. The client then shows the names on buttons (or near checkboxes) and if the user selects one, the UID is set as the mp-syndicate-to field of the post. The server is then responsible for syndicating the post to, say, Twitter or Facebook.

This mechanism is widely supported among clients. And since the client does not have to do any work actually related to the syndication, it can also be used for other things.

Imagine the server implementing private posts. The support for private posts in Micropub clients is not really existing at the moment of writing. But we can get a button to toggle the state of the post created, quite easily:

GET /micropub?q=syndicate-to
Authorization: Bearer xxxxxxxxx
Accept: application/json

HTTP/1.1 200 OK
Content-type: application/json

{
  "syndicate-to": [
    {
      "uid": "private-post",
      "name": "Private post"
    }
  ]
}

Since it’s up to the server to syndicate to private-post, it can decide not to syndicate it, but to mark it private. There are a number of possibilities with this: toggle audiences, mark the post as draft. All these things could have their own queries at some point, but until then, this will work in almost all the clients.

Also notice the Bearer token. The server can know which client is asking, so it could show a different set of buttons, depending on the client. Quill supports draft posts? Don’t show that button in Quill.

Enter the Weather Service

Back to the idea of Berlin, which takes this one step further. If we have the Syndication Button Hack in place, we can also hook up external services to enhance our blog posts.

Say I display a location with every entry I post. I could have a button that says: ‘Weather Service’. Activating that button would instruct my server to ping the Weather Service about the existance of this new post. This could be done by WebSub or some other mechanism.

Back when I signed up for the Weather Service, I gave it access to my Micropub endpoint as well. The Weather Service waits for new posts to arrive, reads their location, fetches the weather for that location, and sends a Micropub update request.

The only new part this requires, is the button and the ping to the Weather Service. All the other parts exist in clients and servers. Ah, and someone will need to build that Weather Service.

External services in general

The nice thing about this model, is that the heavy lifting is on neither the Micropub client nor the server. It’s on the external service. And it’s not that heavy of a lifting, because the external service does only one thing and does one thing well. It can give superpowers to both Wordpress blogs and static generated sites.

The external service could provide information about the weather, but think of Aaron’s Overland and Compass: it could also provide the location of the post given a point in time. There might be more. Expanding venue info?

One thing to watch out for, is concurrent processing of these Micropub requests. This might not be a problem for you, but I store my posts as flat files. If two services send an update request for the same post, one might start, and the other might overwrite the first one. (I really need to check how my blog handles this case.)

When you are using a database like MySQL, you should be safe for this kind of stuff, but it still depends on the implementation of your Micropub endpoint.

Other ways of doing it

Peter did not like this first approach, because his post would have multiple visible states (first a few seconds without weather, then with it).

Another appreach would be a sort of Russian doll Micropub request, where you sign in to an external service which signs in to your Micropub endpoint. This would mean that quill.p3k.io posts to weather.example/micropub which intercepts the request, and sends the same request with weather info added to seblog.nl/micropub.

I don’t like that approach either, because now I have to trust the Weather Service with my tokens. In the first approach, every service gets their own scoped token, which is safer.

Since the server knows how many services it has asked to enhance the post, it could also keep it in draft until the last update request comes in. This would require more work on the server’s side of things, and there has to be a timeout on it, but it could be a way to mitigate Peter’s problem.

As always: feel free to steal or improve, but please let me know.

Recently I’ve been to IndieWebCamp Berlin, where I spend the Hack Day on abusing the PushAPI to update ServiceWorker caches.

I would like to start with a small section on what and why, but while I was procrastinating on writing this blog post (the pressure is high), no one less than Jeremy Keith wrote a blog post about it. Since that’s a perfect what and why, there are just two things to do for me here: demo and how.

Demo

I did a demo in Berlin, but the demo-gods where unforgiving. It did not work at all, but when I got back to my seat, it started working again. What happened? My Mac tried to be nice and turned off notifications while I was presenting.

But, as to make up for it, the new macOS Mojave shipped with a screen capturing tool. So here is a retry of the demo in under 5 minutes:

The how

This might not be the most interesting part of it, but it’s nice to share work. It’s not a full comprehensive guide on how to do this stuff, because that would just take way too long. See it as a quick guide behind the different API’s involved.

I googled it all anyway. You can google along.

Oh and if you want to skip ahead: there are some use cases at the end.

Showing a local notification

Like with any Javascript, you should check support before you ask something. There is a list of things to ask in the below code example: we want Notifications, it should not be denied, there has to be ServiceWorker support, and for the part later on, there should be a PushManager too.

Once we prompted the user and got permission, it’s as simple as getting our ServiceWorker registration and ask it to show a notification. As you can see: this involves the ServiceWorker, but it does not involve any other servers.

function activateNotifications() {
    Notification.requestPermission()
        .then(status => this.status = status)
},

function supportsNotifications() {
    return ('Notification' in window) && (this.status !== 'denied') &&
        ('serviceWorker' in navigator) && ('PushManager' in window)
}

async function sendTestNotification() {
    const reg = await navigator.serviceWorker.getRegistration()

    return reg.showNotification('Hallo, test!')
}

Note: the demo code is using Vue, which I leave out in this blog post to simplify things. But that’s where this points to: a collection of variables on the Vue instance.

Subscribing for the PushAPI

Once the user clicks the button ‘Subscribe’, the following function gets triggered. In here, we again get the ServiceWorker registration, and then access the PushManager on it, which we tell to subscribe.

async function subscribe() {
    const reg = await navigator.serviceWorker.getRegistration()
    const sub = await reg.pushManager.subscribe({
        userVisibleOnly: true, // required for Chrome
        applicationServerKey: urlB64ToUint8Array(this.publicVapidKey)
    })

    this.notifications = true
    const key = sub.getKey('p256dh')
    const token = sub.getKey('auth')

    await this.$http.post('/subscriptions', {
        endpoint: sub.endpoint,
        key: key ? btoa(String.fromCharCode.apply(null, new Uint8Array(key))) : null,
        token: token ? btoa(String.fromCharCode.apply(null, new Uint8Array(token))) : null
    })

    this.subscribed = sub !== null
},

Some browsers have their own way of doing authentication, but the most universal is with a Vapid key pair. The package I use for the backend came with a way of creating them. We give the public key to the PushManager, which will give us a Subscription object.

In the end, we send the Subscription’s key, token and endpoint to the server via a POST request.

Note: my HTTP library of choice is axios and the urlB64ToUint8Array() function can be found here

Storing the Subscription

For the backend, I’m using a Laravel package for WebPush, which allows me to save the endpoint with very minimal code:

public function update(Request $request)
{
    $this->validate($request, ['endpoint' => 'required']);
    $request->user()->updatePushSubscription(
        $request->endpoint,
        $request->key,
        $request->token
    );
    return response()->json(null, 201);
}

As you can see, it is using the user to associate the data with. (I fake the auth in the demo, which I do not recommend.) It ends up in a database, with four main columns: user_id, endpoint, public_key, auth_token.

In theory, you can go without users, but you will need to store the other parts. The token and key look like random strings, but the endpoint is an actual URL, on a subdomain of either Mozilla or Google, depending on the browser. (No support on Safari yet, mind you.)

These endpoints and tokens can expire, so you will need to keep an eye on the table.

Sending the notification

I can be short about this part: I have no idea. The following code is all it takes to trigger it:

Notification::send(
    User::all(), 
    new NewBlogPostCreated($content, $notify)
);

... where $content is the content of the post, and $notify a boolean, telling my ServiceWorker whether or not to show a notification (we’ll get to that).

The NewBlogPostCreated class extends Laravel’s build-in Notification class and has these two methods:

public function via($notifiable)
{
    return [WebPushChannel::class];
}

public function toWebPush($notifiable, $notification)
{
    return (new WebPushMessage)
        ->title($this->notify ? 'notify' : 'update-cache')
        ->body($this->content);
}

There is a lot of magic behind the scenes here. I have no idea. In the end, they send a POST request to the endpoints of those users, after signing the right things with the right keys.

Receiving the notification and then don’t

Next, we’re back in Javascript-land, however, this is the ServiceWorker-province. The ServiceWorker, once installed, is a script, written in Javascript, but completely decoupled from any window. It lives in your browser and represents not one page, but your whole website.

It’s quite hard to wrap your head around at first, but, I think the PushAPI makes it easier: there is no window involved with a push message, and there is no page involved with a push message. There is only your ServiceWorker, which acts for your whole website.

The ServiceWorker script itself consists of a series of callbacks, that are executed whenever things happen. In the case of a push message, the 'push' event is triggered:

(function() {
    'use strict';

    self.addEventListener('push', function (e) {
        const data = e.data.json()

        self.caches.open('manual')
            .then(cache => cache.put('hello', new Response(data.body)))

        if (data.title == 'notify') {
            e.waitUntil(
                self.registration.showNotification(
                    'New content!', 
                        {body: data.body}
                )
            );
        }
    });

})();

That’s all I need for receiving push notifications. I first retrieve the data from the message. Then I open the cache named ‘manual’ and I put the body of the message in that cache as the content of a URL (in this case ‘offline.test/hello’). It is made for pages, but I use it as a key-value store here.

Then I check the title field, which I have abused for this purpose. If it is set to the magic string ‘notify’, I will trigger the notification. If it’s something else I will do nothing.

This shows that I don’t have to: I can leave the notification out, but I still get a ServiceWorker activation and I can do whatever I want with it.

Use cases

I think this can be used for creepy things (can I occasionally ping my ServiceWorkers and ask for data like ‘how many windows are open?’ and phone that home?), but I also think there are nice uses for this as well.

As Jeremy wrote: this can be used for magazines, podcasts and blogs to push new content to my phone, to read on a plane or in the subway when I’m offline. I see a nice feature for a web-based IndieWeb Reader too: it can push me copies of posts it collected.

I think the Reader is a nice place to use this. With great power comes great responsibility. Do I want to grand that great power to that weird magazine, that dubious podcast, that blog I visit once or twice a month? I might know you well, I might not. Do I trust you, pushing megabytes on my phone without me noticing?

Web apps like a Reader are easier to bond with. Plus: once I know my Reader supports reading offline, I might visit it in the subway. Will I remember the magazine?

The last bonus of the IndieWeb Reader specifically: it can send me posts from any magazine or podcast or blog, whether they support offline reading or not. But that’s more specific to the Reader than it is to Push.

I’m also very curious to know how things will evolve if ServiceWorkers get even more superpowers. How well will those pair with a free ServiceWorker activation? Lot’s of exploring to do!

One of the discussions this weekend in Berlin was on the topic of private feeds. Martijn and Sven made great progress by implemeting a flow to fetch private pages using various endpoints for tokens and authentication.

Apart from the question how to fetch private feeds, there is also the question how to present private feeds. The easiest way is probably to give every user their own feed, containing only the private posts for them. They can separately follow your public feed, and your queries are easier.

But in line with Silo’s like Twitter and Facebook, I think I would prefer presenting one feed, with both public and private posts, scoped for the authenticated user. When I described this to Aaron he said that he liked it, but that he didn’t know where to begin with writing code that does that. I didn’t either, but it made me want to explore the possibilities.

On a sidenote: this feed design also raises another problem, of how to signal to the user that they can see this post but no-one else. I leave that one for another time.

Drawing rough lines around boxes

Borrowing from Facebook, there are roughly four categories you can share content in:

1. public – These posts can be seen by anyone. This is the default on nearly all IndieWeb sites today.
2. authenticated – These posts can only been seen if you sign in, but, anyone can sign in. Facebook has this category and we can mimic that with IndieAuth, but it might not add that much value.
3. friends only – This is a big category on Facebook, and made possible by the friendslist, which is also a big feature on Facebook.
4. selected audience – Facebook also allows you to pick your audience on a per-post basis. This can be done by either selecting individual users, or selecting lists, which can contain users.

There is also the possibility of excluding specific people or lists from posts, but that one is even more advanced, so I put it out of scope for this exploration.

The first category is easy, for we already have it. The second category is harder, but once you got past the authentication it’s easy again. One could query a database for visibility = 'authenticated' OR visibility = 'public', that would work.

The third category would require us to keep a list of friends. The fourth category could also require us to keep lists of people, so it might be better to merge them.

Throw in some tables

This brings us to a simple database schema. I see three main tables: entries, people and groups, with a pivot table between all of them: entry_group, entry_person and group_person. I have chosen ‘people’ over ‘users’, because I might not want to give these people write access to anything, but they could be users as well.

It should work like this:

• Entries have a field for visibility, which can me marked public, authenticated or private.
• People can belong to groups, which have names. Think ‘Friends’, ‘Family’ and ‘Coworkers’.
• Entries can be opened up to individual people, or for a whole group.

There might be better ways of naming these, but I like the simplicity of this model. With private posts and audiences, I will always have to manage some form of lists, and this is the most simple way of doing it.

Enter the monster query

So, with some trial and error, PHPUnit tests, and a lot of Laravel magic I came to the following monster query for these tables:

(
  select `entries`.* 
  from `groups` 
  inner join `group_person` 
    on `groups`.`id` = `group_person`.`group_id` 
  inner join `entry_group` 
    on `groups`.`id` = `entry_group`.`group_id` 
  inner join `entries` 
    on `entries`.`id` = `entry_group`.`entry_id` 
  where `group_person`.`person_id` = ?
) 
union 
(
  select `entries`.* 
  from `entries` 
  inner join `entry_person` 
    on `entries`.`id` = `entry_person`.`entry_id` 
  where `entry_person`.`person_id` = ?
) 
union 
(
  select * 
  from `entries` 
  where `visibility` = 'public'
) 
order by `published_at` desc

... which is way shorter when expressed in with Laravel’s Eloquent:

class Person extends Model
{
    public function timeline()
    {
        return $this->groups()
            ->join('entry_group', 'groups.id', '=', 'entry_group.group_id')
            ->join('entries', 'entries.id', '=', 'entry_group.entry_id')
            ->select('entries.*')
            ->union($this->entries()->select('entries.*'))
            ->union(Entry::whereVisibility('public'))
            ->orderBy('published_at', 'desc');
    }

    public function groups()
    {
        return $this->belongsToMany(Group::class);
    }

    public function entries()
    {
        return $this->belongsToMany(Entry::class);
    }
}

Since the method timeline() returns the Query object, other where-clauses can be appended when needed.

I am in a bit of a fight with Laravel still, for it adds
'`group_person`.`person_id` as `pivot_person_id`, `group_person`.`group_id` as `pivot_group_id`' to the first query, which makes it blow up, but the raw query works!

There is possibly a better way of doing it, but this is a start! Feel free to steal or improve, but if you improve, let me know.