So I woke up this morning and thought: I should really dig more into using Git within Vim with tpope's vim-fugitive. So I did.
I was already using the :Gblame command it provides a lot. This command opens a vertical split, with on the right the current file and on the left for each line of that file the hash, author and date of the last time that line changed. This is great for quickly identifying the author and age of a piece of code, which helps me a great deal in understanding the purpose behind the code. It's sad that Git gives this feature a name with such a negative connotation. PHPStorm calls this 'Git annotate', which feels nicer. One can also do a git blame from the command line or on Github, but having this command right in your editor is very useful.
I noticed, however, that I can browse the commits even further, by pressing enter when the cursor was on such a line in the righthand buffer. I would instantly be lost in weird screens about commits. I also figured there would be ways to commit from Vim (what a Git plugin would it be else), but I hadn't figured out how it worked.
Switched my design from using a grey background with white backgrounds for the posts, to a white background for both, and a light fading box-shadow around the posts. I am very pleased with the results!
What did you do this weekend?
- I edited my vimrc-file
I am switching jobs and programming languages. I already follow a few Ruby oriented podcasts, but I was looking for some new ones. So in light of that search I just listened to an episode of Remote Ruby, the one with Adam Wathan. That seemed like an especially interesting one, because Adam is a Laravel PHP developer, which is what I am now.
Later on in the episode they of course talk about Tailwind, but they start with some very interesting things about PHP and the direction it's taking. Part of the things they say are things I heard said before, but a lot of the things are things I have thought myself and now heard back.
To summarize in my own words: PHP is moving more and more towards adding types and type annotations everywhere. They are not alone in this, types are hot: in the Javascript ecosystem Typescript is growing fast, and lots of other statically typed languages grow as well. But PHP's approach is a bit odd: there is no compile step with PHP, so all your errors are still runtime errors.
I really feel more at home in the Ruby world (and Laravel, as the home of the "PHP developers who want PHP to be like Ruby"), which is why switching to Ruby made sense. But it's also a bit scary to actually join the shrinking side of the world of programming. Ruby is so dynamic, I don't see it supporting type checking anytime soon.
On the other hand, I see the value of types. I think it's valid to say that computers are getting better and better at understanding programs, and types are an important part of how they do that. I like the idea of the friendly compiler and the promises of Elm. But should that mean my PHP-code has to be full of types everywhere? Only for more specific crashes on runtime?
It was nice to hear Adam talk about this other language called Crystal, which apparently does the complete opposite of PHP: compile with static types but with as minimal type declarations as possible. That made me realise even more that my problem might not be with types, but with these annotations everywhere.
While we're on the topic of types...
Also worth noting is Elixir's take on types. Elixir, like Erlang and Ruby, is dynamically typed, and due to the nature of the runtime it's actually very hard to create a type system for it. Some smart people tried it for Erlang and failed, but wrote an article about it and afterward created this tool called Dialyzer.
Dialyzer does not run on compilation, but more as a tool on the side, during development. It looks at your code (and, in Elixir, at the types you can optionally specify), and if it's certain that your code has an error, it will complain. This means it will not catch every bug there is to catch, which is nearly impossible in such a dynamic language (I have heard the article explains why), but it will still catch some and provide value that way.
To defend PHP a bit here: this looks a lot like how my type enthusiastic co-worker uses types in PHP. His Dyalizer is called PHPStorm, and like Dyalizer it runs analysis on the code during development. This also reduces the runtime errors in the same exact way.
The point where I get the creeps, however, is the place where the type is defined. In Elixir, it's an annotation you can optionally set on a function and in the compiled code remains no trace of this hint. In PHP, the type hints are baked into the language and cluttering the real code. And they are just not that good.
For example. In Elixir you can say "the thing this function returns is a Banana, please don't look at it". The function can then actually return a string, but once you access it as a string, Dialyzer will complain, because it's a Banana, you know. In PHP, there is only strings and integers, and if you want something for yourself you got to wrap it in an object. And before you know it, there's classes everywhere, that do not really add anything more than type information.
I am not the only PHP developer, and it seems like most PHP developers are happy with the direction that is taken. It's a bit sad that due to the way things are adopted, more and more libraries and thus applications are forced into using types. Although PHP says they are optional, because they are not optional when extending classes, type declarations leak into more and more PHP code.
It was also interesting hearing Adam talk about Ruby and especially calling out Elixir as one thing he would like to explore if PHP fails him. That was sort of my plan too, but I'm actually acting on it now. I am very curious how that will turn out.
This is a private copy of my post on the ElixirForum
So I have looked for this topic and found similar ones but not actually this one. Forgive me if I missed it.
While I really love Elixirs pipelines, I see myself writing this pattern from time to time:
def some_function(start) do
result =
start
|> Enum.map(&do_something/1)
|> Enum.filter(&but_without_these/1)
{:ok, result}
end
For this pattern, people seem to have come up with operators that do things with second or last arguments (I found one of those discussions). I'm not really interested in that, I want to make another point.
In the above pattern, I start reading about a result, then I see start, and then the actions that lead to start. Your eyes notice the pipeline quite fast, you see what happens, but especially when the action after the pipeline is more complicated, you start asking yourself: wait, what did they call the result of this pipeline? To find out, you have to go all the way up to read the variable name.
Put in another way: this way of writing messes with the ordering in which things are happening. First, the start is evaluated, then the pipeline is run, and only after that the match is completed which gives result a value.
I think it would be nice to be able to write it like this:
def some_function(start) do
start
|> Enum.map(&do_something/1)
|> Enum.filter(&but_without_these/1)
>>> result
{:ok, result}
end
This removes some indentation and reads nicely in order of what happens when. It might even stop the questions for |2>? You can just start a new pipeline after the first one with this variable where-ever you want.
Oh, and it's a pattern match, so it's quite powerful, as you know.
def some_function(start) do
start
|> Enum.map(&do_something/1)
|> Enum.filter(&but_without_these/1)
>>> [first | _]
Enum.zip(start, first)
|> Enum.reduce(&more_transforms/1)
>>> result
{:ok, result}
end
Note: I used >>> here because it's one of the available custom operators, but I think => would be prettier (but probably taken) or <| (but that's not available). Here's a very naive macro to make it work:
defmacro left >>> right do
{:=, [], [right, left]}
end
Again, if this has been proposed too many times, please pardon the intrusion :)
My site broke, because the /2019 folder in my storage did not yet exist, and somewhere over the last year I added code that relied on that. So 19 years in, the Millennium bug is still active.
Just implemented a Siri Shortcut for IndieAuth! Now let’s see if I can integrate it with my Micropub shortcut.
At IndieWebCamp Berlin this year, at the session about Workflow, we came up with an idea, how to enhance your blogposts with an external service using Micropub. I’ve thought of a few variants, and in spirit of the IndieWeb I should first build them and then show it, but I haven’t got around it yet.
So y’all will have to do with just a description. I might implement it at some point, if I have a real use case for it. I don’t actually want weather on my posts.
But let’s start at an idea I first had at IndieWebCamp Nürnberg.
The Syndication Button Hack
Micropub is an open API standard that allows clients to post to servers. In the spec, there is a mechanism for clients to show buttons for syndication targets. The client asks the server what targets there are, and the server responds with a list of names and UIDs. The client then shows the names on buttons (or near checkboxes) and if the user selects one, the UID is set as the mp-syndicate-to field of the post. The server is then responsible for syndicating the post to, say, Twitter or Facebook.
This mechanism is widely supported among clients. And since the client does not have to do any work actually related to the syndication, it can also be used for other things.
Imagine the server implementing private posts. The support for private posts in Micropub clients is not really existing at the moment of writing. But we can get a button to toggle the state of the post created, quite easily:
Since it’s up to the server to syndicate to private-post, it can decide not to syndicate it, but to mark it private. There are a number of possibilities with this: toggle audiences, mark the post as draft. All these things could have their own queries at some point, but until then, this will work in almost all the clients.
Also notice the Bearer token. The server can know which client is asking, so it could show a different set of buttons, depending on the client. Quill supports draft posts? Don’t show that button in Quill.
Enter the Weather Service
Back to the idea of Berlin, which takes this one step further. If we have the Syndication Button Hack in place, we can also hook up external services to enhance our blog posts.
Say I display a location with every entry I post. I could have a button that says: ‘Weather Service’. Activating that button would instruct my server to ping the Weather Service about the existance of this new post. This could be done by WebSub or some other mechanism.
Back when I signed up for the Weather Service, I gave it access to my Micropub endpoint as well. The Weather Service waits for new posts to arrive, reads their location, fetches the weather for that location, and sends a Micropub update request.
The only new part this requires, is the button and the ping to the Weather Service. All the other parts exist in clients and servers. Ah, and someone will need to build that Weather Service.
External services in general
The nice thing about this model, is that the heavy lifting is on neither the Micropub client nor the server. It’s on the external service. And it’s not that heavy of a lifting, because the external service does only one thing and does one thing well. It can give superpowers to both Wordpress blogs and static generated sites.
The external service could provide information about the weather, but think of Aaron’s Overland and Compass: it could also provide the location of the post given a point in time. There might be more. Expanding venue info?
One thing to watch out for, is concurrent processing of these Micropub requests. This might not be a problem for you, but I store my posts as flat files. If two services send an update request for the same post, one might start, and the other might overwrite the first one. (I really need to check how my blog handles this case.)
When you are using a database like MySQL, you should be safe for this kind of stuff, but it still depends on the implementation of your Micropub endpoint.
Other ways of doing it
Peter did not like this first approach, because his post would have multiple visible states (first a few seconds without weather, then with it).
Another appreach would be a sort of Russian doll Micropub request, where you sign in to an external service which signs in to your Micropub endpoint. This would mean that quill.p3k.io posts to weather.example/micropub which intercepts the request, and sends the same request with weather info added to seblog.nl/micropub.
I don’t like that approach either, because now I have to trust the Weather Service with my tokens. In the first approach, every service gets their own scoped token, which is safer.
Since the server knows how many services it has asked to enhance the post, it could also keep it in draft until the last update request comes in. This would require more work on the server’s side of things, and there has to be a timeout on it, but it could be a way to mitigate Peter’s problem.
As always: feel free to steal or improve, but please let me know.
Recently I’ve been to IndieWebCamp Berlin, where I spend the Hack Day on abusing the PushAPI to update ServiceWorker caches.
I would like to start with a small section on what and why, but while I was procrastinating on writing this blog post (the pressure is high), no one less than Jeremy Keith wrote a blog post about it. Since that’s a perfect what and why, there are just two things to do for me here: demo and how.
Demo
I did a demo in Berlin, but the demo-gods where unforgiving. It did not work at all, but when I got back to my seat, it started working again. What happened? My Mac tried to be nice and turned off notifications while I was presenting.
But, as to make up for it, the new macOS Mojave shipped with a screen capturing tool. So here is a retry of the demo in under 5 minutes:
The how
This might not be the most interesting part of it, but it’s nice to share work. It’s not a full comprehensive guide on how to do this stuff, because that would just take way too long. See it as a quick guide behind the different API’s involved.
I googled it all anyway. You can google along.
Oh and if you want to skip ahead: there are some use cases at the end.
Showing a local notification
Like with any Javascript, you should check support before you ask something. There is a list of things to ask in the below code example: we want Notifications, it should not be denied, there has to be ServiceWorker support, and for the part later on, there should be a PushManager too.
Once we prompted the user and got permission, it’s as simple as getting our ServiceWorker registration and ask it to show a notification. As you can see: this involves the ServiceWorker, but it does not involve any other servers.
function activateNotifications() {
Notification.requestPermission()
.then(status => this.status = status)
},
function supportsNotifications() {
return ('Notification' in window) && (this.status !== 'denied') &&
('serviceWorker' in navigator) && ('PushManager' in window)
}
async function sendTestNotification() {
const reg = await navigator.serviceWorker.getRegistration()
return reg.showNotification('Hallo, test!')
}
Note: the demo code is using Vue, which I leave out in this blog post to simplify things. But that’s where this points to: a collection of variables on the Vue instance.
Subscribing for the PushAPI
Once the user clicks the button ‘Subscribe’, the following function gets triggered. In here, we again get the ServiceWorker registration, and then access the PushManager on it, which we tell to subscribe.
Some browsers have their own way of doing authentication, but the most universal is with a Vapid key pair. The package I use for the backend came with a way of creating them. We give the public key to the PushManager, which will give us a Subscription object.
In the end, we send the Subscription’s key, token and endpoint to the server via a POST request.
Note: my HTTP library of choice is axios and the urlB64ToUint8Array() function can be found here
Storing the Subscription
For the backend, I’m using a Laravel package for WebPush, which allows me to save the endpoint with very minimal code:
public function update(Request $request)
{
$this->validate($request, ['endpoint' => 'required']);
$request->user()->updatePushSubscription(
$request->endpoint,
$request->key,
$request->token
);
return response()->json(null, 201);
}
As you can see, it is using the user to associate the data with. (I fake the auth in the demo, which I do not recommend.) It ends up in a database, with four main columns: user_id, endpoint, public_key, auth_token.
In theory, you can go without users, but you will need to store the other parts. The token and key look like random strings, but the endpoint is an actual URL, on a subdomain of either Mozilla or Google, depending on the browser. (No support on Safari yet, mind you.)
These endpoints and tokens can expire, so you will need to keep an eye on the table.
Sending the notification
I can be short about this part: I have no idea. The following code is all it takes to trigger it:
Notification::send(
User::all(),
new NewBlogPostCreated($content, $notify)
);
... where $content is the content of the post, and $notify a boolean, telling my ServiceWorker whether or not to show a notification (we’ll get to that).
The NewBlogPostCreated class extends Laravel’s build-in Notification class and has these two methods:
public function via($notifiable)
{
return [WebPushChannel::class];
}
public function toWebPush($notifiable, $notification)
{
return (new WebPushMessage)
->title($this->notify ? 'notify' : 'update-cache')
->body($this->content);
}
There is a lot of magic behind the scenes here. I have no idea. In the end, they send a POST request to the endpoints of those users, after signing the right things with the right keys.
Receiving the notification and then don’t
Next, we’re back in Javascript-land, however, this is the ServiceWorker-province. The ServiceWorker, once installed, is a script, written in Javascript, but completely decoupled from any window. It lives in your browser and represents not one page, but your whole website.
It’s quite hard to wrap your head around at first, but, I think the PushAPI makes it easier: there is no window involved with a push message, and there is no page involved with a push message. There is only your ServiceWorker, which acts for your whole website.
The ServiceWorker script itself consists of a series of callbacks, that are executed whenever things happen. In the case of a push message, the 'push' event is triggered:
(function() {
'use strict';
self.addEventListener('push', function (e) {
const data = e.data.json()
self.caches.open('manual')
.then(cache => cache.put('hello', new Response(data.body)))
if (data.title == 'notify') {
e.waitUntil(
self.registration.showNotification(
'New content!',
{body: data.body}
)
);
}
});
})();
That’s all I need for receiving push notifications. I first retrieve the data from the message. Then I open the cache named ‘manual’ and I put the body of the message in that cache as the content of a URL (in this case ‘offline.test/hello’). It is made for pages, but I use it as a key-value store here.
Then I check the title field, which I have abused for this purpose. If it is set to the magic string ‘notify’, I will trigger the notification. If it’s something else I will do nothing.
This shows that I don’t have to: I can leave the notification out, but I still get a ServiceWorker activation and I can do whatever I want with it.
Use cases
I think this can be used for creepy things (can I occasionally ping my ServiceWorkers and ask for data like ‘how many windows are open?’ and phone that home?), but I also think there are nice uses for this as well.
As Jeremy wrote: this can be used for magazines, podcasts and blogs to push new content to my phone, to read on a plane or in the subway when I’m offline. I see a nice feature for a web-based IndieWeb Reader too: it can push me copies of posts it collected.
I think the Reader is a nice place to use this. With great power comes great responsibility. Do I want to grand that great power to that weird magazine, that dubious podcast, that blog I visit once or twice a month? I might know you well, I might not. Do I trust you, pushing megabytes on my phone without me noticing?
Web apps like a Reader are easier to bond with. Plus: once I know my Reader supports reading offline, I might visit it in the subway. Will I remember the magazine?
The last bonus of the IndieWeb Reader specifically: it can send me posts from any magazine or podcast or blog, whether they support offline reading or not. But that’s more specific to the Reader than it is to Push.
I’m also very curious to know how things will evolve if ServiceWorkers get even more superpowers. How well will those pair with a free ServiceWorker activation? Lot’s of exploring to do!
Apart from the question how to fetch private feeds, there is also the question how to present private feeds. The easiest way is probably to give every user their own feed, containing only the private posts for them. They can separately follow your public feed, and your queries are easier.
But in line with Silo’s like Twitter and Facebook, I think I would prefer presenting one feed, with both public and private posts, scoped for the authenticated user. When I described this to Aaron he said that he liked it, but that he didn’t know where to begin with writing code that does that. I didn’t either, but it made me want to explore the possibilities.
On a sidenote: this feed design also raises another problem, of how to signal to the user that they can see this post but no-one else. I leave that one for another time.
Drawing rough lines around boxes
Borrowing from Facebook, there are roughly four categories you can share content in:
public – These posts can be seen by anyone. This is the default on nearly all IndieWeb sites today.
authenticated – These posts can only been seen if you sign in, but, anyone can sign in. Facebook has this category and we can mimic that with IndieAuth, but it might not add that much value.
friends only – This is a big category on Facebook, and made possible by the friendslist, which is also a big feature on Facebook.
selected audience – Facebook also allows you to pick your audience on a per-post basis. This can be done by either selecting individual users, or selecting lists, which can contain users.
There is also the possibility of excluding specific people or lists from posts, but that one is even more advanced, so I put it out of scope for this exploration.
The first category is easy, for we already have it. The second category is harder, but once you got past the authentication it’s easy again. One could query a database for visibility = 'authenticated' OR visibility = 'public', that would work.
The third category would require us to keep a list of friends. The fourth category could also require us to keep lists of people, so it might be better to merge them.
Throw in some tables
This brings us to a simple database schema. I see three main tables: entries, people and groups, with a pivot table between all of them: entry_group, entry_person and group_person. I have chosen ‘people’ over ‘users’, because I might not want to give these people write access to anything, but they could be users as well.
It should work like this:
Entries have a field for visibility, which can me marked public, authenticated or private.
People can belong to groups, which have names. Think ‘Friends’, ‘Family’ and ‘Coworkers’.
Entries can be opened up to individual people, or for a whole group.
There might be better ways of naming these, but I like the simplicity of this model. With private posts and audiences, I will always have to manage some form of lists, and this is the most simple way of doing it.
Enter the monster query
So, with some trial and error, PHPUnit tests, and a lot of Laravel magic I came to the following monster query for these tables:
(
select `entries`.*
from `groups`
inner join `group_person`
on `groups`.`id` = `group_person`.`group_id`
inner join `entry_group`
on `groups`.`id` = `entry_group`.`group_id`
inner join `entries`
on `entries`.`id` = `entry_group`.`entry_id`
where `group_person`.`person_id` = ?
)
union
(
select `entries`.*
from `entries`
inner join `entry_person`
on `entries`.`id` = `entry_person`.`entry_id`
where `entry_person`.`person_id` = ?
)
union
(
select *
from `entries`
where `visibility` = 'public'
)
order by `published_at` desc
... which is way shorter when expressed in with Laravel’s Eloquent:
class Person extends Model
{
public function timeline()
{
return $this->groups()
->join('entry_group', 'groups.id', '=', 'entry_group.group_id')
->join('entries', 'entries.id', '=', 'entry_group.entry_id')
->select('entries.*')
->union($this->entries()->select('entries.*'))
->union(Entry::whereVisibility('public'))
->orderBy('published_at', 'desc');
}
public function groups()
{
return $this->belongsToMany(Group::class);
}
public function entries()
{
return $this->belongsToMany(Entry::class);
}
}
Since the method timeline() returns the Query object, other where-clauses can be appended when needed.
I am in a bit of a fight with Laravel still, for it adds
'`group_person`.`person_id` as `pivot_person_id`, `group_person`.`group_id` as `pivot_group_id`' to the first query, which makes it blow up, but the raw query works!
There is possibly a better way of doing it, but this is a start! Feel free to steal or improve, but if you improve, let me know.
Spent a good evening reading up on Reader-discussions and -ideas, then on refactoring the Microsub endpoint in Leesmap into separate Controllers. Very curious how Aaron this does in Aperture (which is also PHP/Laravel), but still not looking at his code until I'm done with it.
Do you know that feeling when you just get out of a rollercoaster and want more, more, more, but when you are being hoisted up in the cart, you're certainly unsure about why again? That's what I feel with IndieWebCamp Berlin right now. But I'm sure it will be fine once I'm there :)
Oh IndieWebCamp. You come with a few things you want to for your own website, then you do some completely other things, and after that you leave with an even longer list of things to do for your own website.
This year is marked as the ‘Year of the Reader’, and indeed, there was a lot of Reader talk last weekend. I really like the progress we are making with Microsub and apps like Indigenous, but I also noticed we’re not there yet for me. But that’s not a discouragement, quite the opposite!
This blogpost has three parts: first I describe the painpoints I feel at the moment, then I describe what I have been hacking on yesterday, and in the last part I share some other ideas we talked about over dinner in Nürnberg, that where not recorded in any form other than short notes on some phones.
Part 1: The current painpoints of the Readers
In May, at IWC Düsseldorf, I installed Aaron’s Aperture (with Watchtower in the back) on my own server, so I could start getting the joys of having my own reader as well. It was before he offered a hosted version, but more on that one later.
So I started using Aperture as a backend, got into Eddie’s beta for Indigenous, tried to make my own frontend, added all my Twitter feeds and got distracted with my dayjob. Although I am using Twitter lists to split up the giant feed into smaller channels per topic, I could not keep up with the volume of it all. When I arrived at Nürnberg Wednesday, I had over 10k of unread posts.
I have a problem with algorithms that sort my posts by parameters I don’t know about, made by people who want to sell my attention to others. I like having an IndieWeb Reader to solve that problem. But I also have a problem with the volume of posts created by people I follow. I want a tool to manage these streams of information, so I probably need a more sophisticated algorithm than just sorting Twitter-users by channel.
As for my 10k unread posts: I declared bankruptcy and marked them all unread to start over. I am glad I wasn’t wasting Aaron’s resources for this.
Speaking of Aaron’s resources: although there are a few other Microsub projects popping up, his Aperture is stil the dominant Microsub-server. He is limiting the services he offers to store posts only for one week to keep people from being too comfortable, but his instance is still slowly growing and there is no ‘competitor’ on the market yet.
As Sven put it: this is another single point of Aaron in our stack.
Those are roughly the points I started my hackday with. I’m not suggesting I have solved them at all, but I tried – as we say in Dutch – to hit multiple flies in one clap.
Part 2: A graph-based IndieWeb-reader
This part gets a bit technical, feel free to scroll ahead to the section about How it looks in the Reader if you’re more into the main idea behind this reader. Also watch the screenshots.
A little while ago I came across the topic of graph-databases and checked out Neo4j. It’s quite a cool tool. I still have very little experience with it, but the basics are not that hard.
The current iteration of this site is written in Kirby and stores posts in .txt files, in a folder structure like 2018/295/13/entry.txt, in which the numbers refer to the year, the day of the year and the number of the post of the day. In order to make things like categories searchable across my 9000+ posts, I index them in an SQLite database. The nice thing about this database, is that I can throw away the contents at any time and just regenerate it from the .txt files.
The original idea I had in mind with the graph database, was to use it as this indexing database. Neo4j can still answer questions like ‘give me all entries with the #indieweb tag’, so I can use it for that. Some posts, however, point to other posts outside of my site. Things like bookmarks and likes have external URLs associated with them, and it would be nice to plot those as a graph. In Nürnberg, however, I realised that it’s cool as an index for my own posts, but even more powerful for an index for a Reader.
Getting it graphed
So yesterday I started a Microsub server that does that. It’s really not that far, as I have to manually point it at feeds to import, but it’s a base. Take a look, for example, at the following graph I got after importing Aaron’s main feed, Aaron’s like-feed and my own like-feed:
There is something wrong with my own authorship discovery on feeds, but I placed posts on Seblog near each other. As you can see, there are a lot of posts by Aaron, some of which are likes, which point to other posts (which have authors as well). Aaron likes multiple posts my Joschi. Also note that I like one of those posts too.
The importing algorithm currently looks something like this:
fetch the feed (using XRay);
for every entry, save the entry as an (:Entry) node, storing the full jf2-JSON in a content property, as well as some special properties like published and url for better indexing;
if there is an author:
save the author as an (:Author) node, with name and url properties;
save the relation between these nodes as a -[:AUTHOR_OF]-> relation;
if the entry has a like-of property:
fetch the liked entry;
store the liked entry as another (:Entry) node;
save the relation between the entries as a -[:LIKES]-> relation.
Of course, this can be extended for bookmarks, replies, any mentions really. Note that these relations have a direction, just like a like-post has a direction.
Querying the graph
Looking at the graph provided with Neo4j Desktop is really cool, but it’s not yet a Reader. Luckily that’s not the only output. You can actually query this stuff.
I made a Microsub stub endpoint (without authorization) that could return timelines for three different pre-defined channels: ‘Latest posts’, ‘Checkins’ and ‘Liked posts’. Let me walk you through the queries involved.
In Neo4j’s query language named Cypher, you can form SQL-like statements with a bit of ASCII-art to get your data out. It is made out of two parts: a MATCH and a RETURN. (There are others, like CREATE, but this pair is most useful in this context.)
The query below matches any node ((n)) with the label :Entry, and names those e. It then returns 100 of them, ordered by the published date.
MATCH (e:Entry)
RETURN e
ORDER BY e.published DESC
LIMIT 100
The next query does exactly the same, but filters by the post_type of checkin:
MATCH (e:Entry {post_type: "checkin"})
RETURN e
ORDER BY e.published DESC
LIMIT 100
I believe it’s also possible to create the same query by adding a WHERE e.post_type = "checkin" as a second line, and I have no idea what the difference is. The above example is more ASCII-like, but, it gets even better. Say that we would like to see only posts by Aaron:
MATCH (aaron:Author)-[:AUTHOR_OF]->(e)
WHERE aaron.url = "https://aaronparecki.com/"
RETURN e LIMIT 100
The (n) resembles a node (of any kind, unless specificly tagged), and the -[r]- represents the relation, in our query even in a certain direction.
Now the last query is the most interesting. (Any Neo4j experts out there: please tell me how performant this would be on a bigger data collection.)
MATCH (entry:Entry)-[:LIKES]->(liked:Entry)
WITH liked, count(entry) AS likes
RETURN liked
ORDER BY likes DESC, liked.published DESC
LIMIT 100
This one looks for entries that like other entries. It then counts the number of entries that like these newly discovered liked entries, and orders the result by that.
How it looks in the Reader
If you scroll back to the screenshot of the graph above, you will see that there is one post by Joschi, that is liked by Aaron, that I also liked. This one post is now liked by two people in my graph, and thus it will show up above all other posts.
Here is a screenshot of the result of that query in the reader:
Note that these are all posts that do not appear in any of the feeds I follow in this test-reader: in this reader I only follow Aaron, his likes and my own likes. I have discovered new interesting posts by looking at the likes my friends post.
As a bonus, to take it one step further, I can also actively look for posts of people I do not follow, with something like this (completely untested) query:
MATCH (me {url: "https://seblog.nl/"}),
(entry:Entry)-[:LIKES]->(liked:Entry)<-[:AUTHOR_OF]-
(:Author)<-[:FOLLOWS]-(follower)
WHERE NOT me <> follower
WITH liked, count(entry) AS likes
RETURN liked
ORDER BY likes DESC, liked.published DESC
LIMIT 100
That’s just hard to do in SQL.
I will try to hack on it some more, to get a really functional reader with this graph as a database behind it.
Part 3: Some other strategies worth exploring
So in the previous part, I got kind of carried away with explaining how this graph stuff works. Let me get back to the Reader experience itself. This part consists more of ideas that are not yet implemented by any reader. If you feel inspired, go ahead and make something.
Filtering feed data
In the cocktailbar last night (elitists as we are), we discussed the place of my graph-based Reader (codename Leesmap) next to Aperture. It’s sad that, in order to discover these posts liked by friends, you will need to switch your full Reader backend from Aperture to Leesmap.
We discussed that it would be nice to have an way of using Leesmap with Aperture as a sort of plugin. Leesmap could then receive posts from Aperture, index them, and create a few custom channels to fill with interesting posts.
Any Microsub server could of course use the same mechanism to also have Leesmap plug into it, and others could make filters too, for example spam-filtering or crazy Machine Learning stuff.
We need more thought about how such interaction would look like, but the nice thing is that you can use one server to savely store your data, regardless of the fancy filtering services you choose to use as well.
More ways to combat feed overwhelm
Before IndieWebCamp, we had a discussion about Readers in a traditional Nürnberger restaurant. Here also, people came up with some ideas to deal with accruing unread-counts.
One idea came from how Aperture deletes posts after 7 days. This actually prevents the overload. It would be nice if you can tell your reader that, for example your Twitter feed, is ephemeral and that the posts can be discarded if you did not read them in time.
One other idea that came up was to keep track of the average time between posts of a certain feed. This way a Reader could boost posts when they are from a feed that is not regularly updated. These kind of posts are usually lost in piles of more posts from more frequently updates feeds.
Yet a last idea was to tell your reader to leave out posts with certain words for a small period of time. This can come in handy when you haven’t watched the newest episode of Game of Thrones yet, but want to stay connected to your feeds without spoilers.
This year really is the year of the Reader and it’s really exciting. I will continue to work on Leesmap a bit more, and share progress if I make some. Hope you do too!
So at one point I proudly admitted to a co-worker that I knew how to search and replace in Vim: I did /foo, then ciw, bar, and escape (changing the inner word to 'bar', or whatever the right replacement motion would be) and then press n and . for the number of times I needed to (first one goes to the next occurrence of 'foo', and the period key repeats the last edit, which is magic and very powerful in itself).
He replied with a less interesting, but in some cases more appropriate way of doing it: :%s/foo/bar/gc. This command takes the current file (%), and substitutes 'foo' for 'bar', with a flag for global (more than once) and choice. Vim will then stop at each 'foo' and gives you choices for 'y' or 'n' (and others), so you can interactively pick your replacements.
Today I leaned about the gn text object. Where iw stands for 'inner word', gn stands for 'go next'. So one can type /foo, then cgn, bar, and escape, which is almost the same. But then you can just keep hitting ., and it will perform the last action on the next occurrence of 'foo'. No need for n anymore!
I mean if you want to be presented with a choice for each replacement, go use :%s, but this gn thing is darn cool.
I just added alias :q="exit" to my .zshrc file, because I kept doing it.
Maybe I should start a Vim-log of things I encounter and find useful. Most of them seem too advanced for the beginner and too trivial for the expert, but who's really an expert at Vim, aren't we all beginners for some part of it?
So, today I learned about the :cd command, which I instantly memorized (for it's just Unix). It changes the home directory of Vim, (by default this is the folder you start Vim in).
For example, if you're at ~/code and you want to check the contents of some file in my Seblog-project, I would do vim seblog/site/config/config.php. But as always, I need to check another file, so I use :e (short for :edit) to open that file, but because I was in ~/code when I opened Vim, I need to type: :e seblog/site/config/other-file.php. I find that annoying, so I would close Vim altogether, cd seblog and then vim ..
Now, I would just :cd seblog and then :e site/config/other-file.php. Or use :Ex (for :Explore) if I wanted to look around, but I knew that one already.
At Virtual HWC last week, Sven Knebel pointed me to the new Firefox beta. I use it now, and one of the things I noticed is that ships with integration with Pocket, a bookmarking service to save articles you want to read later. It’s owned by Mozilla now, so they accentuate their service by adding a button prominently in your address bar.
Despite the pushiness, I tested it out a bit. I like that I can save articles with one click, so I can read them later, possibly on a different device. It made me think about the way I post bookmarks on my own site.
My bookmarks and likes look too similar
The way I have implemented bookmarks at this moment, is very, very similar to the way I implemented likes. It is a Microformats property (u-bookmark-of vs u-like-of), displayed as an icon (a grey bookmark vs a red heart), with a Dutch text (‘Seb heeft [dit] gebookmarkt.’ vs ‘Seb vindt [dit] leuk.’).
By making bookmarks and likes this similar, one would almost think that there is a clearly defined difference in the words ‘bookmark’ and ‘like’, that keep them apart, since there is no other distinction. I don’t think there is such a definition.
Bookmarks are mostly used as a ‘want to read’-list, or a way of keeping track of things that have been read, but might be of interest on a later moment (‘want to read again’). Likes are more a reply of some sorts, directed at the author of the post, expressing appreciation.
The problem comes when I take readers of my blog into account. The things I like, might be read as a recommendation, but the things I bookmark, might also read as a recommendation. Once I start looking from that perspective, likes and bookmarks fulfill the same role again.
When likes and bookmarks are not recommendations
Sometimes I like things on social media, not because of the contents of the post, but because of the social context around the post. I do not really like the bad picture of the malformed pizza of a friend, I like the fact that I recognize that pizza as the outcome of the enthusiastic plans about making a pizza that my friend told me about earlier that day. I sometimes don’t like the specific check-in a person posted, I just like the person. Some tweets are also quite ambigu: do I like the tweet because of the tweet, or because of the linked article I might or might not have read?
Those likes are not recommendations for readers, they are purely appreciation, or even just social acknowledgement. Within a certain social context they can be of value to other people, but to random strangers, they are not. Currently, I solve this problem by not posting those kinds of likes to my site at all (leaving them on Facebook or whatever silo), but that’s of course not ideal in the IndieWeb scheme of things.
With bookmarks, a similar thing can happen: not all bookmarks are recommendations. The easiest example is an article that I think looks interesting, so I bookmark it to read it later, but I haven’t read it yet. I do not recommend that article, but a reader might think that.
In both cases: sometimes I do want to recommend an article in a single post.
What I don’t want to propose
Let me make a little pause here and say something about likes, favorites, recommendations and what more. We can solve the above things by just adding more webactions to the field. “Let’s also support, next to bookmarks and likes, favorites and recommendations.” But I don’t think that’s the solution, because adding those options means just more post types to keep track of, for both publishing sites and consuming indie-readers.
Keeping things a little bit abstract and minimal helps us focus on the problem at hand. (Which is at this moment, I think, building an functioning indie-reader in the first place.)
What I would add to my bookmarks
Seeing what Pocket does with bookmarks, I think I want to expand what I post as a bookmark. To be fair: a lot of other people on the IndieWeb have more detailed bookmarks too. Other properties of a bookmark include: tags, a little summary of the bookmarked post, the reason why the post was bookmarked and sometimes even a screenshot of the bookmarked page.
I’m not sure how much I want to add to them, but since they are very skinny now, I certainly would like to add some tags. Tags, and possible a reason, make it easier to find a bookmarked article back after a while.
That brings me to what I think a bookmark would be for me: showing an interest in the linked article, without adding too much judgement. I see bookmarks as a personal archive of things I want to read or have read. If technical skills allow it, I would also save a copy of the post for personal use, hidden in the bookmark-post itself.
What I would change to likes
To likes I wouldn’t change much. For likes, I want to make a personal copy of the original too, because I do care about that content, but from the outside, it’s just a link.
But the value of the like would than be more of a vote: this is a piece of content I care about. I think indie-readers should also consume those likes, but just don’t display them the way they display a photo or a note. If there is a post by an author that I don’t follow, but that is liked by, say, three people I do follow, then I want to see that post, accompanied by the names of the three people who liked it.
The threshold for the external posts to show up may vary from person to person, both personal preferences of the reader as well as the status of the poster of the like. But the point is that it’s a indie-reader-problem.
In this way, a like is in fact a recommendation, from the readers perspective, but just appreciation from the poster.
But back to these social likes
Then there are still those likes that depend on some social context, that aren’t solved by the above approaches.
I think that the root of the problem lies in those exact words: social context. If I like a certain badly photographed pizza, I should not post that to my main feed, but set the visibility of that like to ‘friends only’. Chances are that the badly photographed pizza was already published as a private post, only visible to a certain audience, so I can copy that.
The problem here lies more in an easy way of publishing private posts, and an easy way to change the audience of those posts. And of course a way for sites to securely share those posts with the right indie-readers, there is a long way to go still.
And what about the bookmarks you haven’t read?
Sharing unread bookmarks is also, I think, question of adding the right audience to the post. Such bookmarks can be posted with an ‘only me’ visibility. You can then subscribe your indie-reader to your own (private and public) bookmarks-feed. This is, without the private part, what I actually did for a while when I was using an indie-reader. Apart from the bugs in my self-build reader, it worked very well.
And really recommending something? Well, if I really want to recommend something to people who follow me, I can always just post a note, linking the article, and explain why they should read it. That also improves the chance of actually clicking through, no need for extra formatting.
Conclusion
In the end I don’t think we need better boundaries between a bookmark and a like, in the form of extra ‘recommend’ or ‘favorite’ webactions. A bookmark adds something to an archive for yourself, and a like is giving appreciation to the creator of the post. Recommendations can be either explicit by posting a note, or implicit by publicly liking or bookmarking. But, the way this is implied should be the responsibility of your indie-reader, where you can mix to your own taste. Publishers could filter things that they don’t want everybody to pick up as recommendations by using ‘only friends’, ‘only who I follow’ or ‘only me’ posts.
In the end of the day, this is a lot of thinking and talking. I should get back to creating a indie-reader. As should you, because that’s where the one of the undefined parts of the IndieWeb lies now.
In last weeks IndieWeb newsletter I read a blogpost by Sven about having notifications for webmentions and such via IRC. He described more or less the setup I have too: using Aaron’s TikTokBot framework, running on the same machine as a private IRC server for it and your IRC bouncer. In my case, the machine is a Raspberry Pi in my living room, but Sven’s guide describes more or less what I did too, so please check that out.
I divided my webmention notifications into webmentions and silomentions (that’s backfeed from services like Bridgy and OwnYourSwarm) and I also get notifications for logins and Micropub requests, so I can keep track of what’s happening on my blog. But that’s just expanding the notifications. I added some other functionality I wanted to talk about.
Hey bot, please like this post
The one thing that made me post on my site the most, was adding Micropub support. There are various Micropub clients out there that you can use to just write a blogpost, to post a like, to import checkins from Swarm or what have you. But since it’s so simple (once you know how), I also made a lot of Micropub clients out of Workflow, Paw or, in this case, my IRC bot called Bop.
I can now say like https://example.com/a-post in the same channel my notifications come in, and Bop will post a like-post on my site. My site will then post the webmention and if the other site accepts it, it will be shown as a like under the post, just like that.
Please refer to Sven’s blogpost to learn how to set the bot up. I also assume you already have a site that supports Micropub. I will go into obtaining an access token a bit, but that might be a tricky part and depends on how your Micropub endpoint works.
Getting your bot hooked
Sven mentions the file ‘hooks.yml’, which contains hooks: [] for him. The hooks-file defines the things the bot will respond to, and since Sven only want to receive notifications, not talk to his bot, he does not need hooks. But we do. Replace the contents of the hooks file with the following:
As you can see, I like to match for a URL to support like, bookmark and rsvp, and for anything to support notes on your own site, Twitter and Facebook. Feel free to change the syntax to your own needs.
Under channels, I defined @seblog, which means ‘every channel in the server seblog’. That name comes from your config.yml, and it’s a safety measure, so that when I connect my bot to Freenode also, it won’t accept likes from other people. I am the only person connected to the private IRC network Seblog, so that will be fine.
Creating the Micropub.php
As you saw, the hooks define a URL that will be called by the bot. I specified that to be on localhost:8000 and point to micropub.php. In your TikTokBot folder (or where you want, really) create a folder called ‘server’ and a file within it called ‘micropub.php’. Let’s start with the following to test it out:
<?php
header('Content-Type: application/json');
echo json_encode([
'content' => 'Hi, you called for a Micropub?'
]);
Then go to the newly created folder and run the command php -S localhost:8000. This starts up a webserver from that folder, so if you now go to your private channel and say note This is a test, you will see that the bot responds. If you say something that does not match the regex, the bot will stay silent.
Give me my data back
If we want to send data to our Micropub endpoint, we need to make sure that our bot has that data. Let’s make the bot echo whatever we said, so we know that he heard us.
TikTokBot gives us a JSON object of the message via the POST data. Long story short, you can get it by using the following:
<?php
header('Content-Type: application/json');
$message = json_decode(file_get_contents("php://input"));
$full = $message->content;
$action = $message->match[0];
$param = $message->match[1];
echo json_encode([
'content' => "Hi, you said '$full', you wanted to $action $param?"
]);
Now, if you say like https://seblog.nl/2017/08/19/6/micropub-irc-bot, the bot will respond with Hi, you said 'like https://seblog.nl/2017/08/19/6/micropub-irc-bot', you wanted to like https://seblog.nl/2017/08/19/6/micropub-irc-bot?. Try it out, and different actions too, so you know it works!
Make sure not to echo the full message, because since the bot listens to itself, that message will trigger itself again, resulting in the bot repeating the message over and over again. (Just restart the bot if that happens.)
Sending a POST request
A Micropub request is, in the end, just a POST request. Here’s a little helper function to send one for you. Just start with this code at the top of your file, right after <?php.
Make sure to replace that URL with your own Micropub endpoint, and xxx with your own access token. I’ll talk about obtaining one at the bottom of this post.
I don’t have the bot say anything after this, because the bot will give a notification once a Micropub request is made, which is enough feedback for me. You could also return whatever send_post() returns, I leave that up to you.
More post types
As you can see in the code above, you can just add fields to the array. Here are some other elseifs I use.
See how easy that is? Please make up new ones and tell me about it!
I’m particularly proud of the tweet one, which will, thanks to the exit(), not post the tweet unless it fits the 140-chars-rule.
Please note that the use of the syndicate-to fields (or the newer mp-syndicate-to) depend on your own website. This only gives my webserver the order to syndicate, it does not syndicate by itself. If your server does not know how to syndicate to Twitter, it will not work, but explaining how to do that is it’s own tutorial.
In real life, I actually have note set to 'private' => true and 'audience' => 'http://seblog.nl/, so it’s a private note. But: 1) I don’t expect many people to support private posts, 2) how many people are interested in posting things to their website only they themselves can see? and 3) I want to change the 'private' => true to 'visibility' => 'private', or something like that.
That’s it!
There you go, a customisable Micropub chatbot. Please double check that it’s only doing it’s thing on your private IRC server, and not on any public servers where your bot might lurk (if you use any). Have fun liking and posting!
POST-scriptum: Obtaining an access token
The question “what do I replace ‘xxx’ with?” is a hard one to answer, because it totally depends on how your site handles access tokens. Most Micropub clients ship with an IndieAuth flow to obtaining one, but since we’re making this one ourselves, we have to get one manually.
You can do this by following the steps on the wiki, if you have a way to send POST requests. You can also log in to a Micropub client like Quill, which shows you the access token it got, and use that. Since this all depends on your own site, it might offer an easier way, or it might not.
Update 2017-10-04: In an earlier version of this post, I did not set a Content-Type: application/json-header, so the bot didn't actually respond. Setting that header at the top of your file will fix that.
Twitter
Instagram
LinkedIn
Github
Strava
Facebook
