As of now my personal website should support WebSub on all pages, posts, and the RSS feed (basically everything linked in the sitemap). If you have the capability to subscribe to pages via WebSub, you should be able to point it at any page on this site and get notified when that page updates. That’s maybe not the most useful on individual pages, but it is useful for the home page, or if you want to be able to subscribe to individual tags or projects.

This should work well with social readers that let you subscribe to websites, at least in concept. To go with that, I’ve updated the microformats across much of the site to make them accessible to those readers. Things appear alright in web-based validators but I’ve been having trouble getting those changes to appear in the readers themselves. Hopefully that will fix itself as caches expire. But it’s one step further down the road of being a good IndieWeb citizen.

Behind the scenes, WebSub support is handled by Webmaster, my custom-built server that integrates with my Gatsby build system. When GitLab finishes building, it sends a webhook to Webmaster that signals that the site has changed, causing Webmaster to fetch the sitemap and scan the ETag of each file in it for changes. If any of those ETags are different from what’s already known, then that page has changed. The WebSub subscribers for that URL are fetched and notified with the changes. No WebSub hub necessary, but down the road I could switch to one pretty easily.

If you’re reading this, then that means I’ve finished upgrading my website to Gatsby 2. Gatsby is a static site generator that uses React and GraphQL to build the entire website as a set of static HTML files, which I used to build this version of my website. Version 2 has a number of really promising improvements like a component for querying GraphQL from anywhere and improved Webpack and Babel support (which will hopefully let me start trickling in some TypeScript).

The well documented migration process was not as smooth as I’d hoped, but that was to be expected. Gatsby does require some comfort with debugging Webpack and React apps before you can really use it well, and this was no different. The biggest reason for this was that I had a .babelrc file at the root of my project which was causing some difficult to debug problems (and ones with no search results). Ultimately the most important thing I did was to just throw that file out and replace it with the default, a step that is not emphasized enough in the docs. It certainly could have gone much worse and once I discovered this source of my problems, it was much smoother sailing.

Overall the migration took about 8 hours. This included time spent following the migration guide and debugging problems, as well as modernizing some of the new tools in Gatsby 2. Namely, the use of the new StaticQuery API for inlining queries. An example of this is that lovely little photo of me that’s on every page. Previously in Gatsby 1, each page had a single GraphQL query that could be run, so everything had to be shoved in there, including for images like that photo. That meant each page had duplicated query logic for fetching that image. Now, that has been rolled into a component that uses StaticQuery to fetch the image, which simplifies the page-level queries quite a bit. There’s a few ways I use that kind of pattern to clean up the site. You shouldn’t notice anything, but it makes working on the site here much simpler, especially if I want to add something that relies on a query.

I’ve been avoiding ripping this bandage off for awhile, but now it looks like it’s done and I can start using the new features. Gatsby’s got a rich plugin library, including some powerful integrations with service workers. I had this enabled originally but shut it off because it couldn’t detect cache changes very well with it, but Gatsby 2’s updated version of Webpack should make that more viable. There’s also some new query tracing tools which will help me get build times down; right now it takes about 3 minutes to build on the (admittedly very slow) server I run it on, and I’d like to get that to be under a minute. And I am dying to start moving stuff to TypeScript, which Babel 7 now supports.

Congratulations to the Gatsby team on shipping and doing such a huge release!

In my last website post I talked about my plans for setting up website notifications on AWS Lambda and DynamoDB. The idea is that a function on AWS Lambda would get called when the site had an update, which would fetch all the site data, diff it against the previous state, and determine which pages actually changed. Those changes would get saved to AWS DynamoDB, which has a streaming feature that other AWS Lambda functions can be triggered by for each event. Multiple Lambda functions (one for each service) would get those updates and fire off whatever integration was necessary for each service.

This would put the burden of running the service and hosting the data to Amazon’s ops crew, which is undoubtedly better than what I would have set up. As long as I stayed within the limits of the AWS free tier, which looked pretty decent, I would be able to run this in perpetuity, right?

The new Gatsby website has been soft launched for a couple weeks now and seems to be working pretty well so far, so I’m now moving on from building out the site to building out a bunch of tools for sending out notifications of new content. My hypothesis here is that if I can make the barrier to entry as low as possible for people to get notified of new posts, they’ll be more inclined to read them, and that means going where people already are. Obviously RSS is a great technology for this, but it never really recovered from Google Reader meeting its far too early demise. People use algorithmic news apps, social networks, chat apps, and more to find and share posts, and so that’s where I need to be.

So far I’m planning on distributing posts to:

• RSS/Atom/JSON feeds
• A Twitter account
• A Telegram channel
• Medium
• Flipboard
• Apple News
• Email
• Mastodon

I’m a big believer in making a bunch of little systems that work together rather than one big monolithic system. I chose to build this project using the Serverless framework built on Amazon’s AWS technologies like Lambda. I really didn’t want to manage keeping processes alive and setting up things like cron jobs, so this makes a lot of sense. There are also a few other endpoints I want to implement later for the site like Micropub and Webmentions which would be nice to have managed through Lambda rather than running forever on some VPS somewhere.

Once it’s working, the basic pipeline looks something like this:

1. I create a post (or make some change the site) and commit it to a Git repo
2. I push the repo to my GitLab server
3. When GitLab receives a commit, it kicks off a CI job to build the site and deploy it.
4. Once deployed, it sends a message to an AWS Lambda function to signal the site has changed.
5. The function on Lambda downloads a copy of the current website data and a copy of the last seen website data from AWS DynamoDB
6. New and updated posts are sent to a WebSub server (you might remember this as “PubSubHubbub”) to signal to anyone who subscribes that the page changed. (Which means that, once this is working, the entire site will support WebSub as a server-to-server push mechanism!)
7. New posts are inserted as rows into a DynamoDB table
8. A series of AWS Lambda functions will be listening for insert events on that table, one for each service listed above, and they will farm out the API calls to each service

I like this approach because it’ll hopefully mean there is some resilience to normal maintenance things like server restarts and database failures. It fits nicely with a lot of open standards being pushed by the IndieWeb movement. And it’ll be automatically integrated into the build process, so I won’t forget to actually do the crossposting. It’s already well underway with a lot of the infrastructure in place; steps 1-5 and step 7 are all finished. All that’s left is to set up the individual integrations with each service and connect the WebSub plumbing.