AWS is a Pain Without a Big Budget

August 31, 2018

In my last website post I talked about my plans for setting up website notifications on AWS Lambda and DynamoDB. The idea is that a function on AWS Lambda would get called when the site had an update, which would fetch all the site data, diff it against the previous state, and determine which pages actually changed. Those changes would get saved to AWS DynamoDB, which has a streaming feature that other AWS Lambda functions can be triggered by for each event. Multiple Lambda functions (one for each service) would get those updates and fire off whatever integration was necessary for each service.

This would put the burden of running the service and hosting the data to Amazon’s ops crew, which is undoubtedly better than what I would have set up. As long as I stayed within the limits of the AWS free tier, which looked pretty decent, I would be able to run this in perpetuity, right?

Homebrew Website Club Seattle Notes

August 09, 2018

Tonight I went to my first Homebrew Website Club at the Wayward Coffeehouse, a real nice coffee shop. I got to meet Doug Beal who put the Seattle club together. It was mostly just the two of us, as well as Margo Vansynghel, a reporter who was interested in what we were up to and how it related to other tech stories about silos like Facebook.

Neither of us really had been to one of these events before, so we weren’t sure what to do. We mostly worked on our own sites, and helped explain some of the IndieWeb principles about things like data ownership and building tools for yourself. I think by the end of it she had sent her first Webmention, a standard I need to learn how to use soon for this site.

I continued working on the syndication system for this site I outlined in the last post. I’ve now got the DynamoDB set up working, so when I commit any post change to the website, it gets built and triggers a Lambda function to scan the entire site for changes, logging those into DynamoDB. This creates a stream of create/update/delete events that other Lambda functions can be triggered by. With tonight’s work, I hooked a new microservice up to that which publishes created posts (but not updated or deleted posts) to a Telegram channel. It works locally with hardcoded keys, so now I just need to make it work when deployed, which should be quick; I just need to figure out how to store credentials properly. Then I can start building out more syndication methods.

Toward the end, we answered some questions about IndieWeb, took our group photo, and went our separate ways. It was a great time, and hopefully we can start building out a proper Seattle club. If you’re interested in taking control of your online data, consider stopping by an event near you, or a virtual one online. They take place every two weeks; the next one will be on August 22.

Setting up Website Notifications with Serverless

August 03, 2018

The new Gatsby website has been soft launched for a couple weeks now and seems to be working pretty well so far, so I’m now moving on from building out the site to building out a bunch of tools for sending out notifications of new content. My hypothesis here is that if I can make the barrier to entry as low as possible for people to get notified of new posts, they’ll be more inclined to read them, and that means going where people already are. Obviously RSS is a great technology for this, but it never really recovered from Google Reader meeting its far too early demise. People use algorithmic news apps, social networks, chat apps, and more to find and share posts, and so that’s where I need to be.

So far I’m planning on distributing posts to:

RSS/Atom/JSON feeds
A Twitter account
A Telegram channel
Medium
Flipboard
Apple News
Email
Mastodon

I’m a big believer in making a bunch of little systems that work together rather than one big monolithic system. I chose to build this project using the Serverless framework built on Amazon’s AWS technologies like Lambda. I really didn’t want to manage keeping processes alive and setting up things like cron jobs, so this makes a lot of sense. There are also a few other endpoints I want to implement later for the site like Micropub and Webmentions which would be nice to have managed through Lambda rather than running forever on some VPS somewhere.

Once it’s working, the basic pipeline looks something like this:

I create a post (or make some change the site) and commit it to a Git repo
I push the repo to my GitLab server
When GitLab receives a commit, it kicks off a CI job to build the site and deploy it.
Once deployed, it sends a message to an AWS Lambda function to signal the site has changed.
The function on Lambda downloads a copy of the current website data and a copy of the last seen website data from AWS DynamoDB
New and updated posts are sent to a WebSub server (you might remember this as “PubSubHubbub”) to signal to anyone who subscribes that the page changed. (Which means that, once this is working, the entire site will support WebSub as a server-to-server push mechanism!)
New posts are inserted as rows into a DynamoDB table
A series of AWS Lambda functions will be listening for insert events on that table, one for each service listed above, and they will farm out the API calls to each service

I like this approach because it’ll hopefully mean there is some resilience to normal maintenance things like server restarts and database failures. It fits nicely with a lot of open standards being pushed by the IndieWeb movement. And it’ll be automatically integrated into the build process, so I won’t forget to actually do the crossposting. It’s already well underway with a lot of the infrastructure in place; steps 1-5 and step 7 are all finished. All that’s left is to set up the individual integrations with each service and connect the WebSub plumbing.