State of the Site
How my static blog works, summer 2023 edition, with nifty diagrams.
I thought I’d attempt to concisely describe the behind-the-scenes of this site. There’s been some updates after a flurry of new development activity in the last month or so.
The Blog
Endgame Viable is a static site, as of 2021. There are pros and cons for static sites. I wouldn’t recommend it unless you have some kind of background in web development.
Every blog post begins as a Markdown file written on my MacBook in a text editor (I use VSCode, although I just learned that Obsidian also works well). Said MacBook is usually near me all day, every day, open and ready for me to type thoughts into said text editor.
That Markdown file goes into a local git repository, in a special content directory, along with all the other files needed to make the blog. On publication day I commit the new post into the repository and hope I haven’t made too many mistakes.
.---.
| Me |
'-+-'
|
v
.----+----. .----------------.
| MacBook +-->| AWS CodeCommit |
'---------' '-------+--------'
|
v
.-------+--------. .---------------.
| AWS Amplify +-->| AWS S3 Bucket |
'----------------+ '-------+-------'
|
v
.-------+--------.
| AWS CloudFront |
'-------+--------'
|
|
v
.-+-.
| You |
'---'
Upon committing a post to the git repository, the changes are pushed to a remote repository in AWS. When the changes are pushed, AWS Amplify runs a static site generator called Hugo and converts all the Markdown content files into a huge directory of static HTML, CSS, and Javascript files. Those files are stored in an AWS S3 bucket, and served at endgameviable.com. (There’s more to it, but that’s the basics.)
An IFTTT process picks up the changes to the RSS feed and posts notifications in appropriate social media locations.
It sounds complicated but it’s all automatic. My work is done after I push the changes from my MacBook.
.----------. .-------. .---------------.
| RSS Feed +-->| IFTTT +-->| Notifications |
'----------' '-------' '---------------+
Images (or any media) is a special case. All of my images are stored in another S3 bucket and served through the AWS CDN. I have a local directory that I fill with the images, and then run an AWS command-line tool to synchronize my local directory with the remote S3 bucket. I keep my images really small (1024x1024) so I don’t use too much space.
Managing images is a massive pain for me, so I try to avoid using them in blog posts. (I still haven’t gotten around to filling in all the older images, so you’ll probably see missing images on older posts.)
.---.
| Me |
'-+-'
|
v
.----+------. .---------------. .----------------.
| PC Images +-->| AWS S3 Bucket +-->| AWS CloudFront |
'-----------' '---------------' '--------+-------+
|
v
.-------+------.
| Blog Posts |
'--------------'
I have to upload all images to the bucket before I can use them in the text of a blog post, so I have to really plan ahead. It’s annoying.
Everything needed to build my site is sourced from my local hard drives, so if anything happened to endgameviable.com, I could rebuild it anywhere, in any format.
I recently counted 1,634 Markdown files in the content directory, containing an estimated 2,719,652 words of writing, going back to around 2010. Hugo creates 2,355 HTML pages with every site update. Building the site takes a few seconds.
Eventually I hope to put my entire online writing catalog, as best I can gather it, into the site somehow, but it’s a complicated problem to figure out how to convert old HTML to Markdown and redirect all the old links here.
Blog Cost
You might think it’d be really expensive to use AWS for this, but one of the major advantages of static sites is that they’re super cheap to operate. The site is small enough that I’m still largely within the AWS free tier limits. It costs me about $3/month right now to run the site. (And $1 of that is the fixed cost of AWS managing the domain name.)
The Microblog
The newest addition to the site for 2023 is an experimental microblogging system. I have random thoughts all through the day and it’s nice to write them down. Sometimes they’re really, really funny. To me. Anyway, Twitter used to work for that, but things change.
It all started with the unrelated task of building an Unraid server on my network to replace my old crashed NAS. Unraid also lets you run Docker containers easily. So I setup a container that runs a very lightweight ActivityPub server instance called GoToSocial, and the rest was easy.
No, actually, the rest was really frickin’ hard. I setup a Postgres database and a reverse proxy on my network. Then I setup AWS CloudFront as another proxy layer between the Internet and my home network. Weeks of tinkering later, and I have a working federated account instance at https://gts.endgameviable.com, at which I can microblog.
.------------. .----------. .---------. .------.
| GoToSocial +<-->| Postgres +-->| Scripts +-->| Blog |
'-----+------' '----------' '---------' '------'
^
|
v
.-----+------. .--------. .----------------.
| Rev Proxy +<-->| Router |<-->| AWS CloudFront |
'------------' '--------' '----+---+-------'
^ ^
+-----------+ |
| |
v v
.----+-----. .----+----.
| Client | | Fediverse |
'----+-----' '----+----'
^ ^
| |
v v
.-+-. .-+-.
| Me | | You |
'---' '---'
Originally I wanted to federate directly from endgameviable.com, but I read there would be federation problems if you ever decide to change the server type running at a domain. So I figured it’d be best to stick with a subdomain, at least for now.
This accomplishes the goal of keeping my writing on my local network (in the Postgres database), but I didn’t like it resting in proprietary GoToSocial tables, so I wrote a hacky Python script to make a daily archive of all my notes and insert them into the blog repository as daily Markdown digest posts.
So I write my microblogs at https://gts.endgameviable.com, using a Mastodon-compatible client called Semaphore.social and a Twitter-like interface I’m comfortable with, and then it’s all backed up to the blog repository the next day. Writing permanence ensured.
The microblog digests go into the RSS feed, but they aren’t shown on the front page unless you look for them. It was just too much clutter.
It’s not fully automated yet, but maybe someday.
Microblog Cost
It’s completely free (discounting electricity) for me to run the GoToSocial instance and obtain all the required TLS certificates, however, the additional traffic through AWS CloudFront is probably going to increase my AWS bill. It hasn’t run long enough yet for me to know, and it depends on how popular my account gets, but at the moment it looks like it might add another $1/month to my bill.
There’s also a modest yearly charge (I think it was $20/year) for the dynamic DNS setup required to get the Internet to talk to my home network. It’s possible to get a free plan, but with the one I use (noip.com), you have to remember to keep re-renewing it every month and it’s physically impossible for me to do that.
The Future
I wanted to add some clientside Javascript features (particularly a floating blogroll of some kind) but I haven’t gotten around to it. I also want to figure out how to hook up some kind of web mentions system, maybe using AWS Lambda functions. All of that is in the far-flung science fiction future.
The site is currently very Hugo-centric, but I’d like to reorganize all the file and directory structures so I can port the content more easily to Eleventy, or anything else. Possibly even work out a way to import the content straight into a Next.js site framework. But if I were to pick a static site generator today, it would probably be Eleventy. It’s a lot easier to work with. Hugo is fantastic, but it suffers from being too flexible. It’s really hard to understand.
I’d like to add in some more integrations with audio and video, but I don’t have any concrete plans. I think it would be cool to have a section of the blog that you could add to any podcast app and do something fun there. Previously, I hooked up a system to attach an audio file at any blog post, which will show up in the RSS feed, but it was too much of a logistical hassle to use regularly.
I’d also love to setup a full searchable index of all my YouTube videos on the blog somewhere, because YouTube’s interface definitely isn’t made for browsing large archives of videos. It’s only made to show you the latest videos and trending videos you don’t care about.
Lastly, I made a mistake originally dividing content into sub-directories of dev, gaming, and writing. I want to get them all back into the same directory and use other means to sub-categorize them.
And that’s the site. For now.
P.S. This is my first attempt at creating diagrams using Hugo support for something called GoAT.