My playground project started from a Google Sheet I found on Reddit to track my weight. I initially only wanted to play about and create some visualisations for it. But,eventually I stopped using that spreadsheet and, like most of my weight-loss escapades, got fat again. In the meantime, the playground developed into a hodgepodge of data sources. There was little rhyme or reason to a lot of it but kept me busy and off the streets.
Since then, I bought myself a fancy Withings body fat scale and have continued to track my weight with that. It does a job, but I don’t like that it locks me into Withings’ platform forever. One day it might shut down – leaving me at best with the hope that they’ll make my data available as a half-arsed CSV file.
I also missed some of the features in the original weight tracker. For instance, I implemented rolling average graphs. These give an idea of how my weight is trending over time, rather than day-to-day fluctuations. As much as I care about how big the poo I did was that day, the average is much more useful.
So I set to extract that weight to my own data store of choice, WordPress. It is not the first place I’d visit for storing time-series data that I could later display as graphs. Or to create an Event store to create my own personal audit trail. It is also not what I would consider a “sexy” technology choice. The primary APIs are all in PHP, a language I’ve not written in any significant manner since 2015. But it’s free, open-source, has a huge vault of useful plugins, and is malleable.
The querying/visualisation is still a work in progress. You can see how it’s going at this top-secret, hard to guess URL: https://playground.breakfastdinnertea.co.uk/weight/. With that said, the main method of data ingress has been up and running for well over 6 months now. It feels stable enough to talk about.
I was recently sharing this with a new Slack that I’ve joined (Si Jobling‘s On The Side slack). Si asked me about this particular integration, which I’ve taken as a prompt to make me write some of this down. If only so that when it next breaks, I have a little dummies guide to help me debug it all.
Here’s a bit of a dodgy and incomplete diagram showing what the information flow looks like. The “A” logo at the right is for the fantastic Astro framework which I use to render the playground. It is very early in its development, but a lot of fun to use. I’d encourage you to give it a look over if that’s your bag.
I won’t go into too much detail on the rendering part, if only because it’s still early days. But the rest of it, I’ll dig into a little below.
Custom Post Types
WordPress allows you to create your own data types using Custom Post Types. There are plugins to define these, but I like having the control you get from setting them yourself. Also, the custom post types API is dead simple. Custom types are exposable using WPGraphQL. Which gives the added bonus of not worrying about a domino chain of dependencies being up to date.
It registers the 3 main fields in that custom post type. The “sanitize_callback” property allows you to translate from Withings format to something UTC-like.
GraphQL allows me to export those post types for consumption off-site. From there, I can aggregate and filter data without having to write custom business logic. I didn’t choose GraphQL from the outset. I started using it because I was integrating against Gatsby, where the primary API is GraphQL. Fortunately, it’s a common interchange method, so I have stuck with it.
WPGraphQL is a plugin that allows you to expose data in your blog as a GraphQL schema. Once installed, it provides a single endpoint from which I can query for lots of different data at once. For simple integrations like a single feed of weights, this is likely overkill. But here it presents new interesting opportunities. I could one day, for example, create a no-extra-code FriendFeed-like aggregation. Because no, I still haven’t realised it’s not 2008 anymore.
It is a bit repetitive, but it allows me to tune the schema and set up some of the fields I want to filter and sort by. It also gives me extra resilience – I can keep the APIs inline myself, instead of being at the mercy of plugin authors staying up to date with each other. This generally isn’t a problem, but I have seen instances where one plugin in a dependency chain updates before the others follow. This has created some dead hard to debug problems, sometimes only solvable by pinning to specific plugin versions.
WordPress JSON Rest API
I don’t know how well known it is, but WordPress has an incredible REST API. Seriously, a really good API. Without doing anything special, once you’ve turned it on, you can post into it (and read from it, if you’re a traditionalist) and it will act on those posts as if you were entering data into its admin interface. The only problem you need to solve is authentication. There are a billion plugins for this – from OAuth2 through to basic HTTP. You should only use basic HTTP if you’re comfortable giving a third party service your username and password. (You should not be comfortable doing that, btw).
I use the fantastic IndieAuth plugin – it’s a bit of a hangover from when I integrated everything with the IndieWeb suite.I ultimately stepped away from that suite because all my customisations broke anytime the plugins updated. I should probably switch out to the OAuth2 plugin at some point, but this works for me as it stands. You generate a token with a scope of what it can do in your WP instance, and then use that token anytime you need to use a privilege.
IFTTT is a nice little service that pipes data between lots of different APIs. It has connectors for both Withings and WordPress, although the WordPress one only covers basic post types. So I make heavy use of its “Make A Web Request” connector.
A previous iteration of this web connector sent a request to a relay lambda I created to pass through the access token. But at the end of last year, IFTTT added the ability to create custom headers to your web requests. This means I can now retire that relay. I enjoy deleting code.
I have that hooked up to the Withings connector. And now, any time I weigh in, data starts trickling into WordPress.
And that’s pretty much it
At this point, I have a database slowly filling up with my weigh-ins from Withings:
It’s far from pretty, but it works. There is a TODO somewhere to fix that “Untitled” link. And it’s also queryable from GraphQL, in a much more digestible format:
Admittedly, this a fair bit of messing on, and it is absolutely too much work to justify a single content type. But I now have films, podcasts, and weigh-ins wired up and I intend to pull in more. I’m hoping that the benefits multiply as I add more stuff. But even if they don’t, the worst-case scenario is that I have a nice PESOS style backup for when services either shut down or arbitrarily kick me off.
And with that cheery thought, thanks very much for reading this far. I hope it is of some use to someone out there. If you want to dig into the implementation, it’s poorly documented but available in my mini-repo. Some pointers of where to look:
A couple of weeks back, I subtly released Jankteki 0.6, which brought with it a refactor, lots of instability, and a feature that I’m hoping will be useful to people that aren’t me: The ability to add user notes against a user on jinteki.net.
Have you ever added a friend on Jinteki and forgot why they’re there? Me too – now I don’t need to, I can add user notes. Here’s a demo of it in action:
As I said, it has brought with it a bit of instability, and it is still in its early days in terms of quality / interaction / etc. But I need to tell people about it, in case they think the software got buggier for no reason whatsoever. I also think it’s a nice basis to build further features upon, and want to get input on it as early as possible.
One of the silver linings of the breakages is that it has led to me tightening things up in the code base.
Scarily enough, there are now just shy of 1500 users of this extension now, and when it breaks, it seems to annoy people. Going below 5 stars in the chrome extension store gave me some real impetus to try to stop that all happening again.
There are now some tests, and build scripts, and stuff that makes it look like I have any idea what I’m doing, to aid with keeping things stable in the future.
If however, something does get through these very porous nets – head on over to http://github.com/simons/jankteki to raise a bug. There’s a proper readme over there now too, so that should give you all of the information needed to get problems sorted in good time.
And if you have no idea what I am talking about, but do know what Android: Netrunner and jinteki.net are, download the Jankteki extension from the Chrome webstore. It’s pretty useful.
The more eager eyed Jankteki users may have noticed the latest release. It consists of a small in-game ‘fixes’ panel where you can manually manipulate the game state without having to look up relatively arcane chat commands.
Here’s a video of this feature in action:
It’s the first interaction I’ve done with the game directly and was made pretty simple through Clojurescript / Om’s use of websockets. It was my first experience inspecting websocket frames through the Chrome Dev tools, and it must be said – I found it to be an utter joy. Hopefully this leaves room for other enhancements down the way.
In the meantime, the next slate of work is going to be around creating a fuller user-model – adding notes, annotations, that sort of thing. Maybe even the dreaded “shitlist” feature. More information on all of this / priorities etc can be found on my Trello roadmap
In the past year or so, the collectible card game, Android: Netrunner has just about taken over my life. I won’t go on about why here, but it’s great. You should play it.
One of the more popular ways of playing it is through an online open-source implementation, jinteki.net. Legally dubious, it’s a Clojure implementation of the game – involving a huge percentage of cards, providing automations for 1000s of rules and all sorts of interactions / exceptions / custom rulings. Needless to say, it’s a fantastic amount of effort being provided by a dedicated team of developers, led by Minh Tran.
As the game and platform have become more popular, the types of players on there have varied with it. More players is good, it’s more people and decks to play against. It also comes with a trolly underbelly. Ragequits and rude users used to be unheard of there, they’re now part of the landscape.
Navigating all of this can be frustrating at times. Whether it’s remembering who the bad players are, or looking for people you know to play (or watch play) – these are currently hard to do. And improving the platform at the same time as keeping up to date with card implementations is a slow process. So I built myself a tool – Jankteki, a chrome extension for jinteki.net.
I intend for it to be a suite of tools to make using jinteki.net a slightly better experience day-to-day. It’s called Jankteki, because the hacky nature of building features over a ReactJS UI makes for flickers, jank, and breakages all over the place. So use it at your own risk.
The only real feature currently implemented is friends, which you can see wonderfully narrated here:
Why not raise a PR?
I know a bit of Clojure, and I enjoy writing it – but I’m nowhere near being able to contribute to a project the scale of Jinteki. There’s also the fact that my roadmap might not correlate with the Jinteki roadmap, I have features planned to scratch my itch that might be a year or two away from being even discussed in the main repo.
Some of them might not even be the job of the main webapp.
And what is this roadmap?
The full in-progress roadmap can be seen on a Trello board I am working from. But here’s a quick overview (it is completely subject to change):
I’m aiming to put in a toolbar for running console commands (deal net damage, remove counters, etc) in order that you don’t have to remember syntax or look it up every time the game state needs manually adjusting.
Notes for users. I constantly forget who people are from their usernames, a notes field could track that. It could be used to remember what type of decks they play. Or even to note who “gg”s before they leave (this is disproportionately important to me).
The killer feature I’ve talked about, but not implemented yet (I half implemented it and took it out), is some sort of bad-players list. Just a way of flagging a user visually as someone to avoid in the future. It would purely be a personal shit-list, not like a communal feature – both by design, and cos it’s outside the scope of this thing.
Game log recording / analysis. This sits in the “I’m unsure this would ever become a feature of the webapp” column. But I love the idea of logging wins and losses vs opponents, factions – methods of losing – and obviously which deck you were playing as at the time. Just as a way of tracking how you’re getting on and what works.
Building this plugin hasn’t been smooth. React’s virtual DOM makes it a nightmare to manipulate constantly changing elements – an element that indicated a friend two seconds ago may contain something else completely.
This plugin is also tightly coupled to class and element names in the jinteki.net codebase, if they change something, we’re always going to be playing catch-up. I’m ok with that. I don’t know how Minh will react to the existence of this (or if he’ll even care :D), but I hope Jankteki will benefit the Jinteki.net team in both taking the demand for certain features, and also proving and disproving features before implementing them.
And yes – it’s a bit rubbish at the mo, but it proves a concept, so please be open minded with it. There are bugs with pinned friends not showing when navigating to pages, if you have issues, try going directly to http://jinteki.net/play or just refreshing the page – I am actively working on that.
Also – things are liable to break, pre version 1 (I don’t know what that will look like), I’m pushing to the web store as often as I’m adding features / fixing bugs. So prepare to be annoyed.
This post is about one of those not-so-common ones, namely the one used here on this site to print those shit-looking attempts at Piet Mondrian canvases on the background. It all came about because a) I saw a Mondrian piece and figured ‘those look kinda automatable’ and b) I’d been looking for an excuse to piss about with canvas.
But all of that really comes down to time available, there are a couple of other avenues that I’d still like to explore with regards to canvas, and they might get played with first – but the bottom line is that I’m writing in this blog more often in order to push myself to do more fun stuff to have more bollocks to write about.