logo

ShrimpWorks

// why am I so n00b?

Frequently while implementing HTTP API or other HTTP clients, you want to be able to test your client implementation against an actual HTTP service, which helps validate that your headers are set correctly, body is serialised appropriately, and responses are parsed as expected.

This can be done through the use of various additional libraries and mocking frameworks, however I’d argue that for most use cases, something that can simply validate and respond to an HTTP request is more than enough.

For such cases, the example below achieves just that. I find this much quicker and easier to set up, requiring no additional dependencies or learning of a new DSL, and test setup, execution and teardown are at least a factor of 3-4 times faster for the same test suite.

import com.sun.net.httpserver.HttpServer;

public class MyAPIClientTest {
	private static final int PORT = 56897;

	private HttpServer server;

	@BeforeEach
	public void before() throws IOException {
		this.server = HttpServer.create(new InetSocketAddress("127.0.0.1", PORT), 0);
		this.server.setExecutor(Executors.newSingleThreadExecutor());
		this.server.start();
	}

	@AfterEach
	public void after() {
		this.server.stop(0);
	}

	@Test
	public void shouldGetBalance() {
		// test setup - define expectations, set up expected response
		this.server.createContext("/za/pb/v1/accounts/172878438321553632224/balance", e -> {
			try {
				assertEquals(e.getRequestMethod(), "GET");
				assertEquals(e.getRequestHeaders().getFirst("Authorization"), "Bearer Ms9OsZkyrhBZd5yQJgfEtiDy4t2c");

				// JSON.toBytes() is a simple wrapper around a Jackson writeValueAsBytes() call
				byte[] result = JSON.toBytes(Map.of(
						"data", Map.of(
								"accountId", "172878438321553632224",
								"currentBalance", BigDecimal.valueOf(28857.76),
								"availableBalance", BigDecimal.valueOf(98857.76),
								"currency", "ZAR"
						),
						"links", Map.of("self", "/za/pb/v1/accounts/172878438321553632224/balance"),
						"meta", Map.of("totalPages", 1)
				));
				e.sendResponseHeaders(200, result.length);
				e.getResponseBody().write(result);
			} finally {
				e.close();
			}
		});
    
		// run test using my client against the API
		MyClient client = new MyClient("127.0.0.1", PORT);
		Balance = client.getBalance("172878438321553632224");
		// validate response client gathered, etc...
    }
}

As you can see, we’ve simply set up an expectation based on URL and method, validated that the Authorization was provided as expected, and then constructed a suitable response in the format the upstream API should be providing.

In place of the constructed API response I’ve used here, one could also easily place the contents of real or documented example API responses directly into the response, for your client to consume.

Seems like it’s only been 4 years since the last time I did this. Really feels a lot longer.

Anyway, I was becoming rather annoyed with trying to keep Ruby Gems and things updated and working, with plugins breaking and stuff with Jekyll, and I recently built a few other websites using Hugo, which made me quite keen on switching.

Hopefully everything’s carried over properly with all the right URLs and things. I’ve also tried to improve some things, and will be expanding the Projects section which I feel doesn’t represent everything that goes on.

As for the style: I just wanted to do something a little more fun and retro looking.

My first website (“Shrimp’s Maps”, the precursor to “ShrimpWorks” and the first website I ever made, which I used for sharing Unreal Tournament levels in early 2000), featured a silly “tech” look with blue gridlines and a cool angular header graphic thing, and all the text was monospace Courier New. Unfortunately it’s completely lost in time.

Once I upgraded to Wordpress in ~2005 after using a system called Geeklog for a while after Shrimp’s Maps, I adopted the content-with-sidebar style, which I switched away from at some point in favor of menus, so for more throwback fun, I’ve replicated that old layout style as well.

It’s perhaps a little busy, but everything’s going to “clean” and stripped down these days, a bit of busy-ness is not the end of the world :).

Here’s how we looked in 2005! Thanks archive.org!

Here’s how we looked in 2014!

And here’s what we’ve just come from.

There’s a strong tendency to want to run everything in Docker these days, especially if you’re trying to run something as an always-on service, since passing --restart=always to your run invocation or Docker Compose configuration ensures that running containers start back up after reboots or failures, and seems to involve a little less “black magic” than actually configuring software to run as services directly on a host.

The downside to this is the approach is that running a service in a container leads to significantly longer startup times, more memory and CPU overhead, lost logs, and in my opinion offer a false sense of security and isolation since most images are still configured to run as root, and more often than not large swathes of the host filesystem are mounted as volumes to achieve simple tasks.

There’s also a belief that your software will magically run anywhere - but if you’re writing Java (or any JVM language) code - that’s one of Java’s biggest selling points - it already has its own VM your code is running in, no most platforms!

Therefore, let’s see how easy it actually is to configure our software to run as a standard system service, providing us with the ability to run it as a separate restricted user, complete with standard logging configuration, and give us control over via standard service myservice start|status|restart|stop commands.

arrow Continue Reading ...

It’s a really simple thing, but I’ve been using this simple “pattern” for defining simple value objects for years, and it has served me well.

While there’s nothing particularly special about this style, I still see a significant amount of Java code needlessly following the JavaBeans style, when using these objects as Beans in the strict sense is not actually desired, intended, or required, and simply makes code needlessly verbose and leaves objects implemented as Beans open to abuse due to leaving their internal state open for mutation.

This pattern works well over traditional JavaBeans because:

  • it’s immutable - invaluable for concurrent or multi-threaded applications where you don’t want to give applications the ability to change values as they please
  • it’s neat - due to being immutable, there’s no need for superfluous “setters”, and if there are no setters, there’s no need for “getters”, so the code is dead simple and easy to work with
  • it’s portable - these objects are trivial to serialise using either Java Serialisation (or any of the preferable drop-in replacements), almost any serialisation library will be able to serialise them, and Jackson can deserialise them without any additional code
  • due to all the above, they’re also ideal for use as messages in event-driven systems

Here’s an example of a simple object implemented in this style:

import java.beans.ConstructorProperties;

public class User implements Serializable {
  private static final long serialVersionUID = 1L;

  public final String email;
  public final String name;
  public final Address address;

  @ConstructorProperties({ "email", "name", "address" })
  public User(String email, String name, Address address) {
    this.name = name;
    this.email = email;
    this.address = address;  
  }
}

This object is now serialisable and deserialisable via Java serialisation or better alternatives such as FST (just leave off Serializable if you don’t need that), as well as JSON serialisation libraries such as Jackson or GSON.

Unreal Archive

Over the past several months, I’ve been working on a project to provide a place to catalogue and preserve the vast amounts of user-created content the Unreal and Unreal Tournament community has been creating over the past 20+ years.

This has resulted in the Unreal Archive.

While it may seem a silly cause to invest so much time (and money) into, this stuff directly influenced the lives of myself and thousands of others. I would certainly not be in the profession I’m in, driving my car, living in my house, if not for the direct influence of working on Unreal Tournament maps, mods and community, and personal websites.

This stuff made many of us who we are today, and a lot of it has already been lost in time. The internet may not ever forget, but it certainly misplaces things in ways it can’t be found again.

A lot of content is in fact mirrored in various places on the internet, but it can be hard to download, as people generally don’t appreciate you mirroring 100s of gigabytes off their shared hosting.

Thus, the Unreal Archive is an initiative to gather up, index, and catalogue as much Unreal, UT99 and UT2004 content as possible. So far, we have maps, map packs, voices, skins, mutators, player models, as well as support for things such as patches, updates and drivers as well as a (currently very empty) section for written documents with the intent of providing guides, tutorials, manuals, and other related documented knowledge which also seems to get lost and forgotten.

The tech stack and some of the decisions involved may seem odd, but in keeping with the theme of longevity, preservation, and the general ease of losing things on the internet, these are some of my motivations:

  • statically generated content - the website is generated as a collection of plain HTML pages. this ensures no dependence on having to host a website with any dependency on any sort of back-end service beyond the simplest of HTTP servers. specific pains have been taken to ensure it works well with file:// local resources as well, so it doesn’t even need to be hosted!
  • written in Java - largely because I know it well enough to do this, but also because it’s not going anywhere soon, so the indexing and site generation capabilities will remain in action for a long time.
  • data stored as YAML files - a dead simple format that’s also easily human- readable. in 30 years when all the YAML parsers have died, if someone looks at these files, they’ll be easy to write new parsers for, if that’s ever needed.
  • the “database” is Git - easy to distribute amongst many people, and since this is primarily an archive, the data does not change rapidly enough to require anything more real-time.
  • the entire project is “licensed” under UNLICENSE, with the intent of it being as absolutely open as possible, for as long as possible.

As I’m collecting a lot of the data for the archive directly from the pieces of content themselves, a large part of implementing this also involved figuring out the Unreal Package data formats. Thankfully there are still several references for this hanging around, and many people have made their research on the topic public.

I’ve released a separate Unreal Package Library (Java) which some people may find useful. I’m using it to read map information, such as authors, player counts, titles, etc, export images such as screenshots and player portraits, as well as for parsing Unreal’s INT and UPL metadata files (more-or-less glorified INI files).

All the code for the project is up on GitHub, as is the content database.

UTStatsDB is a player and match statistics system for Unreal Tournament 99, 2003, 2004 and 3, which parses match logs generated by each game (sometimes requiring additional server-side mutators), and makes stats for each game available through a website.

The stats are also aggregated by player, map and server, allowing you to browse and analyse quite a number of in-depth stats for each.

The project was developed and maintained by Patrick Contreras and Paul Gallier between 2002 and around 2009, where the original project seems to have been abandoned some time after the release of UT3. (addendum: by some coincidence, after 9 years of inactivity, the original author did create a release a few days after my revival/release) Locating downloads (the download page is/was not working) or the source (their SCM system seems to require auth or is simply gone) was quite troublesome.

Thankfully it was released under GPL v2, so I’ve taken it upon myself to be this project’s curator (addendum: since the original author also made a new release, I may now need to look into a rename or major version bump), and have since released two new versions, 3.08 and 3.09 which focus firstly on getting PHP support up to scratch so it runs without issue on PHP 7+, as well as implementing PHP’s PDO database abstraction layer for DB access, rather than using each of the supported DB drivers (MySQL, MSSQL, SQLite) directly.

In addition to many other bug fixes and issues, I’ve thus far revised the presentation significantly, provided Docker support, improved performance of several SQL operations by implementing caching and better queries, etc.

UTStatsDB can be found on GitHub, where the the latest release can also be downloaded.

A live example of UTStatsDB in action can be found at the UnrealZA stats site.

Said a teary farewell to some old things, got a new thing. Very happy with the new thing.

With all the talk of Unreal Tournament 4 possibly being cancelled one of these days, due to Epic’s runaway success with Fortnite, I’ve decided there’s really no reason to not be playing UT99.

Thus, we set about trying to run it on modern hardware, with a modern Linux installation.

As much as this is about setting things up on Linux, it’s also partially my own attempt at some knowledge preservation, as a lot of this stuff ends up being forgotten or lost over time (it’s been almost 20 years! a lot of the old sites and things you expect to find this info on simply do not exist anymore :()

This is part one of two, and will focus on installing and running the game using Wine.

arrow Continue Reading ...

I recently wanted to set up a couple of rough monitoring services to keep track of simple server status, load, disk etc. While there are options available like Munin which can do this by installing agents on the machines to be monitored, I wanted something a little simpler and more portable.

I’m quite fond of the StatsD + Graphite + Grafana stack, which is quite easy to run thanks to Kamon’s grafana_grafite Docker image, and I realised you can actually quite simply write counters, gauges and timers to StatsD using nothing but the standard Linux tools nc and cron.

For example, every minute on each server being monitored, a simple cron job is executed which uses nc to write a bunch of information to my StatsD service:

#!/bin/bash

HOST=$(hostname)

STAT_HOST="statsd-host"
STAT_PORT=8215

# load average
echo "load.$HOST.avg:`cat /proc/loadavg | cut -d ' ' -f 1 | awk '{print $1*100}'`|g" | nc -w 1 -u $STAT_HOST $STAT_PORT

# memory
echo "memory.$HOST.perc.free:`free | grep Mem | awk '{print $3/$2 * 100.0}'`|g" | nc -w 1 -u $STAT_HOST $STAT_PORT
echo "memory.$HOST.bytes.total:`free -b | grep Mem | awk '{print $2}'`|g" | nc -w 1 -u $STAT_HOST $STAT_PORT
echo "memory.$HOST.bytes.used:`free -b | grep Mem | awk '{print $3}'`|g" | nc -w 1 -u  $STAT_HOST $STAT_PORT

# disk
echo "disk.$HOST.kbytes.total:`df -k --output=size / | grep -v [a-z]`|g" | nc -w 1 -u $STAT_HOST $STAT_PORT
echo "disk.$HOST.kbytes.used:`df -k --output=used / | grep -v [a-z]`|g" | nc -w 1 -u $STAT_HOST $STAT_PORT
echo "disk.$HOST.kbytes.avail:`df -k --output=avail / | grep -v [a-z]`|g" | nc -w 1 -u $STAT_HOST $STAT_PORT

# mail queues
for i in maildrop hold incoming active deferred bounce; do echo "postfix.$HOST.queues.${i}:`find /var/spool/postfix/${i} -type f | wc -l`|c"; done | nc -w 1 -u $STAT_HOST $STAT_PORT

It’s perhaps a bit inefficient in places, but gets the job done fairly well. One minute resolution may be a bit rough, but it’s sufficient for most of these data points which don’t change too dramatically over time.

Some other more specific variations include HTTP accesses, ping times, etc. Pretty much any parameter you can parse down to a single number can be published as a counter, gauge or timer to StatsD, and then neatly graphed over time.

I have finally decided to release version 1.0 of Aurial, my implementation of a music player/client for the Subsonic music server.

I started this around two years ago, some time after switching my primary desktop from Windows to Linux, and I really missed foobar2000 - it has been my primary music player ever since. Unfortunately I have an irrational aversion to using Wine to run Windows applications, and none of the native music players on Linux felt good to me. As I already ran a Subsonic music server, I thought I’d just make use of that.

The existing browser-based clients for Subsonic were either too basic, or the state of their code and some implementation features made me uncomfortable. I just wanted a nice music player that allowed me to browse my collection similar to how I did in foobar2000 (using Subsonic’s ID3 tag based APIs, rather than the directory-based browsing offered by other clients), perhaps manage playlists, make ephemeral queues, and importantly, scrobble played tracks.

Podcasts, videos, and other things some clients support don’t interest me at all, and are a bit out of scope of a foobar2000-like client I believe.

Aurial allows me to build a music player the way I prefer to browse, manage and play music (which admittedly, is quite heavily influenced by my prior foobar200 configuration and usage habits).

aurial

This was my first attempt at a React application, and it started off simply enough, with JSX transpiling and stuff happening directly in the browser. At some point Bable was no longer available for browsers, which led to my adoption of Webpack (and eventually Webpack 2) for producing builds.

This also led to things like needing some sort of CI, and I’ve recently begun producing builds via TravisCI which automates building the application, and deploying it to GitHub Pages, which I think is pretty neat.

I also got to play with HTML5’s <audio/> a bit, as the player library I was using previously had some reliance on Flash, and was occasionally tricky to coax into using HTML rather than that. The result is infinitely smaller and less complex audio playback implementation (it’s amazing how much easier life is when you ignore legacy support).

Anyway, altogether it’s been fun, and as I’m using it constantly, it’s always evolving bit by bit. Hopefully someone else finds it useful too.