ShrimpWorks

TLDR; There’s now an MQTT Client implementation written in UnrealScript

I’ve been doing a bit of stuff in UnrealScript recently, and reacquainting myself with it.

Something I’ve always been aware of, but have never really looked at in much detail, is that it has an actual TCP client you can extend to implement whatever remote communications protocol you’d like.

For whatever reason MQTT popped up as my candidate to play with, with the thought that you’d be able to publish in-game events to some topics and build interesting things with (the first thing that came to mind was a match stats collection service which doesn’t rely on the traditional process of log scraping), in addition to allowing in-game functionality to respond to incoming events by way of topic subscriptions. And being something targeted at supporting very simple IoT devices, the protocol should be fairly easy to work with.

Thus, we jump into the comprehensive but sometimes strangely documented MQTT version 5.0 protocol documentation to find out how it works. It is indeed fairly straight forward.

Now to find out how the Unreal Engine 1 TcpLink class works. Keeping in mind this was implemented in the late 90s, data was smaller, data structures were generally less complex, and not everything was networked.

Firstly, opening a connection is a bit of a process.

1. Request resolution of a hostname by Resolve(hostname);
2. An event, Resolved(ipAddr) will fire, with the resolved IP address (integer representation)
3. Then, you manually bind the client’s ephemeral port with, a simple BindPort - this immediately returns a bound port number
4. If your port was bound, you can call Open(ipAddr)
5. An event, Opened() will fire when the connection is established, and you may now send and receive data.

So slightly more manual than a higher level implementation in most modern languages, but when you consider the engine is single-threaded, it’s quite a reasonable process to get around blocking on network I/O.

Sending data is fairly simple, via the SendBinary(count, bytes[255) function. If you have more than 255 bytes of data to send, it’s a simple matter of re-filling the 255 byte array and repeating, until you’re done.

Initially, I tried to use the ReceivedBinary(count, bytes[255]) event for processing inbound data, but due to a known engine bug, this only serves up garbage data, so we’re left relying on ReadBinary(count, bytes[255]) which similar to sending, you can call multiple times on a re-usable buffer until the function returns 0 bytes read.

To make working with data using these processes a bit easier, I implemented a ByteBuffer class, modelled exactly after Java NIO’s ByteBuffer. I feel allocating a re-usable fixed size buffer array which can be compact()ed, followed by a series of put(bytes[255]), and an eventual flip() to allow reading is both performant and simple to reason about.

Implementing this ByteBuffer class also gave me a better understanding of the Java ByteBuffer in the process, even though I’ve been using it for years, it helps to reinforce understanding of some of the implementation details.

So, using this process of connecting, filling buffers, parsing them according to the specification, sending responses and so on gives us a nice suite of functionality within the client itself. We also want to support custom subscribers which allow other code and mods to receive events from MQTT subscriptions.

UnrealScript of course does not have the concept of Interfaces, but does support inheritance, so by extending MQTTSubscriber, custom code can do what it needs to, using the receivedMessage(topic, message) on subclasses of that class.

UnrealScript also provides a very neat child/owner relationship between spawned Actors, and so we’re making use of this to attach subscribers to the MQTT Client. Two standard events the MQTTClient makes use of for this are GainedChild(child) and LostChild(child), which notify the client when a subscriber has been spawned as a child of the client. On gaining a child, the client can automatically establish a subscription for the subscriber’s topic, so it can start receiving those messages. Similarly, when it loses a child, the client can automatically clean up any related topic subscriptions.

This process allows neat life-cycle management of both the subscriber classes themselves, as well as the actual server-side topic subscription, by leveraging built-in language/system functionality.

Overall, I’m happy with the end result, both in final utility of the implementation, and it’s usability for users of the classes involved. It was also pretty educational and enlightening to see how this old single-threaded engine deals with network connectivity, and the process of building the ByteBuffer also helped reinforce my understanding of Java’s implementation as well.

Frequently while implementing HTTP API or other HTTP clients, you want to be able to test your client implementation against an actual HTTP service, which helps validate that your headers are set correctly, body is serialised appropriately, and responses are parsed as expected.

This can be done through the use of various additional libraries and mocking frameworks, however I’d argue that for most use cases, something that can simply validate and respond to an HTTP request is more than enough.

For such cases, the example below achieves just that. I find this much quicker and easier to set up, requiring no additional dependencies or learning of a new DSL, and test setup, execution and teardown are at least a factor of 3-4 times faster for the same test suite.

import com.sun.net.httpserver.HttpServer;

public class MyAPIClientTest {
	private static final int PORT = 56897;

	private HttpServer server;

	@BeforeEach
	public void before() throws IOException {
		this.server = HttpServer.create(new InetSocketAddress("127.0.0.1", PORT), 0);
		this.server.setExecutor(Executors.newSingleThreadExecutor());
		this.server.start();
	}

	@AfterEach
	public void after() {
		this.server.stop(0);
	}

	@Test
	public void shouldGetBalance() {
		// test setup - define expectations, set up expected response
		this.server.createContext("/za/pb/v1/accounts/172878438321553632224/balance", e -> {
			try {
				assertEquals(e.getRequestMethod(), "GET");
				assertEquals(e.getRequestHeaders().getFirst("Authorization"), "Bearer Ms9OsZkyrhBZd5yQJgfEtiDy4t2c");

				// JSON.toBytes() is a simple wrapper around a Jackson writeValueAsBytes() call
				byte[] result = JSON.toBytes(Map.of(
						"data", Map.of(
								"accountId", "172878438321553632224",
								"currentBalance", BigDecimal.valueOf(28857.76),
								"availableBalance", BigDecimal.valueOf(98857.76),
								"currency", "ZAR"
						),
						"links", Map.of("self", "/za/pb/v1/accounts/172878438321553632224/balance"),
						"meta", Map.of("totalPages", 1)
				));
				e.sendResponseHeaders(200, result.length);
				e.getResponseBody().write(result);
			} finally {
				e.close();
			}
		});
    
		// run test using my client against the API
		MyClient client = new MyClient("127.0.0.1", PORT);
		Balance = client.getBalance("172878438321553632224");
		// validate response client gathered, etc...
    }
}

As you can see, we’ve simply set up an expectation based on URL and method, validated that the Authorization was provided as expected, and then constructed a suitable response in the format the upstream API should be providing.

In place of the constructed API response I’ve used here, one could also easily place the contents of real or documented example API responses directly into the response, for your client to consume.

There’s a strong tendency to want to run everything in Docker these days, especially if you’re trying to run something as an always-on service, since passing --restart=always to your run invocation or Docker Compose configuration ensures that running containers start back up after reboots or failures, and seems to involve a little less “black magic” than actually configuring software to run as services directly on a host.

The downside to this is the approach is that running a service in a container leads to significantly longer startup times, more memory and CPU overhead, lost logs, and in my opinion offer a false sense of security and isolation since most images are still configured to run as root, and more often than not large swathes of the host filesystem are mounted as volumes to achieve simple tasks.

There’s also a belief that your software will magically run anywhere - but if you’re writing Java (or any JVM language) code - that’s one of Java’s biggest selling points - it already has its own VM your code is running in, no most platforms!

Therefore, let’s see how easy it actually is to configure our software to run as a standard system service, providing us with the ability to run it as a separate restricted user, complete with standard logging configuration, and give us control over via standard service myservice start|status|restart|stop commands.

Continue Reading ...

It’s a really simple thing, but I’ve been using this simple “pattern” for defining simple value objects for years, and it has served me well.

While there’s nothing particularly special about this style, I still see a significant amount of Java code needlessly following the JavaBeans style, when using these objects as Beans in the strict sense is not actually desired, intended, or required, and simply makes code needlessly verbose and leaves objects implemented as Beans open to abuse due to leaving their internal state open for mutation.

This pattern works well over traditional JavaBeans because:

• it’s immutable - invaluable for concurrent or multi-threaded applications where you don’t want to give applications the ability to change values as they please
• it’s neat - due to being immutable, there’s no need for superfluous “setters”, and if there are no setters, there’s no need for “getters”, so the code is dead simple and easy to work with
• it’s portable - these objects are trivial to serialise using either Java Serialisation (or any of the preferable drop-in replacements), almost any serialisation library will be able to serialise them, and Jackson can deserialise them without any additional code
• due to all the above, they’re also ideal for use as messages in event-driven systems

Here’s an example of a simple object implemented in this style:

import java.beans.ConstructorProperties;

public class User implements Serializable {
  private static final long serialVersionUID = 1L;

  public final String email;
  public final String name;
  public final Address address;

  @ConstructorProperties({ "email", "name", "address" })
  public User(String email, String name, Address address) {
    this.name = name;
    this.email = email;
    this.address = address;  
  }
}

This object is now serialisable and deserialisable via Java serialisation or better alternatives such as FST (just leave off Serializable if you don’t need that), as well as JSON serialisation libraries such as Jackson or GSON.

Over the past several months, I’ve been working on a project to provide a place to catalogue and preserve the vast amounts of user-created content the Unreal and Unreal Tournament community has been creating over the past 20+ years.

This has resulted in the Unreal Archive.

While it may seem a silly cause to invest so much time (and money) into, this stuff directly influenced the lives of myself and thousands of others. I would certainly not be in the profession I’m in, driving my car, living in my house, if not for the direct influence of working on Unreal Tournament maps, mods and community, and personal websites.

This stuff made many of us who we are today, and a lot of it has already been lost in time. The internet may not ever forget, but it certainly misplaces things in ways it can’t be found again.

A lot of content is in fact mirrored in various places on the internet, but it can be hard to download, as people generally don’t appreciate you mirroring 100s of gigabytes off their shared hosting.

Thus, the Unreal Archive is an initiative to gather up, index, and catalogue as much Unreal, UT99 and UT2004 content as possible. So far, we have maps, map packs, voices, skins, mutators, player models, as well as support for things such as patches, updates and drivers as well as a (currently very empty) section for written documents with the intent of providing guides, tutorials, manuals, and other related documented knowledge which also seems to get lost and forgotten.

The tech stack and some of the decisions involved may seem odd, but in keeping with the theme of longevity, preservation, and the general ease of losing things on the internet, these are some of my motivations:

• statically generated content - the website is generated as a collection of plain HTML pages. this ensures no dependence on having to host a website with any dependency on any sort of back-end service beyond the simplest of HTTP servers. specific pains have been taken to ensure it works well with file:// local resources as well, so it doesn’t even need to be hosted!
• written in Java - largely because I know it well enough to do this, but also because it’s not going anywhere soon, so the indexing and site generation capabilities will remain in action for a long time.
• data stored as YAML files - a dead simple format that’s also easily human- readable. in 30 years when all the YAML parsers have died, if someone looks at these files, they’ll be easy to write new parsers for, if that’s ever needed.
• the “database” is Git - easy to distribute amongst many people, and since this is primarily an archive, the data does not change rapidly enough to require anything more real-time.
• the entire project is “licensed” under UNLICENSE, with the intent of it being as absolutely open as possible, for as long as possible.

As I’m collecting a lot of the data for the archive directly from the pieces of content themselves, a large part of implementing this also involved figuring out the Unreal Package data formats. Thankfully there are still several references for this hanging around, and many people have made their research on the topic public.

I’ve released a separate Unreal Package Library (Java) which some people may find useful. I’m using it to read map information, such as authors, player counts, titles, etc, export images such as screenshots and player portraits, as well as for parsing Unreal’s INT and UPL metadata files (more-or-less glorified INI files).

All the code for the project is up on GitHub, as is the content database.

UTStatsDB is a player and match statistics system for Unreal Tournament 99, 2003, 2004 and 3, which parses match logs generated by each game (sometimes requiring additional server-side mutators), and makes stats for each game available through a website.

The stats are also aggregated by player, map and server, allowing you to browse and analyse quite a number of in-depth stats for each.

The project was developed and maintained by Patrick Contreras and Paul Gallier between 2002 and around 2009, where the original project seems to have been abandoned some time after the release of UT3. (addendum: by some coincidence, after 9 years of inactivity, the original author did create a release a few days after my revival/release) Locating downloads (the download page is/was not working) or the source (their SCM system seems to require auth or is simply gone) was quite troublesome.

Thankfully it was released under GPL v2, so I’ve taken it upon myself to be this project’s curator (addendum: since the original author also made a new release, I may now need to look into a rename or major version bump), and have since released two new versions, 3.08 and 3.09 which focus firstly on getting PHP support up to scratch so it runs without issue on PHP 7+, as well as implementing PHP’s PDO database abstraction layer for DB access, rather than using each of the supported DB drivers (MySQL, MSSQL, SQLite) directly.

In addition to many other bug fixes and issues, I’ve thus far revised the presentation significantly, provided Docker support, improved performance of several SQL operations by implementing caching and better queries, etc.

UTStatsDB can be found on GitHub, where the the latest release can also be downloaded.

A live example of UTStatsDB in action can be found at the UnrealZA stats site.

I have finally decided to release version 1.0 of Aurial, my implementation of a music player/client for the Subsonic music server.

I started this around two years ago, some time after switching my primary desktop from Windows to Linux, and I really missed foobar2000 - it has been my primary music player ever since. Unfortunately I have an irrational aversion to using Wine to run Windows applications, and none of the native music players on Linux felt good to me. As I already ran a Subsonic music server, I thought I’d just make use of that.

The existing browser-based clients for Subsonic were either too basic, or the state of their code and some implementation features made me uncomfortable. I just wanted a nice music player that allowed me to browse my collection similar to how I did in foobar2000 (using Subsonic’s ID3 tag based APIs, rather than the directory-based browsing offered by other clients), perhaps manage playlists, make ephemeral queues, and importantly, scrobble played tracks.

Podcasts, videos, and other things some clients support don’t interest me at all, and are a bit out of scope of a foobar2000-like client I believe.

Aurial allows me to build a music player the way I prefer to browse, manage and play music (which admittedly, is quite heavily influenced by my prior foobar200 configuration and usage habits).

This was my first attempt at a React application, and it started off simply enough, with JSX transpiling and stuff happening directly in the browser. At some point Bable was no longer available for browsers, which led to my adoption of Webpack (and eventually Webpack 2) for producing builds.

This also led to things like needing some sort of CI, and I’ve recently begun producing builds via TravisCI which automates building the application, and deploying it to GitHub Pages, which I think is pretty neat.

I also got to play with HTML5’s <audio/> a bit, as the player library I was using previously had some reliance on Flash, and was occasionally tricky to coax into using HTML rather than that. The result is infinitely smaller and less complex audio playback implementation (it’s amazing how much easier life is when you ignore legacy support).

Anyway, altogether it’s been fun, and as I’m using it constantly, it’s always evolving bit by bit. Hopefully someone else finds it useful too.

The title’s quite silly unfortunately, but I was recently doing some experimentation with uploading images to CouchDB directly from a browser. I needed to scale the images before storage, and since I was talking directly to the CouchDB service without any kind of in-between API services or server-side scripts, needed a way to achieve this purely on the client.

Thanks to modern APIs available in browsers, combined with a Canvas, it’s actually reasonably simple to process a user-selected image prior to uploading it to the server without the need for any third-party libraries or scripts.

Continue Reading ...

After almost exactly two years since the last release of Out of Eve, here is version 3.0.

As may be noted from the release note, the main goal of this release is to catch everything up with the current state of EVE, it’s API, and the static data dump.

Along the way some new stuff was also added an improved, like the new menu system which allows access to all your characters, so there’s no need to switch between them and then view detail pages, and the introduction of memcached caching, which stores and retrieves entities loaded from the static database dump, reducing page load times and database accesses (a single page load may result in hundreds of individual MySQL queries).

I’m rather pleased with this release, and it seems a lot more solid than most before.

I’ve also got the public Out of Eve website back up, now featuring HTTPS courtesy of Letsencrypt, at last.

It seems surprisingly difficult to find a simple lightbox implementation which doesn’t rely on jQuery. I wanted something simple for this site, but did not want to have to re-do any HTML, so came up with a basic JavaScript and CSS solution.

This also turned out to be a useful lesson in “modern” jQuery-less DOM manipulation. I found 10 Tips for Writing JavaScript without jQuery quite useful in this regard.

For the Lightbox/pop-up itself, the Pure CSS Lightbox by Gregory Schier served as an excellent reference and starting point.

Continue Reading ...

More a curiosity than an actual useful project, I just had an Idea I wanted to try out, and this is the result.

This Java application (or library, if you want to include it in your own project) simply takes a source image, a couple of optional parameters, and outputs a new image with a halftone- like effect.

Briefly, works by stepping through the pixels of the source image at an interval defined by the dot size specified, samples the brightness of that pixel, and draws a circle onto the destination image, scaled according to the source pixel brightness.

For reference, take a look at the java.awt Graphics2D, Image and BufferenImage classes. It’s really nice to half a whole bunch of image processing and drawing capabilities available within the standard library, rather than needing to rely on external things (as I recently discovered to be the case with Ruby - pretty much all image processing is done via an ImageMagick dependency).

The source, documentation and a download are available from the image-halftone GitHub project page.

Now that we have dependency management with Ivy working along with everything else covered before, we’ve covered almost everything required to start building real projects with Ant.

Another thing any real project should have, is unit tests. Thankfully, using the scaffolding already put in place in earlier parts of this series, integrating a JUnit testing task into our existing build script is really straight-forward.

Continue Reading ...

So far, we’ve covered the basics of creating a re-distributable .jar package suitable for use as a library, and building a Jar file which can be run by a user or server process.

A major part of any non-trivial application these days is the inclusion and re-use of 3rd party libraries which implement functionality your applications require. When a project starts, it’s probably easy enough to manually drop the odd jar library into a lib directory and forget about it, but maintaining a large application which depends on many libraries, which in turn depend on additional libraries for their own functionality, it can quickly turn into a nightmare to manage.

To solve this problem, many dependency management tools have been introduced, most notably, Apache Maven. Maven however, is so much more than just a dependency management tool, and is actually intended to manage your entire project structure. I believe however, the combination of Ant and Ivy provides far more flexibility, extensibility and control over your build and dependency management processes.

So, let’s integrate Apache Ivy into our Ant script as we left it in part 2.

Continue Reading ...

In part 1, we went over the basics of using Ant to create a redistributable .jar file, suitable for use as a library in other projects. A lot of the time however, you’re probably going to want to be building things which can actually be run as regular Java applications.

Once again, the code for this tutorial is available in GitHub. More usefully, you may want to see the diff between the part 1 script and the new one.

Here’s a quick explanation of what we’ve done to achieve an executable jar file:

Continue Reading ...

Apache Ant is a general-purpose build tool, primarily used for the building of Java applications, but it is flexible enough to be used for various tasks.

In the Java world at least, Ant seems to be largely passed over for the immediate convenience and IDE support of Maven, however long term, I believe a good set of Ant scripts offer far more flexibility and room for tweaking your build processes. The downside is that there’s a lot of stuff you need to learn and figure out and build by hand.

In this series of tutorials, I’ll try to document the process of learning I’ve gone through building and maintaining Ant build files, from the most basic of “just compile my stuff” steps to automatic generation of JavaDoc output, dependency management using Ant’s companion, Ivy, unit testing using JUnit, and integrating with some additional tools I’ve been using, such as Checkstyle and FindBugs.

For part 1 of this tutorial, I’ve created a simple Hello World library. It doesn’t have a main executable itself, the goal of this is to produce a .jar file we can include in other projects, to start our Ant script off fairly simply.

The source for this project can be found in GitHub. Here’s the breakdown of everything going on in this project:

Continue Reading ...

While it amounts to little more than a Markdown renderer in this case (with only a single, relatively small document on show), using GitHub pages and a simple branch from master, you can now see Markdocs in action.

Here’s a thing I’ve been wanting for a while now, and have been unable to something to suite my needs (well, more wants than needs, I guess). I end up generating a lot of text/documentation for various things (both at home and work), normally spread around a little - project descriptions and introductions in READMEs, APIs and design plans in wikis, sometimes random files, etc, and wanted the ability to consolidate these into collections that could be nicely presented, either publicly or for team reference.

My preferred requirements, which were not met by existing solutions such as Sphinx, Read the Docs, Beautiful docs and Daux.io are:

• No need to pre/post processing the input documents as a separate “compile” or parsing step
  • Should use existing plain Markdown documents as input and format output at runtime only
• Along with the above, the documents should be “live” - if I change the source file, I don’t want to “recompile” my documentation pages, they should reflect changes by default
• Not a hosted solution
  • Particularly, something anyone can drop on a private server (work environment) or whatever they want to do with it
• No server-side requirements beyond simple HTTP file serving
• I may be out of the JavaScript development scene, but what’s up with requiring users to use a dozen different build systems and dependency management frameworks to use your JavaScript app these days?
  • Seriously, the attraction used to be that you could simply drop a couple of HTML, CSS and JS files in your www-root and magic came out. Get off my lawn!

My solution is Markdocs - a simple HTML and JavaScript application for organising individual Markdown documents as a documentation collection.

See the README on the Markdocs GitHub page for usage instructions. Basically, you define the documents to include via a simple JSON file, which is loaded at runtime. The required documents are then loaded using jQuery, parsed at runtime with Marked right in the user’s browser, and a table of contents and the documents themselves are generated and presented using a simple Semantic UI interface.

At present it’s perfectly usable, but there are still a couple of things I want to improve and add, including suitable inter-document linking (while not enforcing any magic link syntax - your stand-alone document should still work as stand-alone documents) and ability to provide links to the individual source documents as well as an “Edit” link (for example, let you define a link to the editable document on GitHub).

Will update as it progresses.

I’ve become fond of using nginx on my development machines, rather than a full Apache.

There are no explicit options built-in which allow something along the same lines as Apache’s userdir, however it’s easy enough to tweak the default configuration to support that behaviour without the need for external modules.

I also do some PHP dabbling from time to time, so need to enable that as well.

Install the required bits:

$ sudo aptitude install nginx php5-fpm

Configure nginx (the below is my customised and cleaned out server definition):

/etc/nginx/sites-available/default

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    root /var/www/html;

    # Add index.php to the list if you are using PHP
    index index.html index.htm index.nginx-debian.html index.php;

    server_name _;

    # PHP support in user directories
    location ~ ^/~(.+?)(/.*\.php)$ {
        alias /home/$1/public_html;
        autoindex on;

        include snippets/fastcgi-php.conf;

        try_files $2 = 404;
        fastcgi_pass unix:/var/run/php5-fpm.sock;
    }

    # PHP support in document root
    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
        fastcgi_pass unix:/var/run/php5-fpm.sock;
    }

    # User directories in /home/user/public_html/
    # are accessed via http://host/~user/
    location ~ ^/~(.+?)(/.*)?$ {
        alias /home/$1/public_html$2;
        autoindex on;
    }
}

I also had to make a change to /etc/nginx/snippets/fastcgi-php.conf, to comment out the following line:

#try_files $fastcgi_script_name =404;

After restarting the nginx service (also make sure the php5-fpm service is running), you will be able to serve HTML and PHP files from your ~/public_html directory.

I wanted to add a unit conversion plugin to ZOMB and would really have liked to use an off-the-shelf existing API, but because this didn’t seem to exist in a nice hosted format already - I had to make it :).

The Units API is written in PHP, and is intended to provide an extremely simple and easy-to-use HTTP API for the conversion between various units of measure. Usage documentation is available on the project’s Github page.

I’m also hosting a publicly usable version, at the following URL, so hopefully next time someone needs this they don’t need to reinvent the wheel (again, refer to documentation linked above for usage):

• http://tools.shrimpworks.za.net/units-api/convert.php

As an aside, this project served as my first introduction to PHPUnit for PHP unit testing, and CI is once again provided by Drone.io which has performed admirably. Design-wise, it was another exercise in defining the public-facing API before a line of code was written, which served as an excellent guide and source of documentation as I worked on it (plus, there’s no need to worry about writing documentation when you’re done :D).

As mentioned, I’ve resurrected an old idea, and began work on it as a bit of a learning/practice exercise. I think it’s working out rather well.

The primary application itself, hosted on GitHub here, is essentially complete, barring the ability to persist your plugin configuration (pfft, who needs to store things anyway).

Some stuff learned along the way:

API-driven development:

Designing the external-facing API (actually defining and completely documenting the exact request and response data structures, not just “there will be a request that does things and a response that looks something like X”) was a huge help. Defining the API allows you to see how the system will actually be used up-front before writing a single line of code, and allows you to easily spot gaps and shortcomings. Once done, the “user documentation” becomes the same documentation I used to implement the back-end, which made it incredibly easy.

Git:

Still learning, getting more comfortable with it. IntelliJ IDEA has excellent built-in Git support out-the-box, and although painful to use in a Windows shell (it’s basically Bash, inside cmd.exe), I’m getting more used to the Git CLI.

Free/online continuous integration:

Initially, I started off using Travis-CI. This requires you to store a “.travis.yml” file within the root of your Git repository which I was rather uncomfortable with (I don’t like “external” metadata type things hanging around in my source repository). As an alternative, I’ve switched to using Drone.io, which just “feels” like a nicer solution. It also has additional features like the ability to store artefacts for download, or publish your artefacts to external services or servers - so you could have successful builds automatically deploy the latest binaries.

Persistence/Storage:

Persistence is hard, so once you start a service up, it should run indefinitely so you never need to write anything to disk. Sigh. Also, this part was not designed at all up-front, and my flailing around trying to get a workable solution is evidence of the need for proper design and planning before jumping in with code.

Aside from that, there are additional projects which were spawned:

zomb-web

The first front-end for ZOMB. A simple single-page HTML UI. Had some good practice remembering how to HTML and Javascript here…

zomb-plugins

A growing collection of plugins for ZOMB. At present, they’re all PHP (again, refreshing old skills…) and pretty simple. Currently, there’s time (simple current time/time-zone conversion), lastfm (see what someone’s currently listening to, find similar artists), weather (current and forecast conditions for a given city) and currency (simple currency conversion).

None of the above cannot be achieved without a simple web search, so next up I’d like to create a CLI client - weather updates in your terminal!

(Re-)Introducing ZOMB, an IRC bot back-end, which I planned, started work on some years ago, then promptly lost interest after it became vagely usable.

The general idea of ZOMB (like “zomg”, but it’s a bot, not a god [maybe version 3], and it sounds like “zombie” which is cool too) is to provide a client interface-independent bot framework, where bot functionality can be implemented in remotely hosted plugin scripts/applications, unlike a traditional bot where normally you’d need all the code running on one user’s machine/server.

Being interface-independent means a ZOMB client (the thing a user will use to interact with ZOMB) may be an IRC bot, a CLI application, or a web page. Since I’ve been less active on IRC than I’d like lately, the additional options would be useful to me personally, but since almost nobody uses IRC at all any more, ZOMB should hopefully be useful outside of that context.

So how does ZOMB work? From a user’s point of view, it’s exactly like a traditional bot - you issue a query consisiting of the plugin you want to execute, the command to call, along with command arguments. For example, you’d ask a ZOMB bot:

> weather current johannesburg

Where “weather” is the plugin, “current” is a command provided by the weather plugin, and “johannesburg” is an argument. In response to this, ZOMB would provide you a short text result, something like this:

> The weather in Johannesburg is currently 22 degrees and partly cloudy

In the background, ZOMB looked at the input, found that it knew of the “weather” plugin, and made an HTTP request to the remote plugin service passing the command and arguments along. The plugin then did what it needed to do to resolve the current weather conditions for Johannesburg, and returned a result, which ZOMB in turn returned to the requesting client.

As always, a new project provides some practice/learning opportunities:

• API driven development I know what I want ZOMB to be able to to, so I began by defining the client and plugin APIs, around which the rest of the design must fit. I normally write a bunch of code, then stick an API on top of it, but trying it the other way around this time. Seems to be working.
• Test driven development just to keep practicing :-)
• Git and Github since we’re hopefully going to be using Git at work in the near future, best to get some practice in.
• Custom Ant and Ivy build scripts I like Ant and Ivy and need to practice configuring and maintaining them from scratch.
• Travis-CI continuous integration for Github projects, since it’s cool to have a green light somewhere showing that stuff’s not broken, and I’ve never used any CI stuff outside of work.
• More granular commits committing smaller changes more often - I don’t know if this is a good thing or not, but seeing how it works out
• All on Windows I haven’t really built a proper project on Windows for years :D

After reading a lot of rants and essays about developers, their working environments, tools, the process of getting their work done (in relation to “the business” side of things), and career opportunities, I find myself wondering; do we just whine too much about it all, or do we really have to put up with so much more crap than other industries or professions?

Do we suffer from an inflated sense of entitlement - did we (and are expected to continue to) study, learn and practice for years only to end up like battery chickens, churning out code, or are we “deserving” of extra perks, privileges and financial reward?

Yes, developers are almost entirely responsible for every cent made (and even more so for every loss!) in almost every industry these days, and in most cases, I really don’t believe they get the credit they deserve (new product launched, management team praised and treated to expensive outing, golf day, or conference to show it off while we are at work hacking on the Next Big Thing). But we’re not alone in our suffering and expectations of better things.

Somewhere there’s the Accounting Drone capturing all the cents made possible by the developers. The Accounting Drone also has a shitty manager who expects them to capture more cents every day, under shitty conditions with just as shitty deadlines, for very poor pay. This guy’s job security is also near non-existent. Nobody capturing all the cents means us whiny developers don’t get paid. Maybe there are some rants and essays about the unreasonable conditions and tools the Accounting Drone must endure.

Elsewhere, there is the Sales Bro. Generally the bane of every developer’s life (worse than project managers!). It’s all very well for us to come up with the latest and greatest version of Thing2000 the world has ever seen, but I personally do not know a single developer who would be capable of selling Thing2000 on their own. Yes, we’ll all be highly indigent when Sales Bro “sells” Thing2000 Feature X before it exists, but at least someone out there actually knew Thing2000 existed and (hopefully) needs it to have Feature X (and is thus generating revenue for Accounting Drone to count and pay us with). I’m sure Sales Bro also has a blog where he complains about how he can never get his job done because developers are always so slow and uncooperative in delivering new things his customers need.

Let’s not forget Support Pleb, who has to suffer through customers rants and idiocy, while finding creative ways around developers shortcomings. I’m fairly confident a very large percentage of issues Support Pleb has to deal with day-to-day could be resolved by a bit of development time. This guy puts up with a huge amount of shit that would otherwise fall directly on developers, again, for crappy pay and non-existent job security. I’m pretty sure Support Pleb bitterly resents those developer slackers who always seem to be making more work for him, yet scatter like ants when problems are brought to their attention.

Finally we also have to have the Big Wig at the top somewhere. Those unreasonable people who are always hiring newbs and firing us, placing us in uncomfortable office environments, not shelling out for the tools we need, and dictating unreasonable deadlines. Again, unfortunately, I do not know a single developer capable of establishing and running a business the way Big Wig does. Big Wig possibly posts internet rants about how all he wants is for his Accounting Drones, Sales Bros, Support Plebs and developers to make more money for him - and doesn’t find that an unreasonable expectation, since he’s established and ensures the continuity the enterprise paying all their salaries.

So, are developer rants and whines about how bad they have it justified? For the most part, I’m going to have to say no. I’m also going to hazard that most of the problems developers face, in terms of unreasonable expectations, poor tools, lack of recognition, are not a result of simply being in software development as a profession, but are the result of poor management (at various levels of an organization), and are in fact faced by most other professions as well. We’re just more comfortable getting behind a keyboard and dumping our thoughts on the internet :).

TL;DR: As much as we like to think we might be, we’re not the be-all and end-all of our place of employment, and find a job at a company that makes you happy(-er).

Some thing I’ve been using for a while, and which recently became useful at work as well, is a simple HTTP service written in plain Java with existing JRE functionality, using an HttpServer.

Here’s a simple “main()” which sets up two basic “pages”, a root (/) and one which echoes your browser’s request headers (/headers/).

public class SimpleHTTPService {

    public static void main(String[] args) throws IOException {
        HttpServer server = HttpServerProvider.provider().createHttpServer(new InetSocketAddress(8080), 0);

        server.createContext("/", new HttpHandler() {
            @Override
            public void handle(HttpExchange he) throws IOException {
                byte[] output = "Hello world!".getBytes();
                he.sendResponseHeaders(200, output.length);
                he.getResponseBody().write(output);
            }
        });

        server.createContext("/headers", new HttpHandler() {
            @Override
            public void handle(HttpExchange he) throws IOException {
                StringBuilder result = new StringBuilder("Request Headers");
                for (Entry< String, List< String>> header : he.getRequestHeaders().entrySet()) {
                    result.append(String.format("%s", header.getKey()));
                    for (String val : header.getValue()) {
                        result.append(String.format("%s", val));
                    }
                    result.append("");
                }

                byte[] output = result.toString().getBytes();
                he.sendResponseHeaders(200, output.length);
                he.getResponseBody().write(output);
            }
        });

        server.setExecutor(Executors.newCachedThreadPool());

        server.start();

        System.out.println("HTTP Listening on port " + server.getAddress().getPort());
    }
}

Running this as-is will allow you to load up the URLs http://localhost:8080/ and http://localhost:8080/headers/ and see some output generated by the two registered contexts.

I’ve defined simple anonymous inner class contexts here, as it’s easy to play with, but obviously you can go wild and develop proper structures for these.

Combined with something like FreeMarker, and you’ve got a pretty neat way to deploy simple stand-alone HTTP applications written in Java with minimal external dependencies.

It’s also extremely useful for creating mock-ups services for use in unit tests for HTTP clients.

Here’s a small Java class I’ve been using in loads of applications and things for a few years (it’s evolved a little over the years).

It simply exposes a few very basic HTTP methods (for HEAD, GET and POST) which all just return strings containing the web server’s response. It’s seemed pretty useful and reliable in applications large and small, so maybe it’s of some use to someone else as well.

Download HttpUtils

Things have been very quiet on the code front lately, with bursts of activity here and there.

Primarily, I’ve been hacking on Out of Eve. The structure API classes and implementation of the API requests has been bugging me, especially when I’ve had to add a lot of stuff in recent versions, it just grows and grows and becomes unmanageable. Unfortunately fixing it has obviously resulted in breaking every single piece of the application, which I’ve been slowly refactoring and putting back together slightly more sensibly.

On the other hand it’s also a good opportunity to make allowance for the new customisable API key features CCP are implementing.

Besides that, my totally rad completely unique super secret Java-powered website project is stalled. Java’s turned out to be a bit of a pain in the backside. Sure all the background code is nice and stuff, but actually making the website portion of stuff quite sucks. I suspect this project will also need a near complete refactor at some point… Sigh. Coming soon in 2019!

Happy days. Now fully “Incursion” compatible.

Coming soon is updated API support (mail, taxes), and hopefully some more handy plugins :).

This is just a small update on the status of OOE. As the SVN server previously hosting the OOE source died, and many people have been looking for the source, a Google Code project has been started to facilitate future development.

The source is available as both a regular download and via Subversion.

It’s unlikely I will be doing any development on the project myself, however contributors are welcome.

Hah. Only 3 months late.

Out of Eve has been fully updated to Quantum Rise spec, the promised journal feeds, API key security, and a number of other tweaks. OutofEve.com has been updated to the latest available version, and the source is available for download.

Please leave any feedback in the comments of this post. I’ll set up a proper OOE page on this site at some point, with download links and more detailed information.

Well, shit happens, and unfortunately OOE 1.1 hasn’t as I’d planned. I HOPE to be able to have this going by next week….

I’ve wanted something to make browsing through largish JSON objects a bit easier for work for a long while now, and suddenly got the idea that 01:00 on Saturday morning would be a good time to create such an application.

The result is the rather simple but effective JSON Explorer.

As mentioned previously, I just wanted to outline a few plans for a new Out Of Eve version, mostly for my own reference, as I’m finding it much easier to work toward goals which are actually written down/typed up (lol?).

Obviously first order of business is Empyrean Age compatibility. A number of table and field names have changed and require some code updates. Lots of icons have been added and updated, so I would also like to make use of those. Unfortunately a number of images are actually missing in the EA icon dump (drones, rookie ships), so a simple drop-in replacement doesn’t works so well.

Another essential requirement, which should probably have been included in the original release, is encrypted API keys. My plan is to simply encrypt and decrypt these with a simple key file stored elsewhere in the filesystem - away from the usual configuration file, database and published www documents, so if any of that is compromised, without the key file, the API keys are useless to anyone snooping them. This also requires a method to automatically update existing unencrypted API keys.

Another handy feature would be the introduction of Atom feeds for market and journal transactions. My initial idea was an entry for each new transaction, however anyone doing a lot of trading would find their feed reader overloaded quite quickly. The obviously better solution is to just generate entries with all transactions since the last feed poll (taking into account API caching delays as well). I know I’d find this one particularly useful.

Actually that’s all :-). If all goes well, it should be releasable by the end of the weekend.