logo

ShrimpWorks

// why am I so n00b?

titleMinimum Effort Search

date 1 Jan 0001

Forward: This project came about due to a desire to make the Unreal Archive searchable, without sacrificing user privacy or its stand-alone nature by handing search off to Google or similar “site search” functionality regular search engines implement.

Additionally I wanted data to be searchable using custom criteria rather than hoping Google does the right thing or and trying to convince it to do the right thing based on HTML scraping.

Hosted on GitHub

A simple REST service exposing some RedisSearch functionality to support simple self-hosted site search functionality.

Requirements

Build

The project is built using Gradle:

$ ./gradlew execJar

This generates a fat/uber jar in the build/libs/ directory which may be used to run the service.

Configuration and Running

The service and index schema are configured using a simple YAML config file.

A sample file may be generated by simply running:

java -jar minimum-effort-search-exec.jar > config.yml

The above will write the file config.yml with some example parameters which you may customise.

Thereafter, run the service with the config file as the first parameter:

java -jar minimum-effort-search-exec.jar config.yml

The service will start up, listening on the port you specifier in the config file.

API

Add documents to the index

Add a single document:

POST /index/add

{
  "id": "1",
  "score": 1.0,
  "fields": {
    "title": "Blue T-Shirt",
    "body": "A very basic blue t-shirt you can wear",
    "price": 100,
    "tags": "shirt,blue,clothing"
  }
}

Add multiple documents:

POST /index/addBatch

[
  {
    "id": "2",
    "score": 1.0,
    "fields": {
      "title": "Jean Pant",
      ...
    }
  },
  {
    "id": "3",
    "fields": {
      "title": "Rooi Rokkie",
      ...
    }
  },
]

Search for documents in the index:

GET /search?q=shirt&limit=10&offset=0

Parameters:

  • q: Search query string
  • limit: Limit the result set to this number of documents
  • offset: Return documents starting at this offset. In combination with limit, allows for pagination through results.
{
  "docs": [
    {
      "id": "1",
      "payload": null,
      "score": 1.0,
      "fields": {
        "title": "Blue T-Shirt",
        "body": "A very basic blue t-shirt you can wear",
        "price": 100,
        "tags": "shirt,blue,clothing"
      }
    }
  ],
  "limit": 10,
  "offset": 0,
  "totalResults": 1
}

titleUnits API

date 1 Jan 0001

Hosted on GitHub

Units of measure conversion API, based on the work done by Units API for Drupal, though decoupled from Drupal and usable as a hosted service as well as a PHP library.

Usage as a Hosted Service:

Conversion Request:

GET http://host/convert.php?value=5&from=km&to=miles

Response:

{
  "from": {
    "value": "5",
    "unit": {
      "singular": "kilometer",
      "plural": "kilometers",
      "symbol": "km"
    }
  },
  "to": {
    "value": "3.107",
    "unit": {
      "singular": "mile",
      "plural": "miles",
      "symbol": "mi"
    }
  }
}

Usage as a PHP Library:

require_once('path/to/units-api/unitsapi.php');

$api = new UnitsAPI();

$result = $api->convert(5, 'kg', 'lb');

// $result is an associative array with the same structure as the HTTP JSON response

titleUnreal Archive

date 1 Jan 0001

Unreal Archive Post

Unreal Archive is an initiative to gather up, index, and catalogue as much Unreal, UT99 and UT2004 (and hopefully soon Unreal Torunament 3) content as possible. So far, we have maps, map packs, voices, skins, mutators, player models, as well as support for things such as patches, updates and drivers as well as a (currently very empty) section for written documents with the intent of providing guides, tutorials, manuals, and other related documented knowledge which also seems to get lost and forgotten.

The tech stack and some of the decisions involved may seem odd, but in keeping with the theme of longevity, preservation, and the general ease of losing things on the internet, these are some of my motivations:

  • statically generated content - the website is generated as a collection of plain HTML pages. This ensures no dependence on having to host a website with any dependency on any sort of back-end service beyond the simplest of HTTP servers. specific pains have been taken to ensure it works well with file:// local resources as well, so it doesn’t even need to be hosted!
  • written in Java - largely because I know it well enough to do this, but also because it’s not going anywhere soon, so the indexing and site generation capabilities will remain in action for a long time.
  • data stored as YAML files - a fairly simple format that’s also easily human- readable. in 30 years when all the YAML parsers have died, if someone looks at these files, they’ll be easy to write new parsers for, if that’s ever needed.
  • the “database” is Git - easy to distribute amongst many people, and since this is primarily an archive, the data does not change rapidly enough to require anything more real-time.
  • built-ino mirroring functionality to either download the entire file base, or to transfer it to any compatible S3 bucket store, and contribute mirror links back to the main Archive.
  • the entire project is “licensed” under UNLICENSE, with the intent of it being as absolutely open as possible, for as long as possible.

titleUnreal Package Lib

date 1 Jan 0001

Hosted on GitHub

A small Java library for efficiently reading Unreal Engine packages.

Unreal Packages, are used by Unreal Engine games for packaging content such as maps, textures, sounds, and the gameplay code itself.

Although the files all have different file extensions for organisation purposes only (for example, .unr or .ut2 for maps, .utx for textures, .u for code), they all have the same structure and are capable of holding the same content.

This implementation supports Unreal Engines 1 and 2, with support for Unreal Engine 3 in progress, and has been tested using content and assets from Unreal (1998), Unreal Tournament (1999), and Unreal Tournament 2003/4 (2004).

The implementation has focussed on supporting the above games for the purposes of making data available for the Unreal Archive, however other games using the Unreal Engine may Just Work, though some developers did introduce significant customisations to their engine versions, so your mileage may vary from game to game.

Also provided via the Umod class is the ability to read and extract the contents of .umod installers, commonly used to distribute larger Unreal and Unreal Tournament modifications and total conversions.

These can be combined with a PackageReader to support reading package contents directly from UMOD files without needing to unpack the individual files first.

Finally, reading of Unreal Engine’s .int and .ucl files is provided via the IntFile class, which simplifies processing some of the non-INI file like properties contained within these files.

//