My dconf2014 talk on gameservers in D

August 6th, 2014

dconf2014 is over for quiet some time now, the videos of the talks are online now and I finally find the time to write about it.
The conference was a blast! I am sincerely grateful for being there. We had a lot of awesome talks by some of the brightest guys I have ever met. It was an honor to be among the speaksers and to finally meet the faces behind those internet nicknames that grew familiar over the years!

Without much further ado, here is the video (also available on the internet archive1):

You can find the corresponding discussion on reddit2 and all the other great talks of dconf2014 also on the internet archive3.

I uploaded the slides on slideshare:

  1. https://archive.org/details/dconf2014-day03-talk04 []
  2. http://www.reddit.com/r/programming/comments/2b1uat/dconf_2014_mobile_gameserver_backend_in_d_by/ []
  3. https://archive.org/search.php?query=DConf%202014%20AND%20collection%3Aopensource_movies []

STACK4 and the D Programming Language (Part3)

January 14th, 2014

I finally submitted this topic as a proposal for a talk at dconf20141 and found the time to continue this series.

This is Part3 in a series of articles.
Part1 roughly elaborated the different options of languages+frameworks that were evaluated before deciding to use D.
Part2 gave an overview of the libraries that I developed and open sourced that the server uses and scratched the surface of why I picked D over Node.

The basis for the STACK4 multiplayer feature is the Sockjs long polling connection model described in the previous article.
In the sockjs-unity3d-xhr-example2 github project I implemented a simple chat in a unity3d project using 3 different server implementations to be able to compare them. There is a Java, a Node and a D implementation. The Node and D implementation are pretty much the same in length of code, but the Java version is 50% longer and much more verbose. Other than that, my Java version also performed worse than the Node and D version. But that may be due to me being rusty on the Java front.

The performance difference became obvious when running the server on my resource -constrained consumer server that also handles mail, website and more. But being able to start running on such a machine was exactly the intention.

Node vs. vibe.d

I love Node. It introduced me to a whole new mindset and enabled me to prototype a lot of stuff amazingly fast, but then I learned about Vibe.d3. For me it is just the next evolutionary step, combing all that is great about Node with an even better programming language and paradigm.

I coded some reasonable big server projects with Node already to know that JavaScript was too much of a deal breaker for me to use it for STACK4 again. In this installment of the series I am going to show some code that makes clear why D (Vibe.d) won over JavaScript (Node). There are three main problems for me concerning Node and JavaScript and that is the JavaScript programming language, readability of code and debugging. The problem with JavaScript maybe just personal taste, but I am used to write statically typed programs that pretty much work after I have them compiling. In JavaScript however after writing code the challenge begins, because a lot of coding errors are first popping up once a certain part of the code is being executed. The other two problems are discussed below.

Readable Code

What makes Node code so hard to read on my eyes is the fact, that it is often a hell of a lot nested callbacks.
For the purpose of this article I present a short example to demonstrate this. Imagine a simple http server that should respond after a second of delay with a “hello world” and also logs the request to a file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
var http = require("http");
var fs = require("fs");
 
http.createServer(function (req, res){
	res.writeHead(200, {'content-type': 'text/plain'});
 
	setTimeout(function(){
		fs.appendFile('log.txt', 'data to append', function (err) {
			if (err) throw err;
 
			res.end("World!");
		});
	}, 1000);
}).listen(80);

Now here is the D version using Vibe.d:

1
2
3
4
5
6
7
8
9
10
11
12
13
import vibe.d;
 
void handleRequest(HTTPServerRequest req, HTTPServerResponse res) {
	sleep(1.seconds);
 
	appendToFile("log.txt", "data to append");
 
	res.writeBody("Hello World!");
}
 
shared static this() {
	listenHTTP(new HTTPServerSettings, &handleRequest);
}

Aside the fact that the D code is using just one indentation level it has a couple of other advantages over the Node version. Thanks to a D feature called Unified Function Call Syntax (UFCS)4 line #4 is possible and makes it both easier to read and less error prone. It clearly states our intention to wait for a second before continuing the execution. What UFCS actually brings to the table is that you can rewrite every function foo taking T as a first argument as T.foo(). This way you can extend every type even those that you have no access to with standalone methods that act as if they are members of it. The seconds method is typesafe and is more expressive than a 1000 ms integer magic number.

Note, that the whole server process in not blocking. However the fiber5 , that handles a single http request will be blocked eventually. This is the major difference of Vibe.d and Node. This way the intention of the code is instantly clear and behaves exactly as serial as it is written, even though the whole system is still highly asynchronous and event driven. Two parallel requests get handled in two different fibers and one can already be responding to the client while the other is still waiting in the sleep without any kind of multithreading risks. D calls them fibers, but they are also known as co-routines6.

Note that there is the npm package co7 to get co-routines in Node, in fact there is a couple of libraries to avoid the callback hell. The fact that there are so many libs to mitigate this issue shows that this is a problem that Node users crave to have solved. I also have to mention that I took this sample from a popular Node textbook, so it is safe to say that this callback nesting is common practice.

Debugging

There are two things that bothered me while writing this simple Node showcase:

  1. Not a single typo got caught before actually running the related line of code.
  2. Try reading the callstack when an exception in the appendFile occurs.

While 1. is simply due to the nature of JavaScript being interpreted just in time and maybe subject to taste, 2. is plain annoying when trying to debug even the simplest Node programs. And it does not stop there, since the callstack is totally messed up all error handling in the code based on exceptions is making knots in my head (just try to put try/catch blocks around some of this nested goodness and see…).

Let’s imagine a simple throw “poo” exception in the above code inside the appendFile callback and have a look at the callstack printed:

e:\_docs\_bitbucket\test\js\main.js:10
throw "poo";
^

Following is the D version of this. It is pretty short too, but since the handleRequest runs in ordinary serial order the callstack always is correct and all try/catch and powerful scope guards8 still work:

object.Exception@source\app.d(8): poo
----------------
0x00403972 in void app.handleRequest(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse) at E:\_docs\_bitbucket\test\d\source\app.d(11)
[...vibe.d internal callstack snipped...]

So in Node there is no usable stacktrace. When you mix and reuse callbacks a lot you have no idea from where a certain exception got thrown. This also makes it hard to use try/catch correctly. In D using Vibe.d this just works safe and sound!

Huge systems have been successfully written using Node but it introduces a new execution and error-handling dimension that must be handled manually. With my C mind if just feels more natural to use the Vibe.d approach with it’s own drawbacks.

 

Next time I will go into the design of the STACK4 server architecture.

 

Special thanks to Mason Browne (@neworb) for his constructive feedback on the first version of this post!

  1. http://dconf.org/2014/index.html []
  2. https://github.com/Extrawurst/sockjs-unity3d-xhr-example/tree/master/server []
  3. http://vibed.org/ []
  4. http://dlang.org/function.html#interpretation []
  5. http://vibed.org/features#fibers []
  6. https://en.wikipedia.org/wiki/Coroutine []
  7. https://github.com/visionmedia/co []
  8. http://dlang.org/statement.html#ScopeGuardStatement []

STACK4 and the D Programming Language (Part2)

January 9th, 2014

This is the second installment in a series of articles. This series describes the server backend that I wrote for the mobile game app STACK41.

In the first post I roughly elaborated the different options of languages+frameworks that were evaluated before deciding to use D.

This post describes in more detail what kind of requirements the server infrastructure imposed and what kind of auxiliary libraries were open sourced in the process of developing the STACK4 server backend.

The STACK4 Client uses Unity3D2 as a cross-platform game engine. It makes it easy to develop for all the major mobile systems like Android, iOS and Windows-Phone with a (almost) unified code base.

Long Polling

Unity Basic (Free license) restricts you from using raw sockets. So you can choose to either implement your own native extension for all platforms or to resort to using the web requests that they allow. I chose to use this approach since it is the most proxy/firewall friendly approach and since I do not depend on low latency it is sufficient for me. I knew libraries like socket.io and sockjs from my javascript/typescript adventures so I decided to implement the sockjs protocol on top of the web requests function (WWW class3) in Unity3D. These libraries give me the advantage of using an interface that acts like regular persistent connection socket but can fall back to a long polling method if needed. In my case only this method is needed. This work is available on github: https://github.com/Extrawurst/sockjs-unity3d-xhr

Asynchronous I/O, Fibers and SockJs

I used nodejs a lot the last 2 years, even at work. Using Socket.IO I implemented a realtime multiplayer game in a toy project called beachwars-online (hiding the whole browser inconsistency hell from me). Socket.IO supports long polling as a fallback too, but unfortunately the Socket.IO implementation for D4 did not support that fallback to long polling yet. Since I already decided to use a similar but less feature rich lib called SockJs for the client side I implemented SockJs for the server-side in D too: https://github.com/Extrawurst/sockjs-d.

Through my work with nodejs I was already sold to the concept of asynchronous I/O and the performance of it. As much as I hated coding in JS all the servers I wrote in node where exceptionally fast. Good thing Sönke Ludwig wrote vibe.d5 which offers exactly that for D. It is based on libevent and offers a lot of functionality similar to nodejs making web development and network programming in general very convenient. That is why the sockjs server lib is based on vibe.d.

The major difference in using vibe.d compared to nodejs (other than the fact that it is D code) is the fact that vibe.d chose to use an approach different to cascading callbacks inside of callbacks to implement async events. vibe.d uses a co-routine-like approach (in d-land called Fibers6) which basically leaves the user code synchronous and linearly. As a bonus or more a natural consequence of that is that exceptions and debugging in general still have intact callstacks (sounds small but believe me debugging nodejs in comparison sucks big time).

Dependency Libraries

Here is a simple list of dub7 libs that I used in the project. They all integrate nicely and use the asynchronous approach if needed. The ones that I developed during this project are all published on github8 and are going to be subject of the following posts:

  • sockjs-d
    SockJs server implementation based on vibe.d (see above)
  • sockjs-unity3d-xhr
    The unity3d compatible implementation of the SockJs Protocol (see above)
  • mysql-native
    The mysql connector used (vibe.d/async compatible)
  • gcm-d
    Google Cloud Messaging connector (async)
  • apn-d
    Apple Push Notification Service connector (work in progress)
  • xtea-d
    eXtended Tiny Encryption Algorithm implemented in native D
  • elo-rating-d
    Simple lib implementing the elo rating system in native D
  • forever-d
    Tool based on node’s forever to keep a process running continuously

Since I was asked to show some actual code, I now did in Part 3…

 

  1. https://play.google.com/store/apps/details?id=com.Extrawurst.FIR []
  2. http://unity3d.com/ []
  3. http://docs.unity3d.com/Documentation/ScriptReference/WWW.html []
  4. http://code.dlang.org/packages/socket.io []
  5. http://vibed.org/features#performance []
  6. http://vibed.org/features#fibers []
  7. http://code.dlang.org/ []
  8. https://github.com/Extrawurst []

STACK4 and the D Programming Language (Part1)

January 6th, 2014

As of last year I am the proud developer of a mobile game app for android (and blackberry as of now). STACK41 is a rather simple little game based around the concept of connect-4 but adds 3D and consumable extras to the mix.

This post is about the biggest addition to the game since its launch last summer. In December 2013 I added a realtime multiplayer mode to the game (version 0.3.0).

The server backend for this endeavor is written in the D Programming Language2. You may ask why this is… Well this was kind of a journey! This is gonna be a series of articles about how I developed that server and what kind of challenges occurred on the way.

Part1 elaborates roughly what other possibilities were evaluated and a brief not-so-technical overview about the development in D-land over the last year.

Typescript / Nodejs

I used nodejs a lot over the last 2 years and I really like it… performance-wise. What totally annoys me is Javascript. So naturally the first straw that I grabbed was Typescript which is basically JS with types that compiles down to bare JS. Using this, I implemented a first small browser game with interactive multiplayer gameplay called Beachwars-Online. Unfortunately TS is still JS and all the libraries out there are for JS and writing bindings annoyed the hell out of me. So I dreamt about having a framework as rich as node (+all the npm packages) but with a statically typed language.

Java / Vert.x

Next thing I evaluated was Java! Typesafe, strict OOP and an insane huge framework for all the needs, why not? Vert.x promised the event-driven performance I was used to from Node. In the end the VM and/or Vert.x were too ressource hungry for my commodity hardware server. And by the time I was evaluating Java I got help by a friend who was no java coder nor was he willing to become one. So I thought to look for yet another solution.

PHP

My friend is very proficient in PHP and put that on the table too. But since I wanted to do realtime server connections and wanted to leave scripting languages behind PHP got sorted out pretty quickly too.

C/C++

I am a C++ coder at day and my friend also learned C in the days. But trying to find object-oriented wrappers for stuff like libuv was not as easy as I expected. Writing something ourself seemed unreasonable for the task and not being able to use C++11 for it (remember the commodity = dated hardware) made me expect the worst. That is when I remembered something that cannot deny its C++ roots: D.

D

2013 was also a good year for D. The first real dconf was held in San Fransisco3 with a load of interesting talks and people! I recommend everyone to watch the video recordings!

The D reference compiler “dmd” matured a lot, stability-wise and by finally supporting x64 and shared libraries. The toolchains got orders of magnitudes better through the dub package manager4 that lent a lot of ideas from npm for nodejs (which is a good thing, since this works extremely well). Also in IDE land mono-d5 made my work with D a lot easier.  It got very stable by now and the developer is insanely responsive (lets hope he never finishes university).

In october Andrei Alexandrescu announced the first commit of D code to the Facebook source repository6. In November, they even started to support D financially by putting bounties on compiler bugs7. Others followed.

Facebook is also the sponsor of the location for the D conference in San Fransisco. And I think it was the dconf talk of Vladimir Panteleev8 that first introduced me to vibe.d. The Vibe.d framework offers asynchronous I/O and all you need to write a webserver in D.

 

TL;DR

I know D for a long time, have used it a lot years ago, wrote my B.SC Thesis about it. Vibe.d9 offers me what I like in node.js albeit being written in a usable language for serious sized projects. Despite being asynchronous like node it also looses the callback hell by using a co-routine like design. It is more memory efficient than Java/Vert.x, PHP was no option, bare C not even close and D is the better language compared to the competitors anyway.

 

Part2 is going to describe a little more in detail  what the STACK4 server backend looks like and how I implemented it in D… stay tuned

  1. https://play.google.com/store/apps/details?id=com.Extrawurst.FIR []
  2. http://dlang.org/ []
  3. http://dconf.org/2013/ []
  4. http://code.dlang.org/ []
  5. http://mono-d.alexanderbothe.com/ []
  6. http://forum.dlang.org/thread/l37h5s$2gd8$1@digitalmars.com#post-l37h5s:242gd8:241:40digitalmars.com []
  7. http://forum.dlang.org/thread/l65mvq$du0$1@digitalmars.com#post-l65mvq:24du0:241:40digitalmars.com []
  8. http://dconf.org/2013/talks/panteleev.html []
  9. http://vibed.org/ []

Android Immersive Mode and Unity3D

December 8th, 2013

Today I got an email by one of the users of my mobile game STACK4 asking me to support the new “Immersive Fullscreen Mode”1 of the brand new Android 4.4 (KITKAT). To be honest I didn’t even know this feature was introduced. But it is actually pretty cool for mobile games, since it allows your app to use the fullscreen and to hide those softkeys that are shown in absence of hardware buttons. See the status bar and the softkeys on the photo, they are just moving in as an overlay when you swipe in on the top or bottom of the screen (which is a method first introduced on blackberry 10 devices)

Anyway it is not really difficult to implement when you already have an Android Plugin in Unity. So without much further ado, here is the Java code that you have to include into your UnityActivity in the onResume method:

@Override
protected void onResume() {
	super.onResume();
 
	if (Build.VERSION.SDK_INT >= 19) { //KITKAT
 
		int flagImmersiveSticky = 4096; //View.SYSTEM_UI_FLAG_IMMERSIVE_STICKY
 
		int flags = flagImmersiveSticky
			| View.SYSTEM_UI_FLAG_FULLSCREEN
			| View.SYSTEM_UI_FLAG_HIDE_NAVIGATION;
 
		this.findViewById(android.R.id.content).setSystemUiVisibility(flags);
	}
}

There are a few things to mention about this:

  1. Why don’t I use the sdk identifiers2 for those new values ? Because Unity seems to refuse to build in my java plugin when it is not build against 4.2.2. Any ideas are welcome!
  2. Why in the onResume ? In onCreate it will work too, but whenever the app goes to the background and back to the front those settings seem to be reset either by android or unity, who knows.
  3. Why is there a “>=” in the if clause ? Because my code highlighting plugin in wordpress sucks :D Of course it is greater-or-equal.
  1. https://developer.android.com/training/system-ui/immersive.html []
  2. new flags for immersive fullscreen in android sdk []

Hack to bypass actionscript vtable

May 1st, 2013

Imagine a simple class hirarchy of A extended by B. A has a method foo and B overrides this method.


Now imagine both A and B to be classes of a library. Now you want to implemented C extending B. See the UML at the side for illustration.

Further you want to implement your own foo now and override B’s foo. But now you want to call A’s foo to have the base classes behaviour too.
In many object oriented programming languages like C++,C# and D you can explicitly do that.
In C++ for example you can simply do the following

1
2
3
4
virtual void foo()
{
	A::foo();
}

Now in Actionscript on the otherhand, even though it calls itself object oriented you simply cannot do that.
AS3 has super for calling the baseclass method but that is B’s foo in this case. So that is not intended. super.super.foo(); or A.foo(); does not work either ;)

So I was pretty lost yesterday when presented with that problem. But as always for those dirty edge case problems there is an even more dirty hack to circumvent it.

Since I know that A has this method foo implementation that I need I just have to find a way to explicitly call it. Like stated in the Docs1 a function instance.methodname() can also be called like instance["methodname"].call(this_pointer);. Now the to prevent the vtable lookup from happening we have to create a dummy instance of class A to call the mothod from and pass the this pointer of C upcasted to a pointer of type A.

1
2
3
4
5
override function foo():void
{
	var dummy:A = new A();
	dummy["foo"].call(this as A);
}

And voilà, it works. As you can see we are exploiting actionscripts ECMAscript2 roots a little bit here ;)

  1. Flash documentation of Function object []
  2. ECMAscript on wikipedia []

Marbelous! now in the Appstore

February 28th, 2013

I has been nearly a month now, that the first game I helped creating made it to the Google App Store1.

It is called “Marbelous!”2 and is a really nice 3D sidescroll puzzler. You control the character which is a marble with your thumb and interact with all kinds of triggers, doors and catapults. The leveldesign is full of memories from videogame past and the sound and music makes you feel right at home.

So be sure to grab it now and try the free version in the appstore and watch the trailer3.

Watch the German Let’s play on Youtube:

  1. Marbelous! in the AppStore []
  2. Marbelous Website []
  3. Marbelous! official trailer []

Twitter tool “followsme” to track unfollows

February 22nd, 2013

A little while ago I started a little tool for my own convenience called “followsme”. It is written in Typescript1 and runs under nodejs2. You can use it to watch a twitter account for new followers and ones who stopped following it. The reason I wrote it was the fact that tools providing unfollow tracking tend to be expensive. Ok and for the fun of it ;)

The tool polls the list of followers of the given account on a regular basis and compares to previous lists to find out what changed.

So here is the code for every one to use: https://bitbucket.org/Extrawurst/followsme

Some drawbacks:

  • You will need an Twitter App API Account to use the code. You have to fill in your credentials in the account.js file.
  • Don’t use it on accounts with more than a couple of thousand followers. The Twitter API is too limited (see rate-limit3 in the docs) to process this quick enough and I was too lazy to implement fancy chunked processing (I tried Obamas Twitter Account.. don’t do that^^). 

This is the how a followsme report looks like:

Followsme report

  1. Typescript []
  2. NodeJs []
  3. Twitter API rate-limits docs []

Cookies wtf!!

November 16th, 2012

Just wanna bark that out: Who the hell did the design for document.cookie ?

I was a naive guy coming from my known little world of C++ server backend programming to this godless land of web development and my head blew off :D

Thats how I felt a lot of times making my steps in this unknown and hostile environment. But I never felt as disenchanted as I felt when I start to use cookies in my js code. All the rest was more or less ‘consistent’ in its twisted kind of way and I thought I was starting to get my head around things. But man was I confused when I started to wrap my cookie code together “just to remember the login”. “It can’t be so hard” I thought – I was wrong!

  1. Why on earth is document.cookie like a property where nothing else works like this?
  2. You can get all cookies in one gigantic key=value separated string … Seriously ? Ok I get split that myself…
  3. document.cookie = “” and document.cookie = null does nothing that you would expect (It actually does nothing at all)
  4. document.cookie = “foo=bar” ADDs this single cookie (Like it magically became a function call…)
  5. And finally my favorite: To loose a cookie entry you actually have to update it with a expiration date that is in the past !!!! Seriously ? WTF !!!

That’s when I started looking for a lib to keep my sanity!
Believe me you don’t want to keep up with this crap, go grab jquery-cookie: https://github.com/carhartl/jquery-cookie

Typescript to the rescue!

October 3rd, 2012

Ok all the programmer sphere is making a lot of fuzz about this right now, so I guess I have to take a stand too.

Just a few days ago Microsoft announced a new programming language called Typescript1. According to their homepage “Typescript is a language for application-scale JavaScript” and after a couple of hours playing around with it I get the feeling they are right.

Since I am programming nodejs2 for the server backend of a web based multiplayer game right now I am struggling with javascript for months now. And all I ever whished for was just a tiny bit of compile time type safety that I am used to from my day job as a C++ coder (I know – but I guess I really like C++ after all).

They released the 0.8 version of Typescript to the public in a rather mature state. Considering that: I could already use all the syntactic sugar and completion of it in a Visual Studio 2012 plugin that they released simultaneously3.

Anders Hejlsberg presented Typescript in a channel 9 video4. He is famous for C#, .NET and the original Turbo Pascal IDE and I can really just suggest to watch this video!

Considering that Typescript promises exactly what I was hoping for in JS: strict type safety,classes,inheritance I gave it a try and started to convert some code in my nodejs codebase. For me the most important factor to do that was to be able to slowly migrate code to it. So you copy paste one module to a Typescript file and without changing anything the compiler spits out an rather unchanged JS file. So you can just migrate whatever you like or use your old code and start using the real features of TS for what you are programming from that point on.

All in all I am really impressed so far and constantly convert stuff over to it :)
And guess what, Microsoft open sourced it, too!

I keep you posted…

  1. http://typescript.codeplex.com/ []
  2. http://nodejs.org/ []
  3. http://www.microsoft.com/en-us/download/details.aspx?id=34790 []
  4. https://channel9.msdn.com/posts/Anders-Hejlsberg-Introducing-TypeScript []