Return Zero

Draft Idea: NodeJS Async with Preemption

Mr Silva

The main idea of Node.js is to use non-blocking, event-driven I/O to remain lightweight and efficient in the face of data-intensive real-time applications that run across distributed devices.

How it works under-the-hood is pretty interesting. Compared to traditional web-serving techniques where each connection (request) spawns a new thread, taking up system RAM and eventually maxing-out at the amount of RAM available, Node.js operates on a single-thread, using non-blocking I/O calls, allowing it to support tens of thousands of concurrent connections held in the event loop.

Well it does sound really good on paper, so I tried some load testing against a default HelloWorld node process and I did get a very high number, let us call it N, and then I tried a simple web server in Erlang and in every run, I got a number that was 4 times N, It took me a while to figure this out but the number 4 was basically the number of cores on my machine.

So what does Erlang do that is different from Node.JS, Erlang by default uses pre-emptive scheduling, what this means is that the scheduler has more control over tasks. Aside of assigning a task to a worker, the scheduler also assigns a time slice for the task.

A task gets removed from the worker when it yields (i.e. done or blocking for I/O) or when it has used up its time slice.

When the time is up, the scheduler interrupts (preempts) the task and let another task run in its place, and the original task waits for its turn again.

While in a purely co-operative scheduler (Async I/O being one of the implementation) any operation that blocks has a potential to slow down the process, this includes any kind of tight loops, image manipulation or encryption routines. So one has to be very careful not to block the thread in any way.

You might ask, does my HelloWorld Node program block anywhere?
The answer is yes, even though for only a fraction of a second, the response.write() call which writes back to the socket is a blocking operation.

Now, can we improve NodeJS in anyway?
Not NodeJS, but we could definitely have a different implementation which utilizes pre-emption. There is another implementation of Node called Deno from Ryan Dahl, the creator of NodeJS. There are lesser known ones like JerryScript and Duktape.

Let’s talk about the initial steps that we need to make NodeJS code pre-emptable, The first things you would see a change in is, the use of synchronous code instead of asynchronous code.

so the code below

const fs = require('fs').promises;
async function loadMonoCounter() {
    const data = await fs.readFile("file.txt", "binary");
    return new Buffer(data);
}

will look like the one below,

const fs = require('fs'); 
const data = fs.readFileSync('./file.txt', {options});

Now, How do we make this run at par with Asynchronous code?
The magic here would be to run make it run under a custom GraalVM based runtime, there is a reference implementation available here: https://github.com/oracle/graaljs. If you haven’t heard about the GraalVM, it is essentially trying to be a universal VM, included within it is an SDK that makes it very easy to write new languages or port existing ones that target Graal.

HTTP would need a custom layer that would essentially create new VirtualThreads for each request which would gives us the property of pre-emption and we let Graal’s JIT do the optimization for us and since the newer JVMs network and I/O stacks use co-operative scheduling, we get the best of both worlds. This is the fundamental idea or at least the theory.

What would the performance look like?
unfortunately, There is no such implementation yet, I do have a little experience trying to port parts of PHP C based implementation to GraalVM, so I might try doing a very rough implementation if I ever get a weekend or so for myself. I am not expecting a four fold increase in performance but any improvement is a good indicator that the theory works and my idea then would be to push the idea to the community and hopefully have it as part of NodeJS or Deno.

Note: Most of the post is just rough ideas and me thinking out loud. so if you are reading this, I am sorry, you might find parts which may sound incomplete or something that doesn’t make sense.

Leave a Reply

Back to top