libsass.js - An Emscripten Experiment
Sebastian Golasch and I set out to run the Sass compiler in the browser. Why, you ask? because we can! Or, well, we couldn't - and that needed changing. We'd heard about Emscripten, but never really used it.
TL;DR: See the Sass.js demo and the libsass.js repository for trying to compile it yourself.
It all started with Lea's tweet.
Sebastian (@asciidisco), living in the hotel room next to mine at the time, thought it would be a good idea to have another look at Emscripten. Emscripten is a compiler that can turn C/C++ into JavaScript. Sebastian decided to go with sassc, a CLI (Command Line Interface) wrapper for libsass. It took him a few hours but he got sassc running in the browser.
You might wonder why we were even considering this hassle instead of just implementing Sass in JavaScript. The original Sass library was written in Ruby. Others have tried (e.g. scss-js) and failed at keeping it up to date. We didn't want to add to that growing list of dead-ware™.
Sebastian decided to use sassc so he wouldn't have to build an interface for the library. He's seen this approach work for pngcrush.js and gave it a try. While Sebastian was trying to figure out how to get the compiled program to run more than once, I started looking into emscripting libsass itself.
What follows is a write up of my experiences dabbling with Emscripten. If you are not interested in that but merely care about Sass in the browser stop reading here.
Incident Report
Let me, very quickly, explain who I am so you have a better picture of how things went down. I've been doing web things since 1998. I learned programming in PHP. I thought I knew C++ because I wrote a terminal based game Battleship at the University in 2006. The only exposure I had to C/C++ since, was reading through some of PHP's internals. It is 2014 and it's safe to say: I know jack shit about C/C++.
The steps necessary to compile libsass to JavaScript are detailed in the Readme. I won't repeat that here. Instead I'll explain why this endeavor took us 3 days instead of 3 hours.
I didn't know anything about the C/C++ build environment. I've been using MacPorts and Homebrew for years, but only very occasionally compiled something myself. And by "compiling" I mean typing make install or something I copied off some readme somewhere.
After installing Emscripten following their instructions, I ran emmake make and my terminal responded with env: python2: No such file or directory. It was 01:00 (1AM) and I didn't feel like I wanted to play with this anymore. I had given up before I got started. The next morning Sebastian, who had already fixed the issue on his machine, helped me get my installation up and running. If it wasn't for Sebastian pushing me, I'd not have continued pursuing libsass.js.
Once python2 was available, and modifying the Makefile we got Emscripten to compile libsass. The problem was, the resulting file only contained the Emscripten environment. Reading Calling Compiled Functions From Normal JavaScript helped. We got a JavaScript file exposing sass_compile(). The trouble was, we couldn't call it.
The C++ function sass_compile() expects a struct (if you're a JS person, think of a typed object) but the wiki didn't say how to handle that. Googling around we found out that working with structs is not recommended, as you'd have to do "all kinds of weird pointery things". At this point I decided to wrap sass_compile() in another function that would construct the struct from primitive parameters. This ended up becoming emscripten_wrapper.cpp.
The road to that little function was long and bumpy. It involved consulting Christian Kruse for the most basic things. Questions like "how do I copy a string" seem strange to JavaScript developer, because in JS it's a simple assign. In C++, when working with C strings, you need to #include <cstring> and execute strdup(). To someone actually knowing C/C++ this must seem ridiculous. At the time I felt a bit more like this.
Anyways, after making sure the Makefile used emcc and emar (instead of g++ and ar), I was able to compile valid SCSS to CSS. YAY! For invalid SCSS I got the following JavaScript error thrown:
"5323272 - Exception catching is disabled, this exception cannot be caught. Compile with -s DISABLE_EXCEPTION_CATCHING=0 or DISABLE_EXCEPTION_CATCHING=2 to catch."
I recompiled libsass.js with the proper flag and nothing happened. I got no error message and no exception. I figured the mentioned "exception catching" referred to something within Emscripten and had to do with passing an exception from C++ to JavaScript. This later turned out to be wrong. A day later. I wasted a day because I didn't understand that I needed to tell Emscripten that C++ code must be able to handle exceptions internally. Only after Andres Freund pointed this out, did I get my error message.
The error message was supposed to be returned via parameter (think "call by reference"). As Emscripten Pointers And Pointers shows, it's not as straight forward as I hoped it to be. Considering I'd already wasted a day of work, I did not pursue the idea of reading data from a pointer further, but returned the error message the same way the compiled CSS came back to me. By simply returning the string. This is about as ugly as it can get. It did the job though. I finally had a version that either returned compiled CSS or told me what was wrong. I thought my job was done.
Christian disagreed and figured out how to get all that pointery stuff working. Seeing the working example, I can even make sense of it. Maybe Alon can add that to the docs? Christian also managed to modify the Makefile in a way that would allow us to send the changes back to the original libsass project.
Emscripten API
I'm told people want to see code and stuff. Showing you some of the compiled libsass wouldn't satisfy that need, though. I mean, just look at this childsplay™:
var f=0,g=0,h=0,j=0,k=0;f=i;i=i+24|0;g=f|0;qJ(g,c[d+4>>2]|0);z=0; …);
Let's have a look at how you can access the generated libsass.js instead. After loading the file, Emscpripten will litter itself all over the global scope. A quick comparison using Object.getOwnPropertyNames(window).length shows Emscripten declares ~400 (four hundred) global variables. Ouch! But then, that ~670KB light environment needs to do something, doesn't it? Short story is: you don't want this loaded in your document, but rather have it contained in a web worker. Seriously.
So let's have a look at how you interface with a C++ function from JavaScript:
// it gives you access to the environment
var result = Module.ccall(
// the name of the function to invoke
'the_function_name',
// the data type the function will return
'string',
// the data type the function expects as parameters
['string', 'number'],
// values to pass as arguments
['hello world', 123]
);
That looks pretty straight forward. That was about as far as I got from looking at various examples. It required Christian wasting another night to "reverse engineer" how to pass in a pointered pointer which you can read a result from:
var ptr_to_ptr = Module.allocate([0], 'i8', ALLOC_STACK);
// execute the C/C++ function
var result = Module.ccall(
'the_function_name',
'string',
['string', 'number', 'i8'],
['hello world', 123, ptr_to_ptr]
);
// pull pointer from pointer
var err_str = Module.getValue(ptr_to_ptr, '*');
if (err_str) {
// pull string from pointer
err_str = Module.Pointer_stringify(err_str);
}
Besides the obvious – C/C++ developers not liking CamelCase – what is happening here is quite simple, if you understand pointers. A pointer is an address to some space in memory. A pointered pointer (or pointer to a pointer) is an address to some space in memory, where an address to some space in memory is stored. A construct like a pointer to a string pointer is somewhat common in C, you allocate memory on the heap and let the parameter pointer point to this address. You would use the & (address of) operator to give the called function the address of the variable you want to have the string stored in.
Since JavaScript doesn't know pointers in this form, Emscripten has to emulate the memory management of C to be able to run such code. And I mean this literally: Emscripten takes the byte code generated by the compiler, turns it into JavaScript and emulates the necessary API to let that run. A pointer in JavaScript/Emscripten is basically just an offset for an array (think memory[pointer]). Memory like you have in C is emulated in JavaScript by "abusing" an array. The array is the memory, the pointer ("address to space") is an index of an element in that array. And in such a "space" we can either save data, or an address to another space. In C the stack is generated by the compiler, and the emulated memory array is generated by Emscripten.
After the C++ function was executed, we retrieve the first pointers value (another pointer) from memory. If it is 0, (the numeric presentation of NULL), there is no address. Otherwise we just got another pointer that we can resolve to string (think var address = memory[pointer]; var value = memory[address];) And that's it, we passed a string back to the caller via function parameter.
That sounded like Klingon to you? Maybe this snippet can help explain the idea:
// We can't do any fancy pointer arithmetic, but we can pass
// an object into a function and have the function mutate its
// members
var pointer = {};
var nonPointer = 'mars';
function addProperty(reference, value) {
reference.pointer = 'world';
value = 'jupiter';
}
modifyObject(pointer);
console.log(pointer.pointer, value);
// "world", "mars"
Of course this is not even remotely the same thing. But I think it illustrates the mechanics – or rather basic idea – of pointered pointers quite nicely.
JavaScript API sass.js
We realized that working with the Emscripten API was going to turn people down. Hell, it turned me down. If anyone was actually going to use this thing, it needed a nice-enough API, so we created Sass.js. And there you have it, Sass in the browser:
var css = Sass.compile(scss);
console.log(css);
// .some-selector{
// width: 123px; }
Using the Filesystem
Sass has a feature to include other files using @import "path/to/file";. In our projects we use this quite frequently, because it allows us to split a larger style code base into small modules. I wanted this feature to work in the browser as well. Luckily Emscripten comes with it's own filesystem. This is not the File System API. The Emscripten environment offers low-level filesystem access, for example:
var result = FS.readFile('/test', {encoding: 'utf8'});
console.log(result);
// "Hello World"
If you were to writeFile() to a path that doesn't exist, the function throws an error. You can create directories using FS.mkdir('/some-path');. In this instance low-level means that mkdir doesn't support recursively creating parent directories, so FS.mkdir('/some/path/hello/world') will throw an Error. While these functions allow you to implement pretty much everything you want to, they're not particularly convenient.
By the way, finding out which functions FS provided and what they did was figured out by using Chrome's DevTools. Executing FS.readFile in the console is the same as running console.log(String(FS.readfile)) – it dumps the function's source code to the console. Because the DevTools approach worked so well, I can't even tell you if Emscripten's FS is documented anywhere. But we didn't want developers to have to figure out things on their own. Therefore we added file-support to Sass.js, hiding the ugly parts:
'/unknown/path/test.scss',
'.selector { content: "hello world"; }'
);
With that we can make @import directives work in the browser:
Sass.options({ comments: true });
// add some files
Sass.writeFile('one.scss', '.one { width: 123px; }');
Sass.writeFile('some-dir/two.scss', '.two { width: 123px; }');
// compile the imports
var css = Sass.compile('@import "one"; @import "some-dir/two";');
console.log(css);
resulting in the following output:
.one {
width: 123px; }
/* line 1, /sass/some-dir/two */
.two {
width: 123px; }
There's even a demo trying out loading individual SCSS files from the server and composing things in the browser. We're not quite there yet, but we actually had to get back to our jobs at some point.
Before any of this got published I contacted Lea so she could point out what needed fixing before "going live". It took her all but 3 minutes to find something. Turns out libsass has problems with interpolation. Not our problem - thank the Lords of Kobol!
Conclusion
Emscripten is a hell of a tool - if you know what you're doing. It's lacking - easily findable - documentation on how to interact with compiled code. I mean Interacting with Code is a great start, but doesn't cover "basic" things like structs and pointered parameters all that well.
Huge thanks go to Alon, Andres, Christian and Sebastian for making this work. I did all the failing so you guys could work things out. It was fun. Let's (not) repeat this (any time soon) ☺
Now go see the Sass.js demo and the libsass repository for trying to compile it yourself. Try shit out. Fail at things. Have fun. Sweat over compile errors at 2PM. Do things.
P.S. I really suck at C/C++.
The author does not allow comments to this entry
Comments
Display comments as Linear | Threaded