Amir Chaudhry

thoughts, comments & general ramblings

Unikernels and Modularity at GlueCon 2016

Above are the slides from my talk at GlueCon today. Following the overview, I spend a little more time describing the benefits of having an ecosystem of libraries we can use for unikernels. The demo I gave this time was a little more involved and showed how we can build unikernels using your normal development approaches but then re-target the same application to different backends (i.e. for ARM and the Internet of Things).

Below are some details of how you can step through the demo for yourselves and you can find more information about unikernels over at unikernel.org.

Prerequisites

The following instructions use Docker for Mac, which is currently in beta, but should work with any Docker installation. In order to deploy to a Cubieboard2, you need to have a board set up and running Xen. You can follow the relevant instructions for details.

The 2048 game I used here was originally put together as a tutorial for teaching OCaml and using js_of_ocaml. If you’re interested, you can find more instructions and follow the tutorial yourself on over on the original repo.

Building the 2048 game for the Unix backend

I used Docker for Mac, together with pre-prepared Docker images to speed things up. Bear in mind that things change across beta releases so if the following doesn’t work, please tweet me.

The first thing to show is how you can do development by targeting unix and using the appropriate system libraries. The Docker image already has a clone of the repo so you can use that. To follow my steps from the demo:

# On your terminal
$ docker run -it -p 80:8080 amirmc/gluecon2016-unix bash

# Inside the container
$ eval `opam config env`
$ cd gluecon2016
$ mirage configure --unix --net socket
$ make
$ sudo ./mir-www

You’ll see that the tool runs some checks and tries to install any necessary components for the target — in this case the unix-related libraries.

Once you’ve completed the above steps, you can point your browser at localhost to see the site. Congratulations, you should see the unix-based version of the 2048 game! Try not to get too distracted by playing. The next step is to use the same code base to build an ARM-based unikernel that will run on the Cubieboard2.

Building 2048 for the ARM backend

You will again use Docker but this time with the ARM images, based on resin/armv7hf-debian (NB: This is not a cross-compilation step). You can build the Xen image in the container using Docker but in order to deploy, you’ll need the Cubieboard2 already set up. This time, you can clone the demo repo on your machine and mount it into the container. The steps from this part of the demo were:

# Clone the repo and mount it in the ARM container
$ git clone https://github.com/amirmc/gluecon16.git
$ cd gluecon16
$ docker run -it -v `pwd`:/src amirmc/gluecon2016-arm bash

# Inside the container
$ eval `opam config env`
$ cd /src
$ mirage configure --xen --dhcp=true --network=0 --no-opam
$ make

The --no-opam flag is used because the container already has the necessary packages and you can skip that check. The --network=0 is to override a default in the config.ml file.

You should now have a file on your Mac called mir-www.xen. This is the binary you need to copy over to the Cubieboard2. You’ll notice an additional file, www.xl, which contains some configuration and will also need to be copied over (you will need to update the path in that file). Once these files are on the Cubieboard2, then it’s a simple case of sudo xl create -c www.xl. The IP address of the unikernel will be displayed and you can navigate to <IP address>:8080 from your browser. You’re now playing the 2048 game from a unikernel being served from the Cubieboard2! During this re-targeting, none of the application code had to be changed, the appropriate system libraries were pulled in at the right time.

Although the above may seem quite involved, it does demonstrate how we can use the existing tools to to develop and deploy applications that span both the cloud and IoT. Over time, the tooling for unikernels will improve and mature so that all of the above will become much easier!

Other resources

To get involved in the development work, please do join the unikernel forum, the MirageOS devel list, and try out some of the examples for yourselves!

Thanks to Mindy for reviewing an earlier draft.

Share / Comment


Notes on my Craft Conf Talk

These are the slides from my talk at Craft Conf today. If you’re reading this at the conference now, please do follow the link at the end of my talk to rate it and give me feedback!

Some of the specific items I mention are below, with links to more detail. You can find more information and resources about unikernels over at unikernel.org.

Build the MirageOS website with Docker for Mac

I quickly showed people how to build and serve the MirageOS website from a container on my Mac. I used the Docker for Mac beta (beta 8) for this, together with a prepared Docker image. Bear in mind that things change across beta releases so if the following doesn’t work, please tweet me.

To follow my steps in the demo:

# On your terminal
$ docker run -it -p 80:80 amirmc/demo-craft bash

# Inside the container
$ cd mirage-www/src
$ mirage configure --unix --net socket
$ make
$ sudo ./mir-www

You should be able to see the site by pointing your browser to localhost! Congrats, you just built MirageOS! You can read the hello world guide on the website to understand more about the configure flags we used.

The above image (amirmc/demo-craft) is one I made just for the conference, so that I didn’t have to wait for additional downloads. To do it ‘properly’, it’s better to use the unikernel/mirage images, since those are maintained. The mirage tool within the container will then pull down the required dependencies based on the mirage configure step.

# Clone the repo on your machine
$ git clone https://github.com/mirage/mirage-www.git
$ cd mirage-www

# Pull and run the maintained image, mount
# the local directory, and set up the ports
$ docker run -it -v `pwd`:/src -p 80:80 unikernel/mirage bash

# Now that you're in the container
$ cd src
$ mirage configure --unix --net socket # installs packages
$ make
$ sudo ./mir-www

Then point your browser at localhost.

Using Docker for Mac makes all of the above much easier, especially if you don’t have (or don’t want) a local OCaml environment set up.

The Bitcoin Piñata

This is a unikernel, built using the TLS stack, which holds the private key to some bitcoin. If you manage to break in then you can simply transfer the bitcoin away. The model itself is a novel way to run a bounty programme, since a successful breach leads directly to the reward — though the main benefit is to stress-test the whole stack in the wild. In the talk, I describe how the unikernel is about 4% the size of its equivalent in a traditional OS and you can read the article for more detail about that (and much more).

Other resources

To get involved in the development work, please do join the unikernel forum, the MirageOS devel list, and try out some of the examples for yourselves!

Share / Comment


A logo for unikernels!

I’ve been working towards a logo for unikernels and it’s time to gather feedback from the unikernel community! There’s been a 99designs contest running for a while — where designers have submitted just over 400 entries — and a short-list for community feedback is now at https://99designs.co.uk/logo-design/vote-99gjyi (link open until at least 15th Jan).

At this stage, the aim is to gather wider feedback to inform the next step. The final design will be selected at a later date — after further iteration with designers. Once a logo is chosen, it will be used on unikernel.org and will help inform the design of the rest of the site (with input from a professional firm).

For more background you can read the design brief and you can even look through all the entries.

The unikernel approach as a concept has been getting much more attention recently and this will only increase throughout 2016.

Since there are a number of different projects — each approaching the problem from a different perspective — it will become even more important to ease the process of bringing in new users and contributors across all the projects.

The website at unikernel.org is already helping with this but we will also need a simple visual identity to go alongside it. Building this kind of brand will greatly improve the standing of unikernels as a whole and will be just as important as the work going on in each of the implementations. Ultimately, this will help to increase the rate of adoption.

What about projects’ existing logos?

To avoid any confusion, I’ll say upfront that this logo is not intended to replace any project’s existing brand! It’s meant to help us represent unikernels as a whole and each project will make it own choices, just as they do currently. The logo will mainly be used on unikernel.org and projects can make use of it if they wish.

Projects have already benefited from shared code and the same will be true from this shared brand.

What now?

Share / Comment


CodeMesh 2015

These are the slides from my talk today at CodeMesh. This time around I was earlier in the schedule so I get to enjoy the rest of the conference! If you’re reading this at the conference now, please do follow the link in my talk to rate it and give me feedback!

The specific items I reference in the talk are below with links to more information.

Security and the Bitcoin Piñata

This is a bounty where we have locked away some bitcoin in a unikernel that is running our new TLS stack. This was a new model of running a bounty and has proven a great way to stress test the code in the wild.

You can follow up with more of the background work on the TLS stack by looking at the paper, “Not-quite-so-broken TLS: lessons in re-engineering a security protocol specification and implementation” and find other users of the libraries via https://nqsb.io.

Automated deployment

I’ve previously written about how we do unikernel deployments for MirageOS. Although the scripts themselves have evolved and become more sophisticated, these are still a good introduction.

Summoning on demand

The work on summoning unikernels was presented at Usenix this year and you can read the paper, “Jitsu: Just-In-Time Summoning of Unikernels”. The example I showed in the talk can be found at http://www.jitsu.v0.no.

Other resources

To get involved in the development work, please do join the MirageOS devel list and try out some of the examples for yourselves!

Share / Comment


Governance of OCaml.org

Governance Screenshot

For several months, I’ve been working with the maintainers of OCaml.org projects to define and document the governance structure around the domain name. I wrote about this previously and I’m pleased to say that the work for this phase has concluded, with the document now live.

Recurring themes

There were some recurring themes that cropped up during my email discussions with people and I thought it would be useful to present a summary of them, along with my thoughts. Broadly, the discussions revolved around the philosophy of the document, the extent of its scope, and the depth of coverage. This discourse was very important for refining and improving the document.

Ideals and Reality

Some of the comments I received were essentially that the document did not represent how we should be organising ourselves. There was occasionally the sense (to me at least) that the only appropriate form of governance is a fully democratic and representational one.

That would entail things like official committees, ensuring that various communities/organisations were represented, and perhaps establishing some form of electoral processes. Overall, something relatively formal and quite carefully structured. Of course, instituting such an arrangement would necessarily require somewhat involved procedures, documentation, and systems — as well as the volunteer time to manage those processes.

These may be noble aims — and I expect one day we’ll be closer to such ideals — but one of the critical factors for the current approach was that we record how things are right now. In my experience, anything else is purely aspirational and therefore would have little bearing with how things currently function.

To put it another way, the current document must not describe the structure we desire to have, but the organisation we actually have — warts and all. Yes, right now we have a BDFL*, who personally owns the domain and therefore can do as he pleases with it. Irrespective of this, the community has been able to come together, coordinate themselves, and build very useful things around the domain name. This has happened independently of any formal community processes and, in my view, has largely been driven by people supporting each other’s works and generally trying to ‘do the right thing’.

Another aspect to point out is that is that such documents and procedures are not necessary for success. This is obvious when you consider how far the OCaml community has come in such a relatively short space of time. Given this, one might argue why we need any kind of written governance at all.

To answer that, I would say that once things grow beyond a certain scale, I believe it helps to gather the implicit behaviours and document them clearly. This allows us to be more systematic in our approach and also enables newcomers to understand how things work and become involved more quickly. In addition, having a clear record of how things operate in the present is an invaluable tool in helping to clarify what exactly we should work on changing for the future.

Extent of scope

It’s a little confusing to consider that ‘OCaml.org’ is simultaneously a collection of websites, infrastructural components, and projects. Disambiguating these from the wider OCaml community was important, and relatively straightforward, but there were a few questions about the relationship between the domain name and the projects that use it.

Although the governance covers the OCaml.org domain name, it necessarily has an impact on the projects which make use of it. This matters because anything under the OCaml.org domain will, understandably, be taken as authoritative by users at large. In a way, OCaml.org becomes the sum of the projects under it, hence it’s necessary to have some lightweight stipulations about what is expected of those projects.

Projects themselves are free to organise as they wish (BDFL/Democracy/etc) but there are certain guiding principles for OCaml.org that those projects are expected to be compatible with (e.g. openness, community-related, comms, etc). These stipulations are already met by the current projects, so codifying them is intended to clarify expectations for new projects.

Depth of coverage

Another of the recurring points was how the current document didn’t capture every eventuality. Although I could have attempted this, the end result would have been a lengthy document, full of legalese, that I expect very few people would ever read. The document would also have needed to cover eventualities that have not occurred (yet) and/or may be very unlikely to occur.

Of course, this is not a legal document. No-one can be compelled to comply with it and there are very few sanctions for anyone who chooses not to comply. However, for those who’ve agreed to it, acceptance signals a clear intent to take part in a social contract with the others involved in work around the domain name.

Overall, I opted for a lightweight approach that would cover how we typically deal with issues and result in a more readable document. Areas that are ‘unchartered’ for us should be dealt with as they have been so far — through discussion and action — and can subsequently be incorporated when we have a better understanding of the issues and solutions.

A solid starting position

The current version of the governance document is now live and it is very much intended to be a living document, representing where we are now. As the community continues to grow and evolve, we should revisit this to ensure it is accurate and is meeting our needs.

I look forward to seeing where the community takes it!

In case you’re interested, the set of links below covers the journey from beginning to end of this process.

* Yeah, I made sure to add Xavier to the BDFL list before publishing this. :)

Thanks to Ashish, Philippe and Anil for comments on an earlier draft.

Share / Comment