Research Project

Process Isolation

  • Expanded the shim with so called PipeTriggers.

    • Wrote an example for this (examples/pipes/main.rs).
    • Wrote the specification for this application to be launched with the shim (examples/pipes/spec.json).
    • I think this is very neat and certainly a focal point of the project. This style of request driven programming that the shim enables with a powerful level of privilege dropping each time should be useful.
    • The spec is also pretty easy to parse and even write for this too. It’s getting slightly hard to validate mentally, so a general purpose validation function is written to validate it programattically before launching (Specification::validate).

  • Built the echo example, still in C, ready to be demoed with the shim (examples/echo/echo.c).

    • Started preparing a specification for the echo server (examples/echo/spec.json).
      • This is a work in process to test new features added to the spec format, it doesn’t run yet and needs additional support before it will.
    • Having some trouble nailing down an abstraction of networking that provides enough for existing apps to work, and also supports new apps in a nice and abstract way.
      • I think this might result in two methods: an IP-network firewall based method for outbound traffic, e.g. this process should be able to access my database network on §10.152.163.0/24` but not the public Internet. Then a second abstraction, perhaps in the form of give me specific ports, that allows inbound traffic. Not clear yet though.
    • The specification format still feels pretty good for a more complex app like this, even if it is getting a tad verbose.
    • The dataflow created here of App Listener -> Shim -> n*Request Handler feels pretty good for any request driven development. Although the listener could probably be a library function, it’s still neat to write and simplifies the C program significantly, while providing good isolation between the two.

  • Started work on integrating this solution more tightly with a language (in this case, Rust).

    • Looking into a macro based solution, but I’m not sure it’s that much superior to straight up code.
    • Unfortunately going anywhere near writing macros was causing my IDE to crash, so this needs to be solved before looking more into it.
    • Care must be taken that the language specific bits don’t overtake the language agnostic bits.

Multi-Process Monitor (forktop)

  • Mixed progress on this one. On trying to write a Terminal User Interface for this one, I was disappointed with the available options. Therefore I started writing my own… (stateful-tui).

    • Has support for use\_state, a hook that allows TUI components to store state over multiple renders.
    • Has support for use\_effect, giving the components the ability to run arbitrary asynchronous code when things change.
    • Has partial support for component children, allowing composition of components.
    • Has no (but requires) support for keyboard/mouse input. Keyboard is definitely a requirement. This should be easy to implement…

  • Though this now has a project locality value of three (shim -> forktop -> stateful-tui), I think it is a valuable addition to the OSS ecosystem.

Summary

It’s been a slow couple of weeks, but the groundwork has been laid for some good progress to come now.

Up Next

  • Now that the design of the networking system is starting to be figured out, it’s time to implement it. By next time I MUST have implemented the echo server under the shim.
  • A design for filesystem access MUST be made. This is going to involve research into how others do it, given that it’s such a complex area.
  • Getting forktop to a viewing stage, if not an interactive stage, SHOULD be completed.

Modules

Introduction to Systems Research

  • Read Borg (Verma et al., 2015), Fuxi (Zhang et al., 2014), and Firmament (Gog et al., 2016). Reviewed Borg.
  • Read MapReduce (Dean and Ghemawat., 2004), Dryad (Isard et al., 2007), and Pregel (Malewicz et al., 2010). Presented MapReduce.
  • Exciting stuff here, the cluster scheduling stuff in particular is very cool. Lightweight cgroup containers have been popular for some time, and I think these readings do give good context for this project.