Spider is a platform for developing and coordinating distributed digital infrastructure. Each user runs their own instance of the base program. Each user’s base is responsible for both coordinating that user’s personal devices, as well as managing connections to other users’ bases. This way the system as a whole is supported directly by its userbase and no third party has outsized influence on its operation.
Since the users themselves are more involved with the operation of the system, it is important that the system is designed in a way that is easy to use. Advanced configuration settings should only be available to users who seek them out. Also, it should be easy to develop for this system, as part of its purpose is to make the development of distributed software easy enough for it to be ubiquitous.
Peripherals are any of the user’s software that connects to the base. Peripherals can come in a variety of flavors, since they can be run on any machine as long as they can connect back to the base. Peripherals that run on the same hardware as the base act like services, or background tasks. Peripherals can run on embedded hardware to allow the system to interact with the embedded device. The GUI that lets the user interact with the system is also a peripheral that has registered itself to be notified of changes to the system’s UI pages.
The base provides several things to the peripherals that connect to it. A peripheral may register a page for itself in the UI system. The base tracks these UI pages on behalf of the peripherals and when the GUI is connected it shows the most up-to-date version. This functionality is also responsible for routing UI inputs back to the correct owner of the UI page that experienced the input.
The base also stores data on behalf of the peripherals in ‘datasets’. A dataset is an ordered list of JSON-like elements. Public datasets can be accessed by all attached peripherals. Peripherals can also create and use datasets that are private, and only they can access. In addition to storing and retrieving data the dataset functionality can also interact with the UI functionality. A UI page element can be associated with a dataset. When this element renders, the renderer will copy that element once for each element in the dataset. The base element acts as a template and each of the copies has one of the elements in the dataset applied to it. When the dataset updates, the UI is also automatically updated.
Another set of functionality is the groups feature. The groups feature allows a set of users to share some data. The group data is replicated across each of their bases and all members must agree to updates that are applied to the data.
Finally, the base can route messages between peripherals and peers. When a peripheral sends a message it is sent with a tag name. Any other peripheral can subscribe to the tag name to receive all messages sent with that tag. Messages can also be sent to one or more peers.
The peer relationship is much more passive. Peers only establish a connection when some message needs to be sent from one to the other. The majority of data sent between peers will be messages sent by the peripherals, since they generally do not provide each other the more advanced services like the UI to each other. The exception to this is the coordination messages required by the groups functionality. The members of a group stay in contact with each other to ensure that the group data is properly synched.
To establish a connection between two peers the UI exposes a way to generate an invite code. When a peer connects with the invite code, that peer’s connection is automatically accepted by the base. To make this process easier the invite code is displayed on the screen as a QR code, which can be scanned in the other user’s GUI.
So far there are a few peripherals that have been built to work with the platform. Developing these peripherals helps inform the direction of development for the platform.
One of the first of these peripherals was the GUI peripheral. This was built in parallel with many of the UI features, especially the invite code functionality.
By itself the base and the GUI are not that useful. To test a more realistic scenario a simple chat peripheral, Synapse, was developed. In addition to its obvious usefulness, this also helps test the full message sending pathways. Messages must be sent from the UI to the peripheral, to the base, to the peer’s base, etc. This ensures that no step along the way was skipped.
Lastly, Sluice, a controller for garden drip hoses, was developed to test the durability of the connections and the reconnect logic. This also helped test the daily usability of the GUI/system as a whole. If part of the system fails or becomes sluggish, it will be apparent during the use of this peripheral.
While the proof of concept is working and able to support the daily use of the Sluice peripheral, there are still many improvements to be made. Currently the system has trouble establishing a connection from the wider internet back to the base. This is important when using the GUI from an external network. Adding support for some kind of overlay network would address this issue. Additionally, the datasets are still somewhat in flux. The exact API and storage system still has yet to be nailed down and finalized. And of course the system could always use more testing in general.
The repository link above has more information about installing and using the system.