To celebrate the release of 1.0 of Unison, I am excited to share the app that I have been developing for the past few weeks - atproto-experiments
Unison Programming Language, along with its Cloud platform, is unique in many aspects. My core interests have been their intersection: Typed Functional Programming and Distributed Systems. Unison has some innovative cloud libraries like
Volturno - a large-scale streaming framework with exactly-once processing. It's like Kafka without pain.
Arcella - a library of append-only data structures, implemented solely on top of object-storage.
I want to play around with these libraries, so I was looking for a use case.
Use Case
I am trying to ensure that all my social activities are driven via AT Protocol. That way, I know where my data resides. Now, to move the published data to various app views requires either the publishing platform providing a specific feature or one's programming skills. I chose the latter.
The nice thing about AT Protocol is that even though the development libraries and framework ecosystem is clustered mostly around TypeScript or Rust, I don't have to. The communication is on top of HTTP, and PDSs are just repositories that handle authentication but have no application logic. So, it all boils down to parsing from firehose (Jetstream), manipulating data in the PDS - all based on relevant lexicons, and repeat.
Foundation
The foundation of working with AT Protocol is built on top of Unison's rendition of direct-style algebraic effects - abilities. ATProto's ability will handle operations within the AT Protocol ecosystem, such as making API calls. Xrpc ability handles the details of HTTP API calls, including the various request types such as Query, Procedure, and Subscription. Lexicon's schema definition languages are encoded using the Schemas library, which embodies the aspect of protocol-agnostic description of datatypes, which then gets encoded/decoded into various serialisation technologies, one of which is JSON.
Tools
The first implemented tool synchronises Bluesky replies to Leaflet documents as comments. The sync system uses a streaming architecture powered by Volturno pipelines. It maintains two Jetstream subscriptions: one for the authenticated user's interactions and another for public interactions on tracked collections. Unison's cloud infrastructure provides a way to represent persistent processes or long-running workflows as Daemons.
Volturno pipelines process source relevant Jetstream events which are pushed to the KLog via multiple KStream workflows
it segregates incoming events from processed events
it uses the
Eventsdata structure from theArcellalibrary as the event store to store all processed events and then emits the unique event to the attached sinks
Sinks are where side effects occur as part of the final stage of the overall Pipeline.
Create Leaflet comment sink
The sink prepares the data for creating the Leaflet comment and, finally, makes the relevant createRecord call to update PDS with a new comment. However, one crucial step is to figure out some details about the original Leaflet post, which was shared in Bluesky, and to that shared post, this new reply is made. Large indexes are doing this kind of backlinking, another set of applications on top of the same AT Protocol.
We use Constellation's APIs for those backlinking queries at the moment. However, one of the primary reasons to use our own Event Store is to use the same in due course for such queries.
Here are the logs from the Unison Cloud console depicting the syncing
Up Next
The whole point of creating the foundation is to build more tools on top of the same, like the Bluesky replies-Leaflet comments sync. Some ideas:
custom bridge for other federated social systems
backlink index that could leverage Arcella's incremental map-reduce jobs