Migrating users from centralized platform to unwalled.garden

[guoliu]

Hi everyone,

Thank you Paul for this exciting protocol and the excellent work on Beaker Browser! The 1.0 Beta version is amazing 😌

I'm a developer working on matters.news, a publication platform focusing on sane public discussions. It currently has around 40k register users and around 6m monthly visitor page views, mostly Chinese speakers around the globe.

We've been testing ways to bridge our users to p2p network, since we believe it's the only way moving forward as censorship intensifies and internet fragmentation continues (our site is already blocked in mainland China). We had a hard time choosing between Dat and IPFS, but decided to go with IPFS around 2 years ago. Since then we've integrated IPFS in our platform, utilizing public nodes to deliver contents, and prototyped a desktop client for publishing content without a server. But IPFS is still unstable albeit all the hype, and dynamic data with IPNS is far from production ready. Seeing the maturity of Beaker Browser and the proposal of unwalled.garden, we need to re-evaluate our decision.

Our original idea of the desktop application is essentially a WYSIWYG editor plus a RSS feed renderer, with some social features: an user can edit and publish content to her/his feed, subscribe to others' feed, and comment on others' content, which also gets published to their feed. We also need to bridge publications between matters.news and desktop client, so that the p2p network can have an attractive content base and user base to get through cold start.

From what I understand so far, the p2p RSS feed part is exactly what unwalled.garden is designed for. Instead of developing another desktop client, it looks like we can simply develop an unwalled.garden compatible web app, and direct our user to download Beaker Browser, enter the hash, while running a Dat daemon on our backend for migrating contents. But I still have several questions:

• What will the RSS/unwalled.garden support of Beaker Browser look like? For example, will it render RSS feed, allow user to subscribe, even allow user to comment? Or should them be implemented in a web app?
• Beaker Browser automatically setup a user's personal website compatible with unwalled.garden. If we want to implement more features, e.g. rich media WYSIWYG editor with image and video support, or BitCoin address for donation, is there a way to integrate into this personal website, or should it only exists inside a web app?
• One problem we need to solve is bridging a centralized server with p2p client for data persistency and discovery, similar to what SSB does with Pubs. Is this a general case that fits into the design of Dat/unwalled.garden?

I'm still exploring all the amazing works done in Dat/Beaker Browser space, so please excuse me if my questions are not yet well informed 😄

[pfrazee]

@guoliu ❤️ thank you for the kind words! Let's dig in here...

Seeing the maturity of Beaker Browser and the proposal of unwalled.garden, we need to re-evaluate our decision.

I hope we can pass muster! With the 1.0 Beta, we started fresh with a lot of the pieces and I think have established a much more stable ground. We've got growing pains of our own, so I don't want to over-promise our current maturity, but I'm very optimistic about our future. Mafintosh, Andrewosh, and the team working on Hyper technology are very focused on production-quality results.

That's all to say: you'll be growing with us, and I hope we'll be a good bet.

To wit: the Unwalled Garden site is a bit outdated but should get refreshed soon with the latest, and I'll share what's planned in this comment.

From what I understand so far, the p2p RSS feed part is exactly what unwalled.garden is designed for. Instead of developing another desktop client, it looks like we can simply develop an unwalled.garden compatible web app, and direct our user to download Beaker Browser, enter the hash, while running a Dat daemon on our backend for migrating contents.

That sounds right. It's "Hyper" instead of "Dat" now. Hyperspace is getting soft-launched today, and it should be the tool you'll want on the server side.

What will the RSS/unwalled.garden support of Beaker Browser look like? For example, will it render RSS feed, allow user to subscribe, even allow user to comment? Or should them be implemented in a web app?

I'm actively developing a blog reader right now (hyper://3ea7dfeb5dd2791193295fe976c5e312938fe5948f19c29ab2647c0502df79bc/). It should feel much like historic RSS readers, and we're still deciding on how to approach things like comments and blog-discovery.

Even our implementation will be a hyper app, so there's plenty of room to write your own reader if you're interested.

Beaker Browser automatically setup a user's personal website compatible with unwalled.garden. If we want to implement more features, e.g. rich media WYSIWYG editor with image and video support, or BitCoin address for donation, is there a way to integrate into this personal website, or should it only exists inside a web app?

The goal is that you can choose to do either depending on what you want to accomplish. If you wanted to do it inside a web app, the solution would be to create a separate "profile drive" than the user's primary, which you can then leverage however you see fit.

Compatibility with Unwalled Garden should be straight-forward: it defines an set of standards for publishing but doesn't exclude any kind of extension. Of the ideas you suggested, the hardest might be that we currently only use markdown (without HTML support) for blog posts. However, in microblogging (aka twitter) Unwalled Garden allows users to publish HTML which just gets displayed within an iframe. I would be very open to doing the same with blogging, which really opens you up to more customization.

We may ultimately want to extend the markdown format, because I think it's intensely valuable to have a content format like .md which includes zero UI logic and is therefore safe to embed anywhere. The lack of support for common content like videos is a problem for markdown. So on this particular issue, what I'd do is start by adding .html support, then let the discussion move to extending markdown if we feel the need.

One problem we need to solve is bridging a centralized server with p2p client for data persistency and discovery, similar to what SSB does with Pubs. Is this a general case that fits into the design of Dat/unwalled.garden?

This is precisely the design we intend to leverage. Our belief is that "supernode services" are pretty key to a system like this because you're ultimately limited by the capacity of user-devices. I like to think of it as caching layers: the supernodes are effectively mega-caches for hyper:// data.

We're going to be offering a supernode service which we upsell via Beaker, but it will be highly unopinionated (mainly rehosts hyper data and offers some generic services, like an AWS for consumers). For your use-case of handling discovery or bridging to the HTTPS web, there's a clear model of running your supernodes to provide that aggregation and HTTPS serving.

I'm still exploring all the amazing works done in Dat/Beaker Browser space, so please excuse me if my questions are not yet well informed 😄

Not at all, these are very perceptive questions! Please feel free to ask additional questions or for elaboration on any of the points I made.

[guoliu]

@pfrazee thank you for your fast and detailed response 😄 Lots of information for me to digest! I'm starting to appreciate the design philosophy of Dat/Hyper stack more and more as I dig deeper, and really hope we can find a nice fit, as well as a good starting point to contribute and grow together.

However, in microblogging (aka twitter) Unwalled Garden allows users to publish HTML which just gets displayed within an iframe. I would be very open to doing the same with blogging, which really opens you up to more customization.

Mounting HTML contents in iframe sounds very interesting, it opens the door to additional protection and style customization. Do you have examples of this? Curious to see how iframes work with hyper:// protocol in Beaker Browser.

Our belief is that "supernode services" are pretty key to a system like this because you're ultimately limited by the capacity of user-devices. I like to think of it as caching layers: the supernodes are effectively mega-caches for hyper:// data.

We're going to be offering a supernode service which we upsell via Beaker, but it will be highly unopinionated (mainly rehosts hyper data and offers some generic services, like an AWS for consumers). For your use-case of handling discovery or bridging to the HTTPS web, there's a clear model of running your supernodes to provide that aggregation and HTTPS serving.

Treating the supernode services as caching layers makes total sense to me, the services should act as a persistent seeder. When bridging a centralized web app to a p2p app, sometimes the service also needs to act as a delegate of the user, updating the data for the user (more detail below). Is there a way to switch identify/key pairs in hyperspace? Looking at the hyperspace client API, it seems that there's no concept of identify 🤔

---

Since most of our users are not technology-savvy, I've also started to think about how we can make the on-boarding experience as smooth as possible, while allowing the user to switch freely between https:// site and the hyper:// site. In an ideal on-boarding experience, an user can simply enter the domain name in Beaker Browser, get redirected to hyper:// address, login, get key pair and local storage setup, and start using the web app without https://. So far these all feel doable, and here are my rough ideas on how these could be achieved:

1. On a server, periodically scrape all related pages and assets to filesystem, replacing any http/s links with relative path, and publish the whole directory with hyperspace.
2. On the web app backend, create public and private key pair for each user, and a hyperdrive with corresponding profiles, posts, comments, follows, etc. according to unwalled.garden spec for each user. (Here's where a server needs to act as a delegate of the user, and I'm not yet sure whether it's possible via hyperspace)
3. Inside the web app, redirect user to the hyper:// address from step 1. When the user login for the first time, also transfer public and private key pair to local storage, and setup other local directories. The web app can now reads and writes with hyper:// only after initial setup.

Some parts in the above steps should be useful for migrating any https:// based web app to hyper://. Step 1 should make the site almost ready for read-only use, and could be useful for mirroring important contents like Wikipedia, Project Gutenberg or arXiv.

For step 3, we use GraphQL as our data layer, so switching to hyper:// simply means adding local resolvers to resolve against hyperdirve/Beaker API instead of the backend server. GraphQL wrapper of unwalled.garden could also be useful to other projects (someone already made an apollo link for Dat/Beaker!).

I'm curious to know if the above process sounds reasonable, or if there's other projects solving similar problems 😀

[guoliu]

I also have some questions for Beaker Browser in general, I'll post them here as well, but please feel free to direct me to other repos that are more suitable for these kind of questions.

• What are some known limitations of the current NAT traversal/hole punching mechanism? If the hole-punchable status is false, are there ways for the user to debug and find more information? After some initial testing, it looks like hole-punching works in most network conditions, but not working for cases such as over cellphone hotspot in mainland China. I suspect resolving connectivity issue will be a common case if we see mass adoption.
• My understanding is that Beaker Browser does not provide anonymity itself, but the user can use it with VPN or Tor. Are there consequences for connectivity when using together with VPN or Tor?

[pfrazee]

Okay sorry for the longer turnaround this time, I'll answer briefly but hopefully with enough information.

Mounting HTML contents in iframe sounds very interesting, it opens the door to additional protection and style customization. Do you have examples of this? Curious to see how iframes work with hyper:// protocol in Beaker Browser.

It's pretty straight-forward, the iframes act like they would in any other web context.

Is there a way to switch identify/key pairs in hyperspace? Looking at the hyperspace client API, it seems that there's no concept of identify 🤔

At present, every hyperspace structure (ie a hyperdrive) has its own identity which is tied to the public-key URL. There's not currently any way to transfer the identity to some other key, and we're debating whether to solve this with DNS or some other means.

I'm curious to know if the above process sounds reasonable, or if there's other projects solving similar problems 😀

Your approach sounds reasonable, my one observation is that you'll probably just want to maintain the keypair on their behalf for now. We'll need to cook up some kind of protocol for transferring keys between devices safely.

What are some known limitations of the current NAT traversal/hole punching mechanism? If the hole-punchable status is false, are there ways for the user to debug and find more information? After some initial testing, it looks like hole-punching works in most network conditions, but not working for cases such as over cellphone hotspot in mainland China. I suspect resolving connectivity issue will be a common case if we see mass adoption.

All NAT traversal fails about 10% on average, no matter what the implementation is. Having "supernodes" available via TCP can solve this, as they effectively act as proxies by storing-and-forwarding data. So long as one supernode is reachable, the data will be reachable by everyone.

My understanding is that Beaker Browser does not provide anonymity itself, but the user can use it with VPN or Tor. Are there consequences for connectivity when using together with VPN or Tor?

This has been an active discussion for a while and the reality is we don't have a solution yet. I haven't tested any VPNs or proxies yet, but we know it's something we'll need to tackle directly in the not-too-distant future.

[guoliu]

Finally got some time to try out the above ideas. After building a static site (exported from Next.js), I was able to share it on DAT. If I use dat sync --http and serve it via http, I can navigate the site on http://localhost:8080 without a problem.

If I open the DAT address in beaker browser, the site does render, but there are two problems:

• Request to server API was blocked by CORS policy. Are they any way of sending http request from a DAT web app in beaker browser to a server?
• Clicking <a> tag with href in relative path doesn't have any effect. It seems that hyper:// and dat:// are supported, but most website building tools build pages with relative path. Are they any way of supporting relative path similar to the behavior when serving over http?

Thank you!