ZooKeeper - new membership provider

[shayhatsor]

#69 says:

Statically adding a new implementation to the code, and hooking it all the way in the MembershipFactory - sure, great idea! I am in fact planning to post as an issue an open "student idea" for hookup Zookeeper as a membership provider.

I think that membership providers configuration should be consistent with storage providers. This would make Orleans cleaner and alleviate dragging dependencies to various DBs.
While adding this extensibility, It seems natural to remove the GatewayProviderType and LivenessProviderType enums. This will ease the removal of dependencies on Azure. (#136).

Going back to the main issue of Zookeeper.
ZooKeeper Programmer's Guide states that:

ZooKeeper does not guarantee that at every instance in time, two different clients will have identical views of ZooKeeper data. Due to factors like network delays, one client may perform an update before another client gets notified of the change.

Consequently, Do we need to use a Zookeeper Recipe like Shared Reentrant Read Write Lock?
If we only use a Shared Lock for writing, a silo might get a stale value of the membership table. I'm not sure how Orleans deals with this scenario as the current implementations (Azure,Sql Server) guarantee fresh values.

[gabikliot]

@shayhatsor, I think you had a typo.
I never said I see no value in allowing pluggable membership providers. I said exactly the opposite: "I am all for allowing more membership providers. The question is how exactly, right?"

[gabikliot]

Have you read this: http://dotnet.github.io/orleans/Runtime-Implementation-Details/Cluster-Management.html? It should answer a lot of questions and explain why we went with Azure table and not ZK. There are a bunch of really good technical reasons.

[shayhatsor]

I have read the Cluster Management article. I understand the reasons you chose Azure table over ZK as its support for the protocol is complete out of the box. While looking for an open-source solution for the membership table persistence, I found ZK to be an option using a readers writer lock recipe.
Before starting implementation, I've created this issue to get your input.

[gabikliot]

OK, I think we are in a full agreement on usefulness of allowing to use ZK as yet another membership provider. And I am very excited that you are willing to contribute towards that effort.

The question now is how exactly we are going to integrate with ZK.
First, let me please start with a number of technical "administrative" questions:

1. Is there C# client to ZK? If not how do you plan to integrate with it?
2. How do you plan to deploy it? Will ZK be running as service or would it be responsibility of whomever runs the Orleans service run the ZK nodes as well?
3. If run as a service, do you know of any Cloud provider providing ZK as a service?
4. If not run as a service, do you plan to run them on the same machines or different machines?
5. Is it meant to run in the Cloud or only on premises?

Now to more low level technical details.
Frankly, I am not an expert on ZK. But I have read a bit.

Under Zookeeper Recipes they mention they provide membership out of the box almost. This sounds great at the first look. But reading deeper about ephemeral nodes, my understanding is that the way it works is the following: every client (Orleans silo in our case) connects to the ZK service (master or one of the seed nodes among the ZK cluster) and is represented by an ephemeral node. The client runs I am alive heartbeats between himself and this ZK node. If heartbeats stop, ZK removes this ephemeral node, thus declaring the client disconnected/dead.
Please correct me if my understanding is wrong.

Now, if my understating is correct, what follows is the unfortunate property of this implementation in which if a silo temporarily disconnects from the ZK service, it will result in its disconnecting from other silos, and they will consider it dead. This is undesirable, since the silo may in fact be completely healthy and have a perfect connectivity to other silos, just not to ZK nodes.

In our description here Property 7 under "Properties of the Basic Membership Protocol and FAQ" we explicitly discuss this sutuation and describe how our Azure table based membership is resilient to those kinds of failures.

Perhaps you had in mind a different way to integrate with ZK? I would be very interested to hear. You mentioned "Shared Reentrant Read Write Lock". I am not sure how those could be used, but would be very interested to learn. I am not familiar with how locks work in ZK.

As for your question about client consistency: "ZooKeeper does not guarantee that at every instance in time, two different clients will have identical views of ZooKeeper data"
This sounds OK. Orleans membership has the same property. Every silo learns about the membership configurations asynchronously, at different times. What is important is that:

1. eventually all silos learn about the latest configuration,
2. this eventually is relatively fast and
3. that the membership configurations they learn are always going forward (every silo sees the configurations in the increasing order) [the last going forward property is not strictly required and Orleans can work even without it, as long as eventually they will all learn about the latest configuration, but this property is very desired].

Gabi Kliot

[shayhatsor]

Thanks a lot for the quick response, I'm also very excited to be able to contribute.
about the administrative questions:

1. There's a C# client on GitHub and a Nuget package. It is not async. I plan make it fully async.
2. The deployment will be the responsibility of whomever runs Orleans. The ZK ensemble would be on-premises.
3. We plan to not run it as a service.(there is such a service in IBM Bluemix). In production, We plan to have hundreds of silos in an Orleans cluster. All silos will sync membership via a Zookeeper ensemble of maybe 5 servers. These will run on different machines than the Silos.
4. On-premises.

Now the low level technical details:

Please correct me if my understanding is wrong.

I believe your understanding of ZK ephemeral nodes is correct.

Perhaps you had in mind a different way to integrate with ZK?

I have one. My goal is to mimic Azure and follow Orleans current membership protocols to the letter. Today, I learned that using a ZK recipe for implementing locks is an overkill, there's native support for transactions.
ZK overview states:

Znodes maintain a stat structure that includes version numbers for data changes, ACL changes, and timestamps, to allow cache validations and coordinated updates. Each time a znode's data changes, the version number increases. For instance, whenever a client retrieves data it also receives the version of the data.

For example, updating a membership table entry requires the transaction to check the table version and entry version, and commit the update only if both are as expected. This can be done with multi. This method is supported since version 3.4 ,unfortunately the C# client doesn't have support for that yet, I'll have to add it.

About the guarantee requirements:

1. eventually all silos learn about the latest configuration,
2. this eventually is relatively fast and
3. that the membership configurations they learn are always going forward

Looking at ZK Guarantees it provides all of them:
guarantee requirements 1 and 2 are Timeliness , 3 is Sequential Consistency.

[gabikliot]

@shayhatsor , this is great!
I like this direction very much. It is also going to be much easier to integrate this way.
Essentially, Orleans membership is already written in the way to allow such extensibility. All you need to do is to implement IMembershipTable, which is basically just needs to support one row read, all rows read and conditional transactional 2 rows update or insert operations (rows like znodes in ZK). I just updated the IMembershipTable interface file with more documentation details via #396. You can also look at the 3 implementations of this interface, the simplest to look at is the InMemoryMembershipTable, which just mimics the table in memory. It makes understating the semantics of those operations very easy.

You will also need to implement IGatewayListProvider, which is just a read only view on this table.

There will be a later question about Reminders table. I believe it can also be implemented over ZK. The only challenge there would be to support range query efficiently. But we can leave that question on the side for now. If one does not need Reminders, Orleans can be configured to use one implementation for Membership and another (or none) for Reminders. So one can mix and match and maybe others will implement Reminders over ZK in the future. So I would not worry about it now.

Please let me know how I can help.

[sergeybykov]

@shayhatsor A ZK-based implementation of membership would be a great addition to the list of configuration options. Membership storage is already pluggable in the sources, as Gabi described. We could make it into a real plugin design with dynamically loaded implementations, but I think that's a low priority part at this point, as we don't expect too many of such implementations.

I'm less sure about the value of using ZK for reminders, but if it works, why not.

[shayhatsor]

@sergeybykov I agree with you about the real plugin design being low priority. I pointed it out because I started learning Orleans three weeks ago and Azure is mentioned throughout the code. I had to read the entire Orleans documentation to figure out that Azure is optional, even then I missed the part about reminders.
@gabikliot Thanks for adding the documentation. I think it might be missing a description to when the implementation can throw exceptions, and what are the side effects. The only mention is in UpdateIAmAlive method which describes that the returned task should fail (throw exception) only due to network problems or table unavailability which seems to be true for all other returned tasks in that interface.

[sergeybykov]

I had to read the entire Orleans documentation to figure out that Azure is optional, even then I missed the part about reminders.

@shayhatsor You are right, we started initially with Azure in mind, and it's still not obvious that we do not have a strong dependency on Azure. Any PRs you can submit against https://github.com/dotnet/orleans/tree/gh-pages would be greatly appreciated.

[gabikliot]

@shayhatsor, you are right: all those methods may throw due to network problems or table unavailability or in general due to any other system problem, such as out of memory for example. On the other hand, those methods are not expected to throw in order to express application semantics. For example, all update/insert methods are supposed to return false if update failed due to a legitimate concurrency control failure (etag mismatch) and not throw "Storage etag mismatch" exception instead.
So:

1. All conditional updates can fail to make the update due to table semantical reasons and should signal this by returning false.
2. UpdateIAmAlive cannot fail to make the update due to any table semantical reasons.
3. All methods can fail due to system reasons.

Of course, there are also other less clear cases of errors, such as what to do if the table is suddenly not there. It can in theory happen if someone by mistake deleted it. What should all those operations do? This is considered a system error (since this scenario is not part of the allowed semantics of this protocol), so they should throw.

[shayhatsor]

Update: I was trying to make the current ZooKeeper C# Client have the same capabilities of the java counterpart. After several days of work, I've found out that it's more outdated than i originally thought. Consequently, I've converted the entire java client codebase of the latest stable version of the ZooKeeper 3.4.6 to C#. and made async versions of all methods. currently it's almost finished, all tests I could convert from the java version are passing consistently (currently 81 test). I'll organize it a bit more and perform a pull request on ZooKeeper main branch. the current ZooKeeper C# Client isn't merged with the main branch, it's a fork. I hope my version will be adopted as part of the official version. Meanwhile, I'm starting to work on integrating ZooKeeper with Orleans.

[sergeybykov]

@shayhatsor Sounds like a good progress despite more work than you anticipated. :-) Thanks for the update!

[ghost]

@rgs1, with ewhauser/zookeeper#29, can we have C# implementation in official ZK repository, targeting CoreCLR / CoreFX surface area for cross-platform .NET support?
copy @ewhauser

[shayhatsor]

just made a pull request : #493.

@jasonwilliams200OK, @ewhauser's version is outdated. ewhauser/zookeeper#29 is my PR... but it's an "old" one.
Since then, I've actually reconverted the entire ZK java client code base to the latest stable version 3.4.6. which you can find a PR for the official repository apache/zookeeper#31. Currently, the code is CLSCompliant, but i'll need to check about making it CoreCLR/CoreFX compliant. (any help will be appreciated).