Leave and attempt_simple_transfer
In recent refactoring of cluster changes - https://github.com/basho/riak_core/pull/913 and https://github.com/basho/riak_core/pull/967 - the focus has been on behaviour on `join` operations, and not `leave`.
Recently a customer achieved a balanced ring through a join (as expected with the new cluster claim algorithm), but then leave plans kept creating unbalanced rings i.e. rings whereby partitions were unevenly distributed leading to the potential for "one slow node" problems.
...
[EDIT] There were various incorrect statements made initially here, about how leave works. See [later comments](https://github.com/basho/riak_core/issues/970#issuecomment-929367998) for a more accurate representation of the problem
...
There are perhaps some simple things that can be done:
1. Perhaps a configuration option to force rebalance on leave. This can then be enabled for users of location. This perhaps could also be enabled if repeated re-planning is not leading to balanced outcomes.
2. The `attempt_simple_transfer/4` could check how many partitions are already owned by. a candidate node, and prefer candidate nodes with lower levels of existing ownership. This should be more likely to give balanced results (although it will do nothing in terms of location awareness.
There be demons here. I don't think there's a lot of existing test coverage of leave scenarios (riak_test/src/rt.erl has a `staged_join/1` function but no `staged_leave/1`). There could be the potential for confusing situations when the configuration setting (1) changes between nodes and between staging and committing changes.
关闭于 2023-01-02 14 条评论