Handoff key filtering (repair or ring resize handoff) does not send periodic sync messages to keep TCP connection alive
Confirmed in the lab: while doing repair handoff, the filtering done appears to prevent sending batches of handoff data to the receiver. The receiver's default timeout is 5 minutes, but it's see-it-with-my-own-eyes possible to fold over Bitcask data at over 150MBytes per second and not have enough data to send to trigger the receiver's 5 minute idle timeout.
After the reciever has timed out, the #ho_acc record looks like this (from a stack trace from `riak-admin cluster-info`:
```
y(0) {ho_acc,6,ok,#Fun<riak_core_repair.0.61411323>,riak_kv_vnode,<0.588.0>,#Port<0.8040>,{593735040165679310520246963290989976735222595584,616571003248974668617179538802181898917346541568},{ho_stats,{1395,157067,590863},undefined,125,128161535},gen_tcp,286632,158788528786,[<<803113 bytes>>],1,803113,25,repair,undefined,undefined}
```
Note that element 2, the ack number, is 6. And it remains at 6 for minutes at a time, confirming the hypothesis that the batch isn't big enough to send yet to the remote side.
When the batch finally becomes big enough, we see that the ack number only reaches 7: only one send is required to change the element 3 from `ok` to either `{error,closed}` or `{error,enotconn}`, depending on the timing.
```
{ho_acc,7,{error,closed},#Fun<riak_core_repair.0.61411323>,riak_kv_vnode,<0.588.0>,#Port<0.8040>,{593735040165679310520246963290989976735222595584,616571003248974668617179538802181898917346541568},{ho_stats,[....]
```
Packet capture by tcpdump confirms that there is 5 minutes of inactivity before the receiver closes/sends FIN.
关闭于 2023-02-27 2 条评论