Match other implementations

* Avoid inlining the destructor * ~~Use a load specifically on `count` instead of a `fence`, this is what std and triomphe do.~~