Misreported used memory with the driver 535.129.03

<!-- Feel free to change the order of sections if appropriate. --> **Describe the bug** There's a mismatch between the used memory reported by `nvidia-smi` and gpustat. For example, right now, the former reports 11448 while the latter is 11961 (513 of difference). Also, I see the following warning being displayed: ``` <REDACTED_PATH>/lib/python3.10/site-packages/gpustat/nvml.py:145: UserWarning: Your NVIDIA driver requires a compatible version of pynvml (>= 11.510.69) installed to display the correct memory usage information (See #141 for more details). Please try `pip install --upgrade nvidia-ml-py`. ``` Even when I'm using the latest versions of gpustat (1.1.1) and nvidia-ml-py (12.535.133). **Screenshots or Program Output** ```console $ gpustat --debug <REDACTED_PATH>/lib/python3.10/site-packages/gpustat/nvml.py:145: UserWarning: Your NVIDIA driver requires a compatible version of pynvml (>= 11.510.69) installed to display the correct memory usage information (See #141 for more details). Please try `pip install --upgrade nvidia-ml-py`. warnings.warn( <REDACTED_HOSTNAME> Thu Jan 11 13:41:16 2024 535.129.03 [0] NVIDIA A10G | 25°C, 0 % | 11961 / 23028 MB | <REDACTED_USERNAME>(11420M) ``` ```console $ nvidia-smi Thu Jan 11 13:44:02 2024 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | | 0% 25C P0 57W / 300W | 11448MiB / 23028MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 N/A N/A 1584434 C <REDACTED_PATH>/bin/python 11420MiB | +---------------------------------------------------------------------------------------+ ``` **Environment information:** - OS: Ubuntu 22.04.3 LTS - NVIDIA Driver version: 535.129.03 - The name(s) of GPU card: NVIDIA A10G - gpustat version: 1.1.1 - pynvml version: 12.535.133