### Intro (https://docs.nvidia.com/cuda/curand/index.html):

The cuRAND library provides facilities that focus on the simple and efficient generation of high-quality pseudorandom and quasirandom numbers. A pseudorandom sequence of numbers satisfies most of the statistical properties of a truly random sequence but is generated by a deterministic algorithm. A quasirandom sequence of 𝑛 -dimensional points is generated by a deterministic algorithm designed to fill an 𝑛 -dimensional space evenly.

cuRAND consists of two pieces: a library on the host (CPU) side and a device (GPU) header file. The host-side library is treated like any other CPU library: users include the header file, /include/curand.h, to get function declarations and then link against the library. Random numbers can be generated on the device or on the host CPU. For device generation, calls to the library happen on the host, but the actual work of random number generation occurs on the device. The resulting random numbers are stored in global memory on the device. Users can then call their own kernels to use the random numbers, or they can copy the random numbers back to the host for further processing. For host CPU generation, all of the work is done on the host, and the random numbers are stored in host memory.

The second piece of cuRAND is the device header file, /include/curand_kernel.h. This file defines device functions for setting up random number generator states and generating sequences of random numbers. User code may include this header file, and user-written kernels may then call the device functions defined in the header file. This allows random numbers to be generated and immediately consumed by user kernels without requiring the random numbers to be written to and then read from global memory.

SUMMARY: you have host (cpu) and device (gpu) random generators. there are a bunch of random gen algos which generate nearly completely random numbers given a “seed” for reproducibility. this means that if I want to generate 1 million random numbers with seed 314, 

I can use 314 again whenever I want to get those same random numbers. you just need to be using the same precision (default int64 I think), algorithms, tensor size (maybe this is just a different interpretation of *offset* & *order*), and the seed itself. 

for the purpose of this course, we would only use random numbers for weight inits, which is negligible in the whole training process (we care about faster forward and backward passes in the duration of the training/inference run versus the first few seconds of initialization)

Seed:

The seed parameter is a 64-bit integer that initializes the starting state of a pseudorandom number generator. The same seed always produces the same sequence of results.

Offset:

The offset parameter is used to skip ahead in the sequence. If offset = 100, the first random number generated will be the 100th in the sequence. This allows multiple runs of the same program to continue generating results from the same sequence without overlap. Note that the skip ahead function is not available for the CURAND_RNG_PSEUDO_MTGP32 and CURAND_RNG_PSEUDO_MT19937 generators.

Order:

lots of content so read here ⇒ https://docs.nvidia.com/cuda/curand/host-api-overview.html#order

- **CPU**: On the CPU, PyTorch primarily uses the **MT19937** variant of the Mersenne Twister (referenced by the `std::mt19937` class in C++) as its random number generator.
- **GPU**: On GPU, for CUDA operations, PyTorch employs the **Philox** RNG. This is part of the CUDNN library and is optimized for parallel execution on GPU architectures.

how does the random generator work on the level of physics and raw computation?

- we need two things: source of entropy (messiness, disorder, dynamic) AND an algorithm to further mix up this messiness
    - we can achieve a source of entropy from time itself, if we feed in the time to the last second, we can mash up the emerging numbers and achieve something pseudorandom. a “more” random set of numbers would come from algorithms with more “mixing” steps and a better source of entropy. there are different algorithms researchers have invented. other sources of entropy include physically properties about/around the hardware itself like: temperature, phase noise, clock signals, and more.