To use GPUs:
Make sure your AWS account is subscribed to the EKS-optimized AMI with GPU Support.
You may need to request a limit increase for your desired instance type.
Set instance type to an AWS GPU instance (e.g.
g4dn.xlarge) when installing Cortex.
gpu field in the
compute configuration for your API. One unit of GPU corresponds to one virtual GPU. Fractional requests are not allowed.
processes_per_replica > 1 with TensorFlow-based models (including Keras) in the Python Predictor, loading the model in separate processes at the same time will throw a
CUDA_ERROR_OUT_OF_MEMORY: out of memory error. This is because the first process that loads the model will allocate all of the GPU's memory and leave none to other processes. To prevent this from happening, the per-process GPU memory usage can be limited. There are two methods:
1) Configure the model to allocate only as much memory as it requires, via tf.config.experimental.set_memory_growth():
for gpu in tf.config.list_physical_devices("GPU"):tf.config.experimental.set_memory_growth(gpu, True)
2) Impose a hard limit on how much memory the model can use, via tf.config.set_logical_device_configuration():
mem_limit_mb = 1024for gpu in tf.config.list_physical_devices("GPU"):tf.config.set_logical_device_configuration(gpu, [tf.config.LogicalDeviceConfiguration(memory_limit=mem_limit_mb)])