Using GPUs

To use GPUs:

  1. Make sure your AWS account is subscribed to the EKS-optimized AMI with GPU Support.

  2. You may need to request a limit increase for your desired instance type.

  3. Set instance type to an AWS GPU instance (e.g. g4dn.xlarge) when installing Cortex.

  4. Set the gpu field in the compute configuration for your API. One unit of GPU corresponds to one virtual GPU. Fractional requests are not allowed.


If using processes_per_replica > 1, TensorFlow-based models, and Python Predictor

When using processes_per_replica > 1 with TensorFlow-based models (including Keras) in the Python Predictor, loading the model in separate processes at the same time will throw a CUDA_ERROR_OUT_OF_MEMORY: out of memory error. This is because the first process that loads the model will allocate all of the GPU's memory and leave none to other processes. To prevent this from happening, the per-process GPU memory usage can be limited. There are two methods:

1) Configure the model to allocate only as much memory as it requires, via tf.config.experimental.set_memory_growth():

for gpu in tf.config.list_physical_devices("GPU"):
tf.config.experimental.set_memory_growth(gpu, True)

2) Impose a hard limit on how much memory the model can use, via tf.config.set_logical_device_configuration():

mem_limit_mb = 1024
for gpu in tf.config.list_physical_devices("GPU"):
gpu, [tf.config.LogicalDeviceConfiguration(memory_limit=mem_limit_mb)]

See the TensorFlow GPU guide and this blog post for additional information.