Enable customization of GPU memory use per node
Currently our SPIIR pipeline runs with an expected GPU RAM usage of about 4GB. This constraint has come about because we typically use GPUs on CIT with only 4GB of memory. However, on other computing environments such as OzStar we have GPUs with considerably more memory - but our pipeline still only runs using a maximum of 4GB per GPU per node.
Future development efforts should look into increasing this limit and allow user input to specify the amount of GPU memory utilised by a given pipeline run. This may be as simple as providing more flexibility to change the number of templates/bank per node, although other avenues are definitely possible as well. A successful implementation would allow us to minimise the number of GPU nodes per run, enable maximum GPU efficiency on arbitrary computing environments, and free up resources on our shared supercomputing clusters (e.g. OzStar) for our other members in the OzGrav UWA node.