Investigate consistent 5 minute latency peaks
Sometimes during a run these latency spikes appear every 5 minutes almost exactly on every node. It occurring on every node means it's probably system independent and is probably due to some NFS issue.
I've checked the node_exporter stats for one of the nodes that this occurs on and the only system performance graphs that matches these spikes is the memory graphs, but this is probably due to the latency spikes (pipeline is storing extra buffers) and not a cause of said spikes.
I've also watched the processes on a node where these spikes occur and no new processes appear or use extra cpu/memory before any of these spikes appear.