Diskcache server does not consistently scan hot directories.
The hot directories are not being scanned at an even rate. While investigating the diskcache setup on nds-1.ldas.cit we where looking for llhoft (H1, L1, V1, K1) frames in /kafkacache. Watching the logs we would see long stretches where the V1 or K1 directories would get scanned, but the H1 and L1 directories would not get scanned, despite being marked as hot directories (denoted in the logs). What is expected is some pattern like:
H1 K1 L1 V1 H1 K1 L1 V1 ...
Or permutations
H1 K1 L1 V1 V1 L1 H1 K1 L1 H1 V1 K1 ...
All of the directories where receiving frames in steady rates, below the scan period for hot directories.