Channel selection widget's memory usage is staggeringly high
Following up on the crash that I encountered while loading NDS2 channel lists, in cds/packaging/cds-conda-distribution#4
It turns out that the origin of this crash was the linux kernel's out-of-memory killer. In other words, this is not an aarch64-specific issue, but just a side effect of running DTT in a VM with insufficient memory (1 GB was the initial allocation).
However, this made me wonder why DTT burns multiple GB of RAM just to load channel lists. So I ran heaptrack on it, while loading the full channel list from the Caltech NDS server. Here is what was found:
What I take from this analysis:
- By far the biggest consumer of memory is data associated with TLGLBTreeEntry objects, which are constituents of the TLGChannelCombobox widget used by DTT for channel selection.
- The objects themselves consumed 2.9 GB of memory across 21 million allocations. That implies each one uses about 150 bytes (which seems roughly consistent with the number of 8-byte pointers, ints, and other data held by these objects).
- Also, the constructor for these objects allocates two strings, which consumed 0.9 GB across 42 million allocations (average string length about 20 characters).
It looks like DTT recieves a channel list with 1 million+ entries from the Caltech server, and proceeds to allocate one TLGLBTreeEntry per channel, for each of the 16 channel selection widgets visible on the measurement tab. That is how it ends up holding 21 million of these objects.
Since fully populating these widgets up front results in an insane level of memory bloat, perhaps DTT should defer allocating TLGLBTreeEntry objects until the user starts browsing through the hierarchy, so that it can allocate only the visible ones? If a typical user browses through ~20 entries while picking each channel, the resulting memory usage would be 4 kB, rather than 4 GB.