`gwpy` breaking legend plots

This script

import matplotlib.pyplot as plt
import bilby

plt.plot([0, 0], [1, 1], label='test')
plt.legend()
plt.savefig('test.png')

Generates this image

if gwpy is installed, whereas if gwpy is not installed, it will generate this.

Note that the legend label is a box when gwpy is installed. For dashed lines etc this looks very confused. I can't figure out exactly, but I think gwpy is setting some rcparam. @duncanmmacleod do you understand this at all?