Twilio SMS improvements
This ticket is intended as an information dump to make an informed decision about Twilio messaging from GraceDB in O4. Unfortunately, the SMS logs in Twilio's console only go back one year. So the fine-grained messaging logs for O3 are gone, but the billing rates are available. GraceDB writes a log message with a "Texting...
" keyword, and those logs are gzipped and archived from O3. So it would be possible-- if need be-- to scrape a month's worth of logs to get the absolute number of messages sent and then extrapolate that to O4. But, you know, effort.
Number of users and types of alerts
This is the result of digging around in the database to get some idea of the number of users and alert types that are live in GraceDB. Note that in the following nomenclature, a "Notification" object contains a unique set of parameters that dictates when and how an alert goes out to a user. Using @chad-hanna as an example (phone number redacted):
> chad=User.objects.get(username='chad.hanna@LIGO.org')
> Notification.objects.filter(user=chad)
<QuerySet [<Notification: chad.hanna@LIGO.ORG: Superevent created or updated & FAR < 1.92901235e-07 -> Call and text +1814XXXXXXX>, <Notification: chad.hanna@LIGO.ORG: Event created or updated & group=CBC & pipeline=gstlal & search=AllSky & FAR < 1.92901235e-07 -> Call and text +1814XXXXXXX>, <Notification: chad.hanna@LIGO.ORG: Event labeled with EM_COINC & any group & any pipeline & any search -> Call and text +1814XXXXXXX>]>
So in this example, he signed up for calls/texts for 1) low-far superevent creation, 2) low-far gstlal event uploads, and 3) event uploads with an EM_COINC label applied. In this scenario, there is one user, but three distinct "Notifications". That being said:
- Number of distinct notifications: 649
- Superevent notifications: 586
- Event notifications: 63
- Number of distinct users: 364
- Distinct users signed up for Event notifications: 60 (includes emails)
- Distinct users with Event call and/or sms notifications: 11
- Number of distinct Event call and/or sms notifications: 16
- Event notifications/user: 16/11=1.5
- Distinct users signed up for Superevent notifications: 345 (includes emails)
- Distinct users with Superevent call and/or sms notifications: 241
- Number of distinct Superevent call and/or sms notifications: 413
- Superevent notifications/user: 413/241=1.7
- Distinct users signed up for Event notifications: 60 (includes emails)
- Number of unique SMS notifications: 304
- Number of unique phone call notifications: 30
- Number of call+text notifications: 95
I'll dump more stats in here later on, but that should be a start. I think a first cut should be take the unique number of users for superevent/event notifications and scale that up by how much the collaboration has grown between O3--> O4. This assumes a constant percentage of collaboration members signed up for texts and calls. Then scale that by the expected increase in superevent/event rates in O4, and then use the call/sms per superevent/event per user to find out an expected messaging rate. For now that is left as an exercise for the reader.
Batch processing of calls and SMS alerts
Getting this operation down to a single database query (which is totally doable), and then a single API call to twilio would save loads of time in generating alerts. Via the documentation:
Each new SMS message from Twilio must be sent with a separate REST API request. To initiate messages to a list of recipients, you must make a request for each number to which you would like to send a message. The best way to do this is to build an array of the recipients and iterate through each phone number.
Weak.
Prioritizing recipients
This is also a way to get messages out to the people who need them faster. At the end of O3 it was decided to pare down the number of G-Event recipients to a fixed list of pipeline and followup and control room recipients. I propose to formalize that list, and then have it as a community entity in LIGO's LDAP (similar to how Communities:LVC:GraceDB:GraceDBAdvocates
exists for EM advocates). Then GraceDB will assign a priority to each group (so pipeline experts=2, em advocates=1, everyone else=0), then sort the list of SMS recipients via this priority and then start the messaging loop.
I'm open to other ideas, but this is a start of a strategy for defining twilio account messaging rates and prices.