Allow KafkaCD and KafkaReporter to allow group to be optional and default to a unique group based on tag, job.
I'm going to copy over discussion on a recent merge request talking about implementing this:
I would rather remove the option of a group kwarg in
KafkaReporter
and pass intag
and the job name (evaluate, calibrate, etc) and make thegroup
a unique property based on these (something like tag_job), so that each process gets assigned a unique group. Currently because we read the config from items and not get, we can't get global or general properties like the tag. Do you know of a good way we could do something like this?
Re: generating the group automatically based on tag, it could be as simple as
# somewhere in either batch.py and/or stream.py items = config.parser.evalkwargs(**dict(config.items('evaluate reporting'))) if not items.has_key('group'): items['group'] = names.tag2group(tag, 'evaluate') ... # somewhere in names.py TAG2GROUP_TEMPLATE = '%s_%s' def tag2group(tag, job): return TAG2GROUP_TEMPLATE%(tag, job)
and we could define specific methods for each job that delegate similar to how names.nickname2train_topic et al work. This way, if the user wants to override the default behavior they can. Otherwise, we do something clever automatically.
I'm going to assign this to myself since I can test this easily.