gunicorn and robots changes

.gitlab-ci.yml

Dockerfile

config/gunicorn_config.py

config/settings/base.py

docker/apache-config

docker/entrypoint

docs/user_docs/source/general.rst

docs/user_docs/source/index.rst

gracedb/core/models.py

...anagement/commands/update_user_accounts_from_ligo_ldap.py

gracedb/static/robots.txt

requirements.txt