Modified apache-config and robots.txt

The AWS containers were misconfigured so that every robot
imaginable was able to scrape download whatever they wanted.
Pumping the brakes on that and will see what happens.
2 jobs for async-worker in 17 minutes and 40 seconds (queued for 1 second)
Status Job ID Name Coverage
  Test
passed test:3.7 #1397836

00:09:57

73.0%
 
  Branch
passed branch_image #1397837

00:07:42