Add tracking, unsubscribe, upload, doi, forms to robots.txt

Author: wmnnd

assets/static/robots.txt

@@ -1,5 +1,8 @@
-# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
-#
-# To ban all spiders from the entire site uncomment the next two lines:
-# User-agent: *
-# Disallow: /
+User-agent: *
+Disallow: /c/
+Disallow: /r/
+Disallow: /forms/
+Disallow: /verify-sender/
+Disallow: /double-opt-in/
+Disallow: /unsubscribe/
+Disallow: /uploads/