0000241: aws-1.freshports.org - etc: block general public from there - redirect - MantisBT

ID	Project	Category	View Status	Date Submitted	Last Update

0000241	My infrastructure	General	public	2024-07-07 21:50	2024-09-28 11:34

Reporter	dvl	Assigned To	dvl
Priority	normal	Severity	minor	Reproducibility	have not tried
Status	closed	Resolution	fixed

Summary	0000241: aws-1.freshports.org - etc: block general public from there - redirect
Description	Seems to double traffic? Or serve up robots.txt? Yes, let's do something in nginx.conf which gives a different robots.txt for aws-1
Tags	No tags attached.

dvl 2024-08-26 18:17 manager ~0000306	Added redirect permanent, logs seem to be slowing down: [18:15 aws-1 dan ~] % ls -lt /jails/nginx01/var/log/nginx/aws-1.freshports.org-* -rw-r----- 1 www logcheck 2287573 2024.08.26 18:16 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log -rw-r----- 1 www logcheck 0 2024.08.26 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log -rw-r----- 1 www logcheck 0 2024.08.26 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.0 -rw-r----- 1 www logcheck 3030604 2024.08.26 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.0 -rw-r----- 1 www logcheck 0 2024.08.25 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.1 -rw-r----- 1 www logcheck 3772900 2024.08.25 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.1 -rw-r----- 1 www logcheck 3958384 2024.08.24 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.2 -rw-r----- 1 www logcheck 0 2024.08.24 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.2 -rw-r----- 1 www logcheck 0 2024.08.23 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.3 -rw-r----- 1 www logcheck 3406140 2024.08.23 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.3 -rw-r----- 1 www logcheck 0 2024.08.22 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.4 -rw-r----- 1 www logcheck 3003306 2024.08.22 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.4 -rw-r----- 1 www logcheck 3631024 2024.08.21 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.5 -rw-r----- 1 www logcheck 0 2024.08.21 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.5 -rw-r----- 1 www logcheck 0 2024.08.20 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.6 -rw-r----- 1 www logcheck 3808031 2024.08.20 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.6 -rw-r----- 1 www logcheck 0 2024.08.19 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.7 -rw-r----- 1 www logcheck 3889280 2024.08.19 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.7 -rw-r----- 1 www logcheck 0 2024.08.18 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.8 -rw-r----- 1 www logcheck 4054665 2024.08.18 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.8 -rw-r----- 1 www logcheck 4420638 2024.08.17 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.9 -rw-r----- 1 www logcheck 0 2024.08.17 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.9 -rw-r----- 1 www logcheck 0 2024.08.16 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.10 -rw-r----- 1 www logcheck 4490878 2024.08.16 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.10 -rw-r----- 1 www logcheck 4363774 2024.08.15 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.11 -rw-r----- 1 www logcheck 0 2024.08.15 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.11 -rw-r----- 1 www logcheck 6116192 2024.08.14 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.12 -rw-r----- 1 www logcheck 0 2024.08.14 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.12 -rw-r----- 1 www logcheck 0 2024.08.13 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.13 -rw-r----- 1 www logcheck 5197834 2024.08.13 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.13 [18:16 aws-1 dan ~] %

dvl 2024-08-29 21:19 manager ~0000307	server { listen 127.163.0.80:443 ssl; listen [fd80::80]:443 ssl; http2 on; server_name aws-1.freshports.org; # see https://serverfault.com/questions/734816/how-do-i-persuade-nginx-to-redirect-all-requests-except-robots-txt location ^~ /robots.txt { return 200 "User-agent: *\nDisallow: /"; } location ^~ / { return 301 https://www.freshports.org$request_uri; } # we dont' want this here, because it interfers with the location statements # above. # include "/usr/local/etc/freshports/virtualhost-common.conf"; include "/usr/local/etc/freshports/virtualhost-common-ssl.conf"; # this seems to log in addition to what's in virtualhost-common.conf # we get double logs, but that's OK error_log /var/log/nginx/aws-1.freshports.org-error.log; access_log /var/log/nginx/aws-1.freshports.org-access.log combined; ssl_certificate /usr/local/etc/ssl/aws-1.freshports.org.fullchain.cer; ssl_certificate_key /usr/local/etc/ssl/aws-1.freshports.org.key; }

dvl 2024-08-29 21:20 manager ~0000308	Looking better now that the restrictive robots.txt is being returned. [21:19 aws-1 dan ~] % ls -lt /jails/nginx01/var/log/nginx/aws-1.freshports.org-* -rw-r----- 1 www logcheck 501261 2024.08.29 21:19 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log -rw-r----- 1 www logcheck 0 2024.08.29 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log -rw-r----- 1 www logcheck 219 2024.08.29 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.0 -rw-r----- 1 www logcheck 952063 2024.08.29 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.0 -rw-r----- 1 www logcheck 0 2024.08.28 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.1 -rw-r----- 1 www logcheck 2554977 2024.08.28 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.1 -rw-r----- 1 www logcheck 0 2024.08.27 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.2 -rw-r----- 1 www logcheck 2923748 2024.08.27 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.2 -rw-r----- 1 www logcheck 0 2024.08.26 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.3 -rw-r----- 1 www logcheck 3030604 2024.08.26 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.3 -rw-r----- 1 www logcheck 0 2024.08.25 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.4 -rw-r----- 1 www logcheck 3772900 2024.08.25 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.4 -rw-r----- 1 www logcheck 3958384 2024.08.24 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.5 -rw-r----- 1 www logcheck 0 2024.08.24 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.5 -rw-r----- 1 www logcheck 0 2024.08.23 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.6 -rw-r----- 1 www logcheck 3406140 2024.08.23 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.6 -rw-r----- 1 www logcheck 0 2024.08.22 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.7 -rw-r----- 1 www logcheck 3003306 2024.08.22 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.7 -rw-r----- 1 www logcheck 3631024 2024.08.21 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.8 -rw-r----- 1 www logcheck 0 2024.08.21 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.8 -rw-r----- 1 www logcheck 0 2024.08.20 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.9 -rw-r----- 1 www logcheck 3808031 2024.08.20 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.9 -rw-r----- 1 www logcheck 0 2024.08.19 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.10 -rw-r----- 1 www logcheck 3889280 2024.08.19 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.10 -rw-r----- 1 www logcheck 0 2024.08.18 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.11 -rw-r----- 1 www logcheck 4054665 2024.08.18 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.11 -rw-r----- 1 www logcheck 4420638 2024.08.17 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.12 -rw-r----- 1 www logcheck 0 2024.08.17 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.12 -rw-r----- 1 www logcheck 0 2024.08.16 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.13 -rw-r----- 1 www logcheck 4490878 2024.08.16 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.13 [21:19 aws-1 dan ~] %

dvl 2024-09-28 11:34 manager ~0000309	My work here is done: 17558 total [11:33 aws-1 dan ~] % sudo wc -l /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log* 621 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log 1071 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.0 1102 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.1 1701 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.10 1459 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.11 2874 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.12 2131 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.13 976 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.2 564 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.3 440 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.4 518 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.5 631 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.6 585 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.7 1105 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.8 1780 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.9 17558 total

Date Modified	Username	Field	Change
2024-07-07 21:50	dvl	New Issue
2024-07-21 00:53	dvl	Assigned To	=> dvl
2024-07-21 00:53	dvl	Status	new => assigned
2024-08-26 18:17	dvl	Note Added: 0000306
2024-08-26 18:18	dvl	Description Updated
2024-08-29 21:19	dvl	Note Added: 0000307
2024-08-29 21:20	dvl	Note Added: 0000308
2024-09-28 11:34	dvl	Note Added: 0000309
2024-09-28 11:34	dvl	Status	assigned => closed
2024-09-28 11:34	dvl	Resolution	open => fixed