View Issue Details

IDProjectCategoryView StatusLast Update
0000241My infrastructureGeneralpublic2024-09-28 11:34
Reporterdvl Assigned Todvl  
PrioritynormalSeverityminorReproducibilityhave not tried
Status closedResolutionfixed 
Summary0000241: aws-1.freshports.org - etc: block general public from there - redirect
DescriptionSeems to double traffic?

Or serve up robots.txt?

Yes, let's do something in nginx.conf which gives a different robots.txt for aws-1
TagsNo tags attached.

Activities

dvl

2024-08-26 18:17

manager   ~0000306

Added redirect permanent, logs seem to be slowing down:

[18:15 aws-1 dan ~] % ls -lt /jails/nginx01/var/log/nginx/aws-1.freshports.org-*                                
-rw-r-----  1 www logcheck 2287573 2024.08.26 18:16 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log
-rw-r-----  1 www logcheck       0 2024.08.26 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log
-rw-r-----  1 www logcheck       0 2024.08.26 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.0
-rw-r-----  1 www logcheck 3030604 2024.08.26 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.0
-rw-r-----  1 www logcheck       0 2024.08.25 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.1
-rw-r-----  1 www logcheck 3772900 2024.08.25 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.1
-rw-r-----  1 www logcheck 3958384 2024.08.24 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.2
-rw-r-----  1 www logcheck       0 2024.08.24 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.2
-rw-r-----  1 www logcheck       0 2024.08.23 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.3
-rw-r-----  1 www logcheck 3406140 2024.08.23 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.3
-rw-r-----  1 www logcheck       0 2024.08.22 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.4
-rw-r-----  1 www logcheck 3003306 2024.08.22 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.4
-rw-r-----  1 www logcheck 3631024 2024.08.21 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.5
-rw-r-----  1 www logcheck       0 2024.08.21 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.5
-rw-r-----  1 www logcheck       0 2024.08.20 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.6
-rw-r-----  1 www logcheck 3808031 2024.08.20 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.6
-rw-r-----  1 www logcheck       0 2024.08.19 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.7
-rw-r-----  1 www logcheck 3889280 2024.08.19 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.7
-rw-r-----  1 www logcheck       0 2024.08.18 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.8
-rw-r-----  1 www logcheck 4054665 2024.08.18 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.8
-rw-r-----  1 www logcheck 4420638 2024.08.17 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.9
-rw-r-----  1 www logcheck       0 2024.08.17 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.9
-rw-r-----  1 www logcheck       0 2024.08.16 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.10
-rw-r-----  1 www logcheck 4490878 2024.08.16 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.10
-rw-r-----  1 www logcheck 4363774 2024.08.15 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.11
-rw-r-----  1 www logcheck       0 2024.08.15 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.11
-rw-r-----  1 www logcheck 6116192 2024.08.14 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.12
-rw-r-----  1 www logcheck       0 2024.08.14 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.12
-rw-r-----  1 www logcheck       0 2024.08.13 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.13
-rw-r-----  1 www logcheck 5197834 2024.08.13 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.13
[18:16 aws-1 dan ~] % 

dvl

2024-08-29 21:19

manager   ~0000307

server {
  listen 127.163.0.80:443 ssl;
  listen [fd80::80]:443   ssl;
  http2 on;

  server_name aws-1.freshports.org;



  # see https://serverfault.com/questions/734816/how-do-i-persuade-nginx-to-redirect-all-requests-except-robots-txt
  location ^~ /robots.txt {
        return 200 "User-agent: *\nDisallow: /";
  }
  location ^~ / {
    return 301 https://www.freshports.org$request_uri;
  }


# we dont' want this here, because it interfers with the location statements
# above.
#  include "/usr/local/etc/freshports/virtualhost-common.conf";
  include "/usr/local/etc/freshports/virtualhost-common-ssl.conf";

  # this seems to log in addition to what's in virtualhost-common.conf
  # we get double logs, but that's OK
  error_log     /var/log/nginx/aws-1.freshports.org-error.log;
  access_log    /var/log/nginx/aws-1.freshports.org-access.log combined;

  ssl_certificate     /usr/local/etc/ssl/aws-1.freshports.org.fullchain.cer;
  ssl_certificate_key /usr/local/etc/ssl/aws-1.freshports.org.key;

}

dvl

2024-08-29 21:20

manager   ~0000308

Looking better now that the restrictive robots.txt is being returned.

[21:19 aws-1 dan ~] % ls -lt /jails/nginx01/var/log/nginx/aws-1.freshports.org-*                              
-rw-r-----  1 www logcheck  501261 2024.08.29 21:19 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log
-rw-r-----  1 www logcheck       0 2024.08.29 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log
-rw-r-----  1 www logcheck     219 2024.08.29 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.0
-rw-r-----  1 www logcheck  952063 2024.08.29 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.0
-rw-r-----  1 www logcheck       0 2024.08.28 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.1
-rw-r-----  1 www logcheck 2554977 2024.08.28 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.1
-rw-r-----  1 www logcheck       0 2024.08.27 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.2
-rw-r-----  1 www logcheck 2923748 2024.08.27 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.2
-rw-r-----  1 www logcheck       0 2024.08.26 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.3
-rw-r-----  1 www logcheck 3030604 2024.08.26 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.3
-rw-r-----  1 www logcheck       0 2024.08.25 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.4
-rw-r-----  1 www logcheck 3772900 2024.08.25 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.4
-rw-r-----  1 www logcheck 3958384 2024.08.24 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.5
-rw-r-----  1 www logcheck       0 2024.08.24 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.5
-rw-r-----  1 www logcheck       0 2024.08.23 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.6
-rw-r-----  1 www logcheck 3406140 2024.08.23 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.6
-rw-r-----  1 www logcheck       0 2024.08.22 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.7
-rw-r-----  1 www logcheck 3003306 2024.08.22 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.7
-rw-r-----  1 www logcheck 3631024 2024.08.21 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.8
-rw-r-----  1 www logcheck       0 2024.08.21 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.8
-rw-r-----  1 www logcheck       0 2024.08.20 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.9
-rw-r-----  1 www logcheck 3808031 2024.08.20 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.9
-rw-r-----  1 www logcheck       0 2024.08.19 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.10
-rw-r-----  1 www logcheck 3889280 2024.08.19 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.10
-rw-r-----  1 www logcheck       0 2024.08.18 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.11
-rw-r-----  1 www logcheck 4054665 2024.08.18 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.11
-rw-r-----  1 www logcheck 4420638 2024.08.17 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.12
-rw-r-----  1 www logcheck       0 2024.08.17 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.12
-rw-r-----  1 www logcheck       0 2024.08.16 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-error.log.13
-rw-r-----  1 www logcheck 4490878 2024.08.16 00:00 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.13
[21:19 aws-1 dan ~] % 

dvl

2024-09-28 11:34

manager   ~0000309

My work here is done:

   17558 total
[11:33 aws-1 dan ~] % sudo wc -l /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log*
     621 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log
    1071 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.0
    1102 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.1
    1701 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.10
    1459 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.11
    2874 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.12
    2131 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.13
     976 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.2
     564 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.3
     440 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.4
     518 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.5
     631 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.6
     585 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.7
    1105 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.8
    1780 /jails/nginx01/var/log/nginx/aws-1.freshports.org-access.log.9
   17558 total

Issue History

Date Modified Username Field Change
2024-07-07 21:50 dvl New Issue
2024-07-21 00:53 dvl Assigned To => dvl
2024-07-21 00:53 dvl Status new => assigned
2024-08-26 18:17 dvl Note Added: 0000306
2024-08-26 18:18 dvl Description Updated
2024-08-29 21:19 dvl Note Added: 0000307
2024-08-29 21:20 dvl Note Added: 0000308
2024-09-28 11:34 dvl Note Added: 0000309
2024-09-28 11:34 dvl Status assigned => closed
2024-09-28 11:34 dvl Resolution open => fixed