On Thu, Sep 03, 2026 at 01:57:31AM +0330, chara wrote:
> Hi there, i run a public gotweb instance, which has public archives of
> various foss projects i'm interested in, in case of internet shutdowns in my era.
> 
> while got works well with my setup, gotwebd is painfully slow, you can see it in
> https://got.cloud9p.org/
> 
> I'm running OpenBSD 7.9 with latest got, gotd and gotwebd available in the repos.
> any ideas how to improve the performace (at least for the front page, being slow for
> other pages is acceptable, albeit if those can be improved it'd be wonderful)?
> 
> top shows most of the time CPU Is busy running got-read-pack, up to 80-90% of both cores.
> 
> Size of the repository mirrors i host is about 17Gb, which are listed below:

I see similar problems on all my self-hosted servers and on the Game
of Trees Hub hosting service. It's also a known common problem with
public servers running forges backed by stock Git, which perform a
lot better than gotwebd already, but still suffer from too much load.
There is only so much which can be done in terms of optimizing the
software. There will always be a point where too many requests to
serve are simply too many requests to serve.

The types of repositories you are hosting attract web crawler bots from
big tech companies. If you start blocking them by IP range they will
farm requests out to complicit mobile phone apps by middlemen companies,
and the request load comes back within a matter of days from residential
IP network ranges. At which point you won't know whether blocking the IP
will eventually end up blocking a legitimate user.

Their behaviour is ridiculous and abusive. As one example, OpenBSD src.git
was once available on a gotwebd instance which used to be hosted at the
top-level gothub.org domain by patrick@. This domain is now poisoned and
is still being crawled by them, even though for more than a year it has
been hosting a static web site which describes our hosting service instead
of Git repositories.

Log excerpt from this morning:

Sep  3 08:34:47 p0 httpd[68995]: gothub.org 114.119.155.235 - - [03/Sep/2026:08:34:47 +0200] "GET /?action=tree&commit=f0b3f88179a9cb03873c56fec0c8e0de1aa03d4e&folder=regress/libexec/ld.so/dependencies&path=src.git HTTP/1.1" 200 2953
Sep  3 08:36:10 p0 httpd[77912]: gothub.org 114.119.157.237 - - [03/Sep/2026:08:36:10 +0200] "GET /?action=tree&commit=f0b3f88179a9cb03873c56fec0c8e0de1aa03d4e&folder=regress/libexec/ld.so/constructor&path=src.git HTTP/1.1" 200 2953
Sep  3 08:37:34 p0 httpd[68995]: gothub.org 114.119.136.184 - - [03/Sep/2026:08:37:34 +0200] "GET /?action=tree&commit=f0b3f88179a9cb03873c56fec0c8e0de1aa03d4e&folder=regress/lib/libtls/gotls&path=src.git HTTP/1.1" 200 2953
Sep  3 08:38:58 p0 httpd[68995]: gothub.org 114.119.136.184 - - [03/Sep/2026:08:38:58 +0200] "GET /?action=tree&commit=f0b3f88179a9cb03873c56fec0c8e0de1aa03d4e&folder=regress/lib/libssl/unit&path=src.git HTTP/1.1" 200 2953
Sep  3 08:40:21 p0 httpd[77912]: gothub.org 114.119.136.184 - - [03/Sep/2026:08:40:21 +0200] "GET /?action=tree&commit=f0b3f88179a9cb03873c56fec0c8e0de1aa03d4e&folder=regress/lib/libssl/rust-openssl&path=src.git HTTP/1.1" 200 2953
Sep  3 08:41:44 p0 httpd[77912]: gothub.org 114.119.136.184 - - [03/Sep/2026:08:41:44 +0200] "GET /?action=tree&commit=f0b3f88179a9cb03873c56fec0c8e0de1aa03d4e&folder=regress/lib/libssl/interop/version&path=src.git HTTP/1.1" 200 2953
Sep  3 08:43:08 p0 httpd[77912]: gothub.org 114.119.146.185 - - [03/Sep/2026:08:43:08 +0200] "GET /?action=tree&commit=f0b3f88179a9cb03873c56fec0c8e0de1aa03d4e&folder=regress/lib/libssl/dtls&path=src.git HTTP/1.1" 200 2953
Sep  3 08:44:31 p0 httpd[68995]: gothub.org 114.119.155.54 - - [03/Sep/2026:08:44:31 +0200] "GET /?action=tree&commit=f0b3f88179a9cb03873c56fec0c8e0de1aa03d4e&folder=regress/lib/libssl/client&path=src.git HTTP/1.1" 200 2953
Sep  3 08:45:56 p0 httpd[62335]: gothub.org 114.119.167.51 - - [03/Sep/2026:08:45:56 +0200] "GET /?action=tree&commit=f0b3f88179a9cb03873c56fec0c8e0de1aa03d4e&folder=regress/lib/libssl/certs&path=src.git HTTP/1.1" 200 2953
Sep  3 08:47:19 p0 httpd[62335]: gothub.org 114.119.155.235 - - [03/Sep/2026:08:47:19 +0200] "GET /?action=tree&commit=f0b3f88179a9cb03873c56fec0c8e0de1aa03d4e&folder=regress/lib/libssl/bytestring&path=src.git HTTP/1.1" 200 2953
Sep  3 08:48:42 p0 httpd[62335]: gothub.org 114.119.133.12 - - [03/Sep/2026:08:48:42 +0200] "GET /?action=tree&commit=f0b3f88179a9cb03873c56fec0c8e0de1aa03d4e&folder=regress/lib/libssl&path=src.git HTTP/1.1" 200 2953
Sep  3 08:50:06 p0 httpd[77912]: gothub.org 114.119.146.185 - - [03/Sep/2026:08:50:06 +0200] "GET /?action=tree&commit=f0b3f88179a9cb03873c56fec0c8e0de1aa03d4e&folder=regress/lib/libsndio/rec&path=src.git HTTP/1.1" 200 2953
Sep  3 08:51:33 p0 httpd[77912]: gothub.org 114.119.167.51 - - [03/Sep/2026:08:51:33 +0200] "GET /?action=tree&commit=f0b3f88179a9cb03873c56fec0c8e0de1aa03d4e&folder=regress/lib/libsndio/play&path=src.git HTTP/1.1" 200 2953
Sep  3 08:52:53 p0 httpd[77912]: gothub.org 114.119.155.54 - - [03/Sep/2026:08:52:53 +0200] "GET /?action=tree&commit=f0b3f88179a9cb03873c56fec0c8e0de1aa03d4e&folder=regress/lib/libskey&path=src.git HTTP/1.1" 200 2953
Sep  3 08:53:03 p0 httpd[68995]: gothub.org 100.59.132.79 - - [03/Sep/2026:08:53:03 +0200] "GET /?action=blobraw&commit=38de0a39817064d7585df5790627ca9bb36d81da&file=query-pr.html&folder=/pt&path=www.git HTTP/1.1" 200 2953
Sep  3 08:54:17 p0 httpd[68995]: gothub.org 114.119.146.185 - - [03/Sep/2026:08:54:17 +0200] "GET /?action=tree&commit=f0b3f88179a9cb03873c56fec0c8e0de1aa03d4e&folder=regress/lib/libpthread/switch&path=src.git HTTP/1.1" 200 2953
Sep  3 08:55:41 p0 httpd[68995]: gothub.org 114.119.133.108 - - [03/Sep/2026:08:55:41 +0200] "GET /?action=tree&commit=f0b3f88179a9cb03873c56fec0c8e0de1aa03d4e&folder=regress/lib/libpthread/stdfiles&path=src.git HTTP/1.1" 200 2953
Sep  3 08:57:04 p0 httpd[62335]: gothub.org 114.119.155.235 - - [03/Sep/2026:08:57:04 +0200] "GET /?action=tree&commit=f0b3f88179a9cb03873c56fec0c8e0de1aa03d4e&folder=regress/lib/libpthread/stdarg&path=src.git HTTP/1.1" 200 2953
Sep  3 08:58:27 p0 httpd[77912]: gothub.org 114.119.133.108 - - [03/Sep/2026:08:58:27 +0200] "GET /?action=tree&commit=f0b3f88179a9cb03873c56fec0c8e0de1aa03d4e&folder=regress/lib/libpthread/stack&path=src.git HTTP/1.1" 200 2953


Because httpd lacks a feature to filter requests by querystring parameters,
and I am too lazy to write it, all such requests hit the root directory of
the web site get a 200 response with gothub.org/index.html contents.
It's been like this for more than a year. These bots don't care what
they get back. As long as they get something they will always continue.

Once you have this type of load on the public service, you cannot easily
shed it anymore, unless someone upstream from you filters it out for you.

The options at your disposal I'm aware of are:

1) Put something like Anubis or iocaine in front with a reverse proxy.

2) Move the entire service off the public web, into a closed VPN network,
   or into a local public wifi/mesh network, or a public onion-routing
   network, or similar.

3) Enable authentication in gotwebd, and grant read-only access to an
   "anonymous" or similarly named user who can log in without providing
   a password. This is enough to make the bots go away after a while and
   is the only way I'm aware of which will actually remove load.
   Apparently they do not like sites which appear to require authentication,
   and over time the requests will stop.

We use option 3) on the Game of Trees Hub for OpenBSD repositories mirrored
there. We also advise users who run into this issue with their gothub.org
VMs to do the same. So far, this has had the desired effect. If that changes,
I don't see any good options for us, only bad ones which will be making it
harder for legitimate users to access the service.

Regards,
Stefan