From: Stefan Sperling <stsp@stsp.name>
Subject: Re: Poor fetching from gotd
To: Christian Weisgerber <naddy@mips.inka.de>
Cc: gameoftrees@openbsd.org
Date: Sun, 26 Feb 2023 23:09:59 +0100

On Sun, Feb 26, 2023 at 08:13:16PM +0100, Christian Weisgerber wrote:
> Christian Weisgerber:
> 
> > Fetching performs poorly.  It is very slow and appears to always
> > download at least ~3 MB of data.
> 
> $ cd got.git/objects/pack
> $ ls -lrt *.pack | tail -5
> -rw-r--r--  1 naddy  naddy   427612 Feb  6 16:25 pack-e2fdbbeab037003a8c7f9c2d6fba537b654bf820.pack
> -rw-r--r--  1 naddy  naddy   495086 Feb 22 23:49 pack-b8bcfda4d9210d5f36ec7d6afa09ddb7c7e39226.pack
> -rw-r--r--  1 naddy  naddy  3203102 Feb 23 19:03 pack-58058359ebaf239fa80ccf02737a1c0430ccb6b9.pack
> -rw-r--r--  1 naddy  naddy  3230094 Feb 25 17:04 pack-d53d7b1a2932b4f0d7e9b5d33197ef09abc5f2a8.pack
> -rw-r--r--  1 naddy  naddy  3234283 Feb 26 16:10 pack-e568a08d8cc5a396b3e1cb52c9109acaf9ca4026.pack
> 
> I think those last three packs are the fetches since I switched to
> got.gameoftrees.org.
> 
> $ gotadmin ls pack-e568a08d8cc5a396b3e1cb52c9109acaf9ca4026.pack | wc
>     5627   41832  482462
> 
> Well, that's a lot.
> 
> Comparing only the first field, the hash, from gotadmin ls's output
> shows that the packs are largely redundant:
> 
> 47 new hashes from pack-5805835 to pack-d53d7b1.
> 5 new hashes from pack-d53d7b1 to pack-e568a08.
> 
> The thousands of other hashes are the same.  So got(1) now appears
> to be downloading the same stuff over and over again.  That doesn't
> seem right.

Hmm, that is indeed strange and should not be happening.

I tried the following but did not yet manage to reproduce the issue:

got clone -m ssh://anonymous@got.gameoftrees.org/got.git

Rewind the main branch by one commit:
- Note the hash of the parent commit of the tip commit of the main branch.
- Delete the reference 'refs/heads/main', then re-create it pointing at
  the parent commit noted previously.

Then fetched with 'got fetch'.
Which only fetched about 35k, and 8 objects in total:

-rw-r--r--  1 stsp  wheel   821696 Feb 26 23:03 objects/pack/pack-d8fb01fc812ff5174fea122444f1b4c82f1a7982.idx
-rw-r--r--  1 stsp  wheel  8262963 Feb 26 23:02 objects/pack/pack-d8fb01fc812ff5174fea122444f1b4c82f1a7982.pack
-rw-r--r--  1 stsp  wheel     1296 Feb 26 23:03 objects/pack/pack-e9bb92cf0e1a0b13b945c01cf2100ecbcc73cde4.idx
-rw-r--r--  1 stsp  wheel    34508 Feb 26 23:03 objects/pack/pack-e9bb92cf0e1a0b13b945c01cf2100ecbcc73cde4.pack

$ gotadmin ls objects/pack/pack-e9bb92cf0e1a0b13b945c01cf2100ecbcc73cde4.pack | wc -l
       8