From: Theo Buehler <tb@theobuehler.org>
Subject: Re: fix support for pack files larger than 2GB
To: gameoftrees@openbsd.org
Date: Fri, 8 Nov 2019 23:50:42 +0200

On Fri, Nov 08, 2019 at 11:29:29PM +0200, Stefan Sperling wrote:
> mpi@ tried to browse a clone of a NetBSD Got repository from github.
> This failed with:  tog: pack index file checksum error
> 
> Our support for pack files larger than 2GB is broken.
> 
> $ du -hs /mnt/netbsd/.git/objects/pack/pack-21b791c37f7f51a9c6bda1daf396e3c0e9068041.*
> 93.5M	/mnt/netbsd/.git/objects/pack/pack-21b791c37f7f51a9c6bda1daf396e3c0e9068041.idx
> 3.6G	/mnt/netbsd/.git/objects/pack/pack-21b791c37f7f51a9c6bda1daf396e3c0e9068041.pack
> 
> The regular offset table in the pack index consists of entries which
> are 32-bit wide. These entries store offsets of objects stored within
> the first 2GB of the pack file verbatim. For objects stored beyond the
> 2 GB range, these entries have the high bit set and specify another index
> into a secondary "large offsets" table, which uses 64-bit wide items to
> represent the actual offsets beyond 2GB.
> 
> There are two problems:
> 
> 1) Our check for size > 2GB is performed on the .idx file's size,
> instead of on the .pack file's size as it should have been.
> 
> 2) The code assumes that the large offset table has the same size as
> the main offset table. But this is only true if all entries in the
> main index table are indices into the secondary table.
> 
> To fix both issues we can count the number of large offsets in the
> main index and size the large offset table accordingly.
> This patch allows me to browse the repository.
> 
> ok?

Makes sense

ok tb