From: Stefan Sperling <stsp@stsp.name>
Subject: Re: process deltas in compressed form
To: Omar Polo <op@omarpolo.com>
Cc: gameoftrees@openbsd.org
Date: Mon, 2 May 2022 14:12:29 +0200

On Mon, May 02, 2022 at 02:02:44PM +0200, Omar Polo wrote:
> i'm happy about hte change, but i remember 'gotadmin pack -a' working on
> a 4GB /tmp partition:
> 
> % df -h | fgrep /tmp
> /dev/sd1d      3.9G    6.8M    3.7G     0%    /tmp
> % /usr/local/bin/gotadmin -V
> gotadmin 0.68
> % time /usr/local/bin/gotadmin pack -a
> 219498 commits colored; 2088729 objects found; 1010361 trees scanned
> packing 5 references; 2088729 objects; deltify: 100%; writing pack: 1.2G 100%
> Wrote fe8e346aa0f23dc77371a3ad76f5bcea7d5ea071.pack
>  1.2G packed; indexing 100%; resolving deltas 100%

For me packing src.git creates a 2.4G pack and that blows up /tmp.
No idea why your pack is that much smaller. Perhaps your pack we are
reusing was created with more aggressive deltification parameters?

> it reads fine, ok op@;  small nitpics inline below.

I'll take those nits into account, thanks.

> slightly related: while reading I've stubled upon csum_output where
> we're potentially truncating a size_t into an 'unsigned int' when
> calling crc32.  It seems unlikely (maybe even impossible) to end up
> calling csum_output with a `len' greater than UINT_MAX, but what about
> using crc32_z?

It would make sense to switch to crc32_z, indeed.

> > +	/* Discard decompressed delta and read it again in compressed form. */
> > +	free(*delta_buf);
> > +	*delta_buf = malloc(*delta_len_compressed);
> > +	if (*delta_buf == NULL)
> > +		return got_error_from_errno("malloc");
> 
> goto done?
> 
> probably not important, I expect callers to check the error before
> looking at the returned data, but the other error conditions don't
> return directly.

Right, I'll fix this. Thanks.