From: Mark Jamsek <mark@jamsek.com>
Subject: Re: fix indexing pack files with ref-deltas
To: gameoftrees@openbsd.org
Date: Tue, 17 Jan 2023 20:39:23 +1100

On 23-01-17 10:27AM, Stefan Sperling wrote:
> The problem which prevents gotadmin and gotd from indexing pack files
> that contain ref-deltas turns out to be a simple loop accounting error.
> 
> $ gotadmin indexpack objects/pack/pack-c5ad9a7a147d1f4846ef41b4e54f11886180ccb6.pack
> 58.2K packed; indexing 0%got-index-pack: could not resolve any of deltas; packfile could be corrupt
> gotadmin: bad pack file
> $ 
> 
> In the code patched below, we need to add the number of objects resolved
> in the current loop iteration to our "valid" counter, not the number of
> objects resolved across all iterations so far:
> 
> diff /home/stsp/src/got
> commit - 4a1a737306fe863c1d6378370d345fae962a2cad
> path + /home/stsp/src/got
> blob - 8d1f70836b3c957b385dad80e0bc73f6739b9089
> file + lib/pack_index.c
> --- lib/pack_index.c
> +++ lib/pack_index.c
> @@ -912,15 +912,15 @@ got_pack_index(struct got_pack *pack, int idxfd, FILE 
>  		if (pass++ > 3 && n == 0) {
>  			err = got_error_msg(GOT_ERR_BAD_PACKFILE,
>  			    "could not resolve any of deltas; packfile could "
>  			    "be corrupt");
>  			goto done;
>  		}
>  		nresolved += n;
> -		nvalid += nresolved;
> +		nvalid += n;
>  	}
>  
>  	if (nloose + nresolved != nobj) {
>  		static char msg[64];
>  		snprintf(msg, sizeof(msg), "discovered only %d of %d objects",
>  		    nloose + nresolved, nobj);
>  		err = got_error_msg(GOT_ERR_BAD_PACKFILE, msg);
> 
> 
> The head of this while-loop is:
> 
> 	/*
> 	 * Second pass: We can now resolve deltas to compute the IDs of
> 	 * objects which appear in deltified form. Because deltas can be
> 	 * chained this pass may require a couple of iterations until all
> 	 * IDs of deltified objects have been discovered.
> 	 */
> 	pass++;
> 	while (nvalid != nobj) {
> 		int n = 0;
> 
> 
> Without the above fix this loop may not terminate as intended because we
> step across nvalid == nobj and then keep looping with nvalid > nobj.
> The unintended extra loop iterations will fail to resolve any deltas
> across several passes and error out with GOT_ERR_BAD_PACKFILE.
> 
> With the above fix applied:
> 
> $ gotadmin indexpack objects/pack/pack-c5ad9a7a147d1f4846ef41b4e54f11886180ccb6.pack
> 58.2K packed; indexing 100%; resolving deltas 100%
> Indexed c5ad9a7a147d1f4846ef41b4e54f11886180ccb6.pack
> $ gotadmin listpack objects/pack/pack-c5ad9a7a147d1f4846ef41b4e54f11886180ccb6.pack  | grep ref-delta
> 06f04f134cbc5e77589fb4fddde50318cda2a801 ref-delta at 23185 size 705 base-id d3675091aa440b7e2fef27695805e76a6936e399
> 245ea55e84955297cd3ac3597720dbf411294d89 ref-delta at 35911 size 137 base-id ea5dfbdb5ce9967e191db6bf221a402c59f7e677
> 28eb0554db5110a58e09358af423a05b465356ea ref-delta at 36250 size 186 base-id 251b79f342189f3350be54a834d39d55d80298e0
> 2e26139adc84e6a8a83cf67054b965f2e449bcfb ref-delta at 33527 size 1178 base-id 3cfff5dd623d436da9beae2a4db1833bde5bf2ce
> 37e1eb1a9fd43280cd159c572b1c1fa8dcf556d6 ref-delta at 28032 size 132 base-id eaaa07244d1f70dc4adb4e27222ce015faf960e9
> 3cfff5dd623d436da9beae2a4db1833bde5bf2ce ref-delta at 36032 size 285 base-id d1db4144a43423c0b0a7c38a7144b2eb4cf8c5e6
> 7924ea51f8eb63215462a4ff4296b858e320d80e ref-delta at 31705 size 4068 base-id e7ae4a93b5a549b92263ae4e936d81dd92a3963c
> c32302a1cd552a87dd849b16fcafbb8d6d62702b ref-delta at 51845 size 168 base-id 97aba15296a87a8df7b0cb2f6e7ad03c54fdaaf6
> d1db4144a43423c0b0a7c38a7144b2eb4cf8c5e6 ref-delta at 39338 size 481 base-id a73cc0640fc563d1d7b9fcd6224fef4a435eee2f
> d3675091aa440b7e2fef27695805e76a6936e399 ref-delta at 28580 size 3006 base-id ea5dfbdb5ce9967e191db6bf221a402c59f7e677
> e7ae4a93b5a549b92263ae4e936d81dd92a3963c ref-delta at 36440 size 1938 base-id c4467d912d87cbaad0b6e22e7009a22719045f44
> $ 

Very nice! ok

You can tick this off the todo list too

-- 
Mark Jamsek <fnc.bsdbox.org>
GPG: F2FF 13DE 6A06 C471 CA80  E6E2 2930 DC66 86EE CF68