"GOT", but the "O" is a cute, smiling pufferfish. Index | Thread | Search

From:
Stefan Sperling <stsp@stsp.name>
Subject:
fix indexing pack files with ref-deltas
To:
gameoftrees@openbsd.org
Date:
Tue, 17 Jan 2023 10:27:37 +0100

Download raw body.

Thread
The problem which prevents gotadmin and gotd from indexing pack files
that contain ref-deltas turns out to be a simple loop accounting error.

$ gotadmin indexpack objects/pack/pack-c5ad9a7a147d1f4846ef41b4e54f11886180ccb6.pack
58.2K packed; indexing 0%got-index-pack: could not resolve any of deltas; packfile could be corrupt
gotadmin: bad pack file
$ 

In the code patched below, we need to add the number of objects resolved
in the current loop iteration to our "valid" counter, not the number of
objects resolved across all iterations so far:

diff /home/stsp/src/got
commit - 4a1a737306fe863c1d6378370d345fae962a2cad
path + /home/stsp/src/got
blob - 8d1f70836b3c957b385dad80e0bc73f6739b9089
file + lib/pack_index.c
--- lib/pack_index.c
+++ lib/pack_index.c
@@ -912,15 +912,15 @@ got_pack_index(struct got_pack *pack, int idxfd, FILE 
 		if (pass++ > 3 && n == 0) {
 			err = got_error_msg(GOT_ERR_BAD_PACKFILE,
 			    "could not resolve any of deltas; packfile could "
 			    "be corrupt");
 			goto done;
 		}
 		nresolved += n;
-		nvalid += nresolved;
+		nvalid += n;
 	}
 
 	if (nloose + nresolved != nobj) {
 		static char msg[64];
 		snprintf(msg, sizeof(msg), "discovered only %d of %d objects",
 		    nloose + nresolved, nobj);
 		err = got_error_msg(GOT_ERR_BAD_PACKFILE, msg);


The head of this while-loop is:

	/*
	 * Second pass: We can now resolve deltas to compute the IDs of
	 * objects which appear in deltified form. Because deltas can be
	 * chained this pass may require a couple of iterations until all
	 * IDs of deltified objects have been discovered.
	 */
	pass++;
	while (nvalid != nobj) {
		int n = 0;


Without the above fix this loop may not terminate as intended because we
step across nvalid == nobj and then keep looping with nvalid > nobj.
The unintended extra loop iterations will fail to resolve any deltas
across several passes and error out with GOT_ERR_BAD_PACKFILE.

With the above fix applied:

$ gotadmin indexpack objects/pack/pack-c5ad9a7a147d1f4846ef41b4e54f11886180ccb6.pack
58.2K packed; indexing 100%; resolving deltas 100%
Indexed c5ad9a7a147d1f4846ef41b4e54f11886180ccb6.pack
$ gotadmin listpack objects/pack/pack-c5ad9a7a147d1f4846ef41b4e54f11886180ccb6.pack  | grep ref-delta
06f04f134cbc5e77589fb4fddde50318cda2a801 ref-delta at 23185 size 705 base-id d3675091aa440b7e2fef27695805e76a6936e399
245ea55e84955297cd3ac3597720dbf411294d89 ref-delta at 35911 size 137 base-id ea5dfbdb5ce9967e191db6bf221a402c59f7e677
28eb0554db5110a58e09358af423a05b465356ea ref-delta at 36250 size 186 base-id 251b79f342189f3350be54a834d39d55d80298e0
2e26139adc84e6a8a83cf67054b965f2e449bcfb ref-delta at 33527 size 1178 base-id 3cfff5dd623d436da9beae2a4db1833bde5bf2ce
37e1eb1a9fd43280cd159c572b1c1fa8dcf556d6 ref-delta at 28032 size 132 base-id eaaa07244d1f70dc4adb4e27222ce015faf960e9
3cfff5dd623d436da9beae2a4db1833bde5bf2ce ref-delta at 36032 size 285 base-id d1db4144a43423c0b0a7c38a7144b2eb4cf8c5e6
7924ea51f8eb63215462a4ff4296b858e320d80e ref-delta at 31705 size 4068 base-id e7ae4a93b5a549b92263ae4e936d81dd92a3963c
c32302a1cd552a87dd849b16fcafbb8d6d62702b ref-delta at 51845 size 168 base-id 97aba15296a87a8df7b0cb2f6e7ad03c54fdaaf6
d1db4144a43423c0b0a7c38a7144b2eb4cf8c5e6 ref-delta at 39338 size 481 base-id a73cc0640fc563d1d7b9fcd6224fef4a435eee2f
d3675091aa440b7e2fef27695805e76a6936e399 ref-delta at 28580 size 3006 base-id ea5dfbdb5ce9967e191db6bf221a402c59f7e677
e7ae4a93b5a549b92263ae4e936d81dd92a3963c ref-delta at 36440 size 1938 base-id c4467d912d87cbaad0b6e22e7009a22719045f44
$