- From:
- Stefan Sperling <stsp@stsp.name>
- Subject:
- Re: creating pack files
- To:
- gameoftrees@openbsd.org
- Date:
- Fri, 4 Jun 2021 14:54:42 +0200
Download raw body.
Thread
creating pack files
Download raw body.
creating pack files
On Fri, Jun 04, 2021 at 01:08:02AM +0200, Stefan Sperling wrote:
> This patch is my current state of 'gotadmin pack' work.
New version with some fixes:
- only pack referenced objects, drop support for packing anything loose
- both including and excluding a ref is no longer an error (will be excluded)
- tweak progress output
- pack tag objects
- ignore refs/got/*
I don't expect detailed code review since this patch is very large.
It would be nice to get some test reports though.
I don't expect many people will have a lot of interest in packing their
repositories. But generating packs is a prerequisite for pushing commits!
Testing this code means you're testing code that will eventually run
every time you push changes.
My current plan is to isolate some changes which can be reviewed in
isolation (such as the deflate fix I've sent out earlier today).
Then I will tidy up the history of my branch and integrate the branch.
diff refs/heads/main refs/heads/packcreate
blob - 920c70117d0c5d2d028f95b8082678537cbf4f45
blob + fa31072bce1bf22d8516963a6d472bf2f45cfea1
--- gotadmin/Makefile
+++ gotadmin/Makefile
@@ -6,8 +6,9 @@ PROG= gotadmin
SRCS= gotadmin.c \
deflate.c delta.c delta_cache.c deltify.c error.c gotconfig.c \
inflate.c lockfile.c object.c object_cache.c object_create.c \
- object_idset.c object_parse.c opentemp.c pack.c \
- path.c privsep.c reference.c repository.c sha1.c
+ object_idset.c object_parse.c opentemp.c pack.c pack_create.c \
+ path.c privsep.c reference.c repository.c repository_admin.c \
+ sha1.c
MAN = ${PROG}.1
CPPFLAGS = -I${.CURDIR}/../include -I${.CURDIR}/../lib
blob - 61c214f5d923cc2ae9198a585bfa364076f830eb
blob + b87613ae1d71ee82a80a2d3c9bdbf4f980c22963
--- gotadmin/gotadmin.1
+++ gotadmin/gotadmin.1
@@ -69,7 +69,119 @@ Use the repository at the specified path.
If not specified, assume the repository is located at or above the current
working directory.
.El
+.It Cm pack Oo Fl a Oc Oo Fl r Ar repository-path Oc Oo Fl x Ar reference Oc Op Ar reference ...
+Generate a new pack file and a corresponding pack file index.
+By default, add any loose objects which are reachable via any references
+to the generated pack file.
+.Pp
+If one or more
+.Ar reference
+arguments is specified, only add objects which are reachable via the specified
+references.
+Each
+.Ar reference
+argument may either specify a specific reference or a reference namespace,
+in which case all references within this namespace will be used.
+.Pp
+.Cm gotadmin pack
+always ignores references in the
+.Pa refs/got/
+namespace, effectively treating such references as if they did not refer
+to any objects.
+.Pp
+The options for
+.Cm gotadmin pack
+are as follows:
+.Bl -tag -width Ds
+.It Fl a
+Add objects to the generated pack file even if they are already packed
+in a different pack file.
+Unless this option is specified, only loose objects will be added.
+.It Fl r Ar repository-path
+Use the repository at the specified path.
+If not specified, assume the repository is located at or above the current
+working directory.
+.It Fl x Ar reference
+Exclude objects reachable via the specified
+.Ar reference
+from the pack file.
+The
+.Ar reference
+argument may either specify a specific reference or a reference namespace,
+in which case all references within this namespace will be excluded.
+The
+.Fl x
+option may be specified multiple times to build a list of references to exclude.
+.Pp
+Exclusion takes precedence over inclusion.
+If a reference appears in both the included and excluded lists, it will
+be excluded.
.El
+.It Cm indexpack Ar packfile-path
+Create a pack index for the pack file at
+.Ar packfile-path .
+.Pp
+A pack index is required for using the corresponding pack file with
+.Xr got 1 .
+Usually, a pack index will be created by commands such as
+.Cm gotadmin pack
+or
+.Cm got fetch
+as part of regular operation.
+The
+.Cm gotadmin indexpack
+command may be used to recover from a corrupt or missing index.
+A given pack file will always yield the same bit-identical index.
+.Pp
+The provided
+.Ar packfile-path
+must be located within the
+.Pa objects/pack/
+directory of the repository and should end in
+.Pa .pack .
+The filename of the corresponding pack index is equivalent, except
+that it ends in
+.Pa .idx .
+.Pp
+.It Cm ix
+Short alias for
+.Cm indexpack .
+.It Cm listpack Oo Fl h Oc Oo Fl s Oc Ar packfile-path
+List the contents of the pack file at
+.Ar packfile-path .
+.Pp
+Each object contained in the pack file will be displayed on a single line.
+The information shown includes the object ID, object type, object offset,
+and object size.
+.Pp
+If a packed object is deltified against another object the delta base
+will be shown as well.
+For offset deltas, the delta base is identified via an offset into the
+pack file.
+For reference deltas, the delta base is identified via an object ID.
+.Pp
+The provided
+.Ar packfile-path
+must be located within the
+.Pa objects/pack/
+directory of the repository and should end in
+.Pa .pack .
+.Pp
+The options for
+.Cm gotadmin listpack
+are as follows:
+.Bl -tag -width Ds
+.It Fl h
+Show object sizes in human-readable form.
+.It Fl s
+Display statistics about the pack file after listing objects.
+This includes the total number of objects stored in the pack file
+and a break-down of the number of objects per object type.
+.El
+.It Cm ls
+Short alias for
+.Cm listpack .
+.El
.Sh EXIT STATUS
.Ex -std gotadmin
.Sh SEE ALSO
@@ -79,3 +191,4 @@ working directory.
.Xr got.conf 5
.Sh AUTHORS
.An Stefan Sperling Aq Mt stsp@openbsd.org
+.An Ori Bernstein Aq Mt ori@openbsd.org
blob - 87e781ed58930ff07563b90ffc8f130eda1d0cfc
blob + 090d85e840f61e0caff5c3956bfd96897183cdc0
--- gotadmin/gotadmin.c
+++ gotadmin/gotadmin.c
@@ -15,11 +15,15 @@
*/
#include <sys/queue.h>
+#include <sys/types.h>
+#include <ctype.h>
#include <getopt.h>
#include <err.h>
#include <errno.h>
#include <locale.h>
+#include <inttypes.h>
+#include <sha1.h>
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
@@ -31,10 +35,11 @@
#include "got_error.h"
#include "got_object.h"
#include "got_reference.h"
+#include "got_cancel.h"
#include "got_repository.h"
+#include "got_repository_admin.h"
#include "got_gotconfig.h"
#include "got_path.h"
-#include "got_cancel.h"
#include "got_privsep.h"
#include "got_opentemp.h"
@@ -57,6 +62,13 @@ catch_sigpipe(int signo)
sigpipe_received = 1;
}
+static const struct got_error *
+check_cancelled(void *arg)
+{
+ if (sigint_received || sigpipe_received)
+ return got_error(GOT_ERR_CANCELLED);
+ return NULL;
+}
struct gotadmin_cmd {
const char *cmd_name;
@@ -67,11 +79,20 @@ struct gotadmin_cmd {
__dead static void usage(int, int);
__dead static void usage_info(void);
+__dead static void usage_pack(void);
+__dead static void usage_indexpack(void);
+__dead static void usage_listpack(void);
static const struct got_error* cmd_info(int, char *[]);
+static const struct got_error* cmd_pack(int, char *[]);
+static const struct got_error* cmd_indexpack(int, char *[]);
+static const struct got_error* cmd_listpack(int, char *[]);
static struct gotadmin_cmd gotadmin_commands[] = {
{ "info", cmd_info, usage_info, "" },
+ { "pack", cmd_pack, usage_pack, "" },
+ { "indexpack", cmd_indexpack, usage_indexpack,"ix" },
+ { "listpack", cmd_listpack, usage_listpack, "ls" },
};
static void
@@ -300,6 +321,572 @@ cmd_info(int argc, char *argv[])
done:
if (repo)
got_repo_close(repo);
+ free(cwd);
+ return error;
+}
+
+__dead static void
+usage_pack(void)
+{
+ fprintf(stderr, "usage: %s pack [-a] [-r repository-path] "
+ "[-x reference] [reference ...]\n",
+ getprogname());
+ exit(1);
+}
+
+struct got_pack_progress_arg {
+ char last_scaled_size[FMT_SCALED_STRSIZE];
+ int last_ncommits;
+ int last_nobj_total;
+ int last_p_deltify;
+ int last_p_written;
+ int last_p_indexed;
+ int last_p_resolved;
+ int verbosity;
+ int printed_something;
+};
+
+static const struct got_error *
+pack_progress(void *arg, off_t packfile_size, int ncommits,
+ int nobj_total, int nobj_deltify, int nobj_written)
+{
+ struct got_pack_progress_arg *a = arg;
+ char scaled_size[FMT_SCALED_STRSIZE];
+ int p_deltify, p_written;
+ int print_searching = 0, print_total = 0;
+ int print_deltify = 0, print_written = 0;
+
+ if (a->verbosity < 0)
+ return NULL;
+
+ if (fmt_scaled(packfile_size, scaled_size) == -1)
+ return got_error_from_errno("fmt_scaled");
+
+ if (a->last_ncommits != ncommits) {
+ print_searching = 1;
+ a->last_ncommits = ncommits;
+ }
+
+ if (a->last_nobj_total != nobj_total) {
+ print_searching = 1;
+ print_total = 1;
+ a->last_nobj_total = nobj_total;
+ }
+
+ if (packfile_size > 0 && (a->last_scaled_size[0] == '\0' ||
+ strcmp(scaled_size, a->last_scaled_size)) != 0) {
+ if (strlcpy(a->last_scaled_size, scaled_size,
+ FMT_SCALED_STRSIZE) >= FMT_SCALED_STRSIZE)
+ return got_error(GOT_ERR_NO_SPACE);
+ }
+
+ if (nobj_deltify > 0 || nobj_written > 0) {
+ if (nobj_deltify > 0) {
+ p_deltify = (nobj_deltify * 100) / nobj_total;
+ if (p_deltify != a->last_p_deltify) {
+ a->last_p_deltify = p_deltify;
+ print_searching = 1;
+ print_total = 1;
+ print_deltify = 1;
+ }
+ }
+ if (nobj_written > 0) {
+ p_written = (nobj_written * 100) / nobj_total;
+ if (p_written != a->last_p_written) {
+ a->last_p_written = p_written;
+ print_searching = 1;
+ print_total = 1;
+ print_deltify = 1;
+ print_written = 1;
+ }
+ }
+ }
+
+ if (print_searching || print_total || print_deltify || print_written)
+ printf("\r");
+ if (print_searching)
+ printf("packing %d reference%s", ncommits,
+ ncommits == 1 ? "" : "s");
+ if (print_total)
+ printf("; %d object%s", nobj_total,
+ nobj_total == 1 ? "" : "s");
+ if (print_deltify)
+ printf("; deltify: %d%%", p_deltify);
+ if (print_written)
+ printf("; writing pack: %*s %d%%", FMT_SCALED_STRSIZE,
+ scaled_size, p_written);
+ if (print_searching || print_total || print_deltify ||
+ print_written) {
+ a->printed_something = 1;
+ fflush(stdout);
+ }
+ return NULL;
+}
+
+static const struct got_error *
+pack_index_progress(void *arg, off_t packfile_size, int nobj_total,
+ int nobj_indexed, int nobj_loose, int nobj_resolved)
+{
+ struct got_pack_progress_arg *a = arg;
+ char scaled_size[FMT_SCALED_STRSIZE];
+ int p_indexed, p_resolved;
+ int print_size = 0, print_indexed = 0, print_resolved = 0;
+
+ if (a->verbosity < 0)
+ return NULL;
+
+ if (packfile_size > 0 || nobj_indexed > 0) {
+ if (fmt_scaled(packfile_size, scaled_size) == 0 &&
+ (a->last_scaled_size[0] == '\0' ||
+ strcmp(scaled_size, a->last_scaled_size)) != 0) {
+ print_size = 1;
+ if (strlcpy(a->last_scaled_size, scaled_size,
+ FMT_SCALED_STRSIZE) >= FMT_SCALED_STRSIZE)
+ return got_error(GOT_ERR_NO_SPACE);
+ }
+ if (nobj_indexed > 0) {
+ p_indexed = (nobj_indexed * 100) / nobj_total;
+ if (p_indexed != a->last_p_indexed) {
+ a->last_p_indexed = p_indexed;
+ print_indexed = 1;
+ print_size = 1;
+ }
+ }
+ if (nobj_resolved > 0) {
+ p_resolved = (nobj_resolved * 100) /
+ (nobj_total - nobj_loose);
+ if (p_resolved != a->last_p_resolved) {
+ a->last_p_resolved = p_resolved;
+ print_resolved = 1;
+ print_indexed = 1;
+ print_size = 1;
+ }
+ }
+
+ }
+ if (print_size || print_indexed || print_resolved)
+ printf("\r");
+ if (print_size)
+ printf("%*s packed", FMT_SCALED_STRSIZE, scaled_size);
+ if (print_indexed)
+ printf("; indexing %d%%", p_indexed);
+ if (print_resolved)
+ printf("; resolving deltas %d%%", p_resolved);
+ if (print_size || print_indexed || print_resolved)
+ fflush(stdout);
+
+ return NULL;
+}
+
+static const struct got_error *
+add_ref(struct got_reflist_entry **new, struct got_reflist_head *refs,
+ const char *refname, struct got_repository *repo)
+{
+ const struct got_error *err;
+ struct got_reference *ref;
+
+ *new = NULL;
+
+ err = got_ref_open(&ref, repo, refname, 0);
+ if (err) {
+ if (err->code != GOT_ERR_NOT_REF)
+ return err;
+
+ /* Treat argument as a reference prefix. */
+ err = got_ref_list(refs, repo, refname,
+ got_ref_cmp_by_name, NULL);
+ } else {
+ err = got_reflist_insert(new, refs, ref, repo,
+ got_ref_cmp_by_name, NULL);
+ if (err || *new == NULL /* duplicate */)
+ got_ref_close(ref);
+ }
+
+ return err;
+}
+
+static const struct got_error *
+cmd_pack(int argc, char *argv[])
+{
+ const struct got_error *error = NULL;
+ char *cwd = NULL, *repo_path = NULL;
+ struct got_repository *repo = NULL;
+ int ch, i, loose_obj_only = 1;
+ struct got_object_id *pack_hash = NULL;
+ char *id_str = NULL;
+ struct got_pack_progress_arg ppa;
+ FILE *packfile = NULL;
+ struct got_pathlist_head exclude_args;
+ struct got_pathlist_entry *pe;
+ struct got_reflist_head exclude_refs;
+ struct got_reflist_head include_refs;
+ struct got_reflist_entry *re, *new;
+
+ TAILQ_INIT(&exclude_args);
+ TAILQ_INIT(&exclude_refs);
+ TAILQ_INIT(&include_refs);
+
+ while ((ch = getopt(argc, argv, "ar:x:")) != -1) {
+ switch (ch) {
+ case 'a':
+ loose_obj_only = 0;
+ break;
+ case 'r':
+ repo_path = realpath(optarg, NULL);
+ if (repo_path == NULL)
+ return got_error_from_errno2("realpath",
+ optarg);
+ got_path_strip_trailing_slashes(repo_path);
+ break;
+ case 'x':
+ got_path_strip_trailing_slashes(optarg);
+ error = got_pathlist_append(&exclude_args,
+ optarg, NULL);
+ if (error)
+ return error;
+ break;
+ default:
+ usage_pack();
+ /* NOTREACHED */
+ }
+ }
+
+ argc -= optind;
+ argv += optind;
+
+#ifndef PROFILE
+ if (pledge("stdio rpath wpath cpath fattr flock proc exec sendfd unveil",
+ NULL) == -1)
+ err(1, "pledge");
+#endif
+ cwd = getcwd(NULL, 0);
+ if (cwd == NULL) {
+ error = got_error_from_errno("getcwd");
+ goto done;
+ }
+
+ error = got_repo_open(&repo, repo_path ? repo_path : cwd, NULL);
+ if (error)
+ goto done;
+
+ error = apply_unveil(got_repo_get_path_git_dir(repo), 1);
+ if (error)
+ goto done;
+
+ TAILQ_FOREACH(pe, &exclude_args, entry) {
+ const char *refname = pe->path;
+ error = add_ref(&new, &exclude_refs, refname, repo);
+ if (error)
+ goto done;
+
+ }
+
+ if (argc == 0) {
+ error = got_ref_list(&include_refs, repo, "",
+ got_ref_cmp_by_name, NULL);
+ if (error)
+ goto done;
+ } else {
+ for (i = 0; i < argc; i++) {
+ const char *refname;
+ got_path_strip_trailing_slashes(argv[i]);
+ refname = argv[i];
+ error = add_ref(&new, &include_refs, refname, repo);
+ if (error)
+ goto done;
+ }
+ }
+
+ /* Ignore references in the refs/got/ namespace. */
+ TAILQ_FOREACH_SAFE(re, &include_refs, entry, new) {
+ const char *refname = got_ref_get_name(re->ref);
+ if (strncmp("refs/got/", refname, 9) != 0)
+ continue;
+ TAILQ_REMOVE(&include_refs, re, entry);
+ got_ref_close(re->ref);
+ free(re);
+ }
+
+ memset(&ppa, 0, sizeof(ppa));
+ ppa.last_scaled_size[0] = '\0';
+ ppa.last_p_indexed = -1;
+ ppa.last_p_resolved = -1;
+
+ error = got_repo_pack_objects(&packfile, &pack_hash,
+ &include_refs, &exclude_refs, repo, loose_obj_only,
+ pack_progress, &ppa, check_cancelled, NULL);
+ if (error) {
+ if (ppa.printed_something)
+ printf("\n");
+ goto done;
+ }
+
+ error = got_object_id_str(&id_str, pack_hash);
+ if (error)
+ goto done;
+ printf("\nWrote %s.pack\n", id_str);
+
+ error = got_repo_index_pack(packfile, pack_hash, repo,
+ pack_index_progress, &ppa, check_cancelled, NULL);
+ if (error)
+ goto done;
+ printf("\nIndexed %s.pack\n", id_str);
+done:
+ got_pathlist_free(&exclude_args);
+ got_ref_list_free(&exclude_refs);
+ got_ref_list_free(&include_refs);
+ free(id_str);
+ free(pack_hash);
free(cwd);
return error;
}
+
+__dead static void
+usage_indexpack(void)
+{
+ fprintf(stderr, "usage: %s indexpack packfile-path\n",
+ getprogname());
+ exit(1);
+}
+
+static const struct got_error *
+cmd_indexpack(int argc, char *argv[])
+{
+ const struct got_error *error = NULL;
+ struct got_repository *repo = NULL;
+ int ch;
+ struct got_object_id *pack_hash = NULL;
+ char *packfile_path = NULL;
+ char *id_str = NULL;
+ struct got_pack_progress_arg ppa;
+ FILE *packfile = NULL;
+
+ while ((ch = getopt(argc, argv, "")) != -1) {
+ switch (ch) {
+ default:
+ usage_indexpack();
+ /* NOTREACHED */
+ }
+ }
+
+ argc -= optind;
+ argv += optind;
+
+ if (argc != 1)
+ usage_indexpack();
+
+ packfile_path = realpath(argv[0], NULL);
+ if (packfile_path == NULL)
+ return got_error_from_errno2("realpath", argv[0]);
+
+#ifndef PROFILE
+ if (pledge("stdio rpath wpath cpath fattr flock proc exec sendfd unveil",
+ NULL) == -1)
+ err(1, "pledge");
+#endif
+
+ error = got_repo_open(&repo, packfile_path, NULL);
+ if (error)
+ goto done;
+
+ error = apply_unveil(got_repo_get_path_git_dir(repo), 1);
+ if (error)
+ goto done;
+
+ memset(&ppa, 0, sizeof(ppa));
+ ppa.last_scaled_size[0] = '\0';
+ ppa.last_p_indexed = -1;
+ ppa.last_p_resolved = -1;
+
+ error = got_repo_find_pack(&packfile, &pack_hash, repo,
+ packfile_path);
+ if (error)
+ goto done;
+
+ error = got_object_id_str(&id_str, pack_hash);
+ if (error)
+ goto done;
+
+ error = got_repo_index_pack(packfile, pack_hash, repo,
+ pack_index_progress, &ppa, check_cancelled, NULL);
+ if (error)
+ goto done;
+ printf("\nIndexed %s.pack\n", id_str);
+done:
+ free(id_str);
+ free(pack_hash);
+ return error;
+}
+
+__dead static void
+usage_listpack(void)
+{
+ fprintf(stderr, "usage: %s listpack [-h] [-s] packfile-path\n",
+ getprogname());
+ exit(1);
+}
+
+struct gotadmin_list_pack_cb_args {
+ int nblobs;
+ int ntrees;
+ int ncommits;
+ int ntags;
+ int noffdeltas;
+ int nrefdeltas;
+ int human_readable;
+};
+
+static const struct got_error *
+list_pack_cb(void *arg, struct got_object_id *id, int type, off_t offset,
+ off_t size, off_t base_offset, struct got_object_id *base_id)
+{
+ const struct got_error *err;
+ struct gotadmin_list_pack_cb_args *a = arg;
+ char *id_str, *delta_str = NULL, *base_id_str = NULL;
+ const char *type_str;
+
+ err = got_object_id_str(&id_str, id);
+ if (err)
+ return err;
+
+ switch (type) {
+ case GOT_OBJ_TYPE_BLOB:
+ type_str = GOT_OBJ_LABEL_BLOB;
+ a->nblobs++;
+ break;
+ case GOT_OBJ_TYPE_TREE:
+ type_str = GOT_OBJ_LABEL_TREE;
+ a->ntrees++;
+ break;
+ case GOT_OBJ_TYPE_COMMIT:
+ type_str = GOT_OBJ_LABEL_COMMIT;
+ a->ncommits++;
+ break;
+ case GOT_OBJ_TYPE_TAG:
+ type_str = GOT_OBJ_LABEL_TAG;
+ a->ntags++;
+ break;
+ case GOT_OBJ_TYPE_OFFSET_DELTA:
+ type_str = "offset-delta";
+ if (asprintf(&delta_str, " base-offset %llu",
+ base_offset) == -1) {
+ err = got_error_from_errno("asprintf");
+ goto done;
+ }
+ a->noffdeltas++;
+ break;
+ case GOT_OBJ_TYPE_REF_DELTA:
+ type_str = "ref-delta";
+ err = got_object_id_str(&base_id_str, base_id);
+ if (err)
+ goto done;
+ if (asprintf(&delta_str, " base-id %s", base_id_str) == -1) {
+ err = got_error_from_errno("asprintf");
+ goto done;
+ }
+ a->nrefdeltas++;
+ break;
+ default:
+ err = got_error(GOT_ERR_OBJ_TYPE);
+ goto done;
+ }
+ if (a->human_readable) {
+ char scaled[FMT_SCALED_STRSIZE];
+ char *s;;
+ if (fmt_scaled(size, scaled) == -1) {
+ err = got_error_from_errno("fmt_scaled");
+ goto done;
+ }
+ s = scaled;
+ while (isspace((unsigned char)*s))
+ s++;
+ printf("%s %s at %llu size %s%s\n", id_str, type_str, offset,
+ s, delta_str ? delta_str : "");
+ } else {
+ printf("%s %s at %llu size %llu%s\n", id_str, type_str, offset,
+ size, delta_str ? delta_str : "");
+ }
+done:
+ free(id_str);
+ free(base_id_str);
+ free(delta_str);
+ return err;
+}
+
+static const struct got_error *
+cmd_listpack(int argc, char *argv[])
+{
+ const struct got_error *error = NULL;
+ struct got_repository *repo = NULL;
+ int ch;
+ struct got_object_id *pack_hash = NULL;
+ char *packfile_path = NULL;
+ char *id_str = NULL;
+ struct gotadmin_list_pack_cb_args lpa;
+ FILE *packfile = NULL;
+ int show_stats = 0, human_readable = 0;
+
+ while ((ch = getopt(argc, argv, "hs")) != -1) {
+ switch (ch) {
+ case 'h':
+ human_readable = 1;
+ break;
+ case 's':
+ show_stats = 1;
+ break;
+ default:
+ usage_listpack();
+ /* NOTREACHED */
+ }
+ }
+
+ argc -= optind;
+ argv += optind;
+
+ if (argc != 1)
+ usage_listpack();
+ packfile_path = realpath(argv[0], NULL);
+ if (packfile_path == NULL)
+ return got_error_from_errno2("realpath", argv[0]);
+
+#ifndef PROFILE
+ if (pledge("stdio rpath wpath flock proc exec sendfd unveil",
+ NULL) == -1)
+ err(1, "pledge");
+#endif
+ error = got_repo_open(&repo, packfile_path, NULL);
+ if (error)
+ goto done;
+
+ error = apply_unveil(got_repo_get_path_git_dir(repo), 1);
+ if (error)
+ goto done;
+
+ error = got_repo_find_pack(&packfile, &pack_hash, repo,
+ packfile_path);
+ if (error)
+ goto done;
+ error = got_object_id_str(&id_str, pack_hash);
+ if (error)
+ goto done;
+
+ memset(&lpa, 0, sizeof(lpa));
+ lpa.human_readable = human_readable;
+ error = got_repo_list_pack(packfile, pack_hash, repo,
+ list_pack_cb, &lpa, check_cancelled, NULL);
+ if (error)
+ goto done;
+ if (show_stats) {
+ printf("objects: %d\n blobs: %d\n trees: %d\n commits: %d\n"
+ " tags: %d\n offset-deltas: %d\n ref-deltas: %d\n",
+ lpa.nblobs + lpa.ntrees + lpa.ncommits + lpa.ntags +
+ lpa.noffdeltas + lpa.nrefdeltas,
+ lpa.nblobs, lpa.ntrees, lpa.ncommits, lpa.ntags,
+ lpa.noffdeltas, lpa.nrefdeltas);
+ }
+done:
+ free(id_str);
+ free(pack_hash);
+ free(packfile_path);
+ return error;
+}
blob - 9244c2c00f8a56e7ef54417632a95c0f7aa36ddd
blob + 851d4ad4128c5bcac0cd5616088fb5063844a83f
--- include/got_error.h
+++ include/got_error.h
@@ -145,6 +145,7 @@
#define GOT_ERR_NO_CONFIG_FILE 128
#define GOT_ERR_BAD_SYMLINK 129
#define GOT_ERR_GIT_REPO_EXT 130
+#define GOT_ERR_CANNOT_PACK 131
static const struct got_error {
int code;
@@ -297,6 +298,7 @@ static const struct got_error {
{ GOT_ERR_BAD_SYMLINK, "symbolic link points outside of paths under "
"version control" },
{ GOT_ERR_GIT_REPO_EXT, "unsupported repository format extension" },
+ { GOT_ERR_CANNOT_PACK, "not enough objects to pack" },
};
/*
blob - e5ebc4d0a23cdad02ce0996c736d469f703ae9f8
blob + 33d4fabb70af7fc3afa0db3fb351ca550a129529
--- include/got_object.h
+++ include/got_object.h
@@ -25,6 +25,7 @@ struct got_commit_object;
struct got_object_qid {
SIMPLEQ_ENTRY(got_object_qid) entry;
struct got_object_id *id;
+ void *data; /* managed by API user */
};
SIMPLEQ_HEAD(got_object_id_queue, got_object_qid);
@@ -34,6 +35,13 @@ const struct got_error *got_object_qid_alloc(struct go
void got_object_qid_free(struct got_object_qid *);
void got_object_id_queue_free(struct got_object_id_queue *);
+/*
+ * Deep-copy elements from ID queue src to ID queue dest. Do not copy any
+ * qid->data pointers! This is the caller's responsibility if needed.
+ */
+const struct got_error *got_object_id_queue_copy(
+ const struct got_object_id_queue *src, struct got_object_id_queue *dest);
+
/* Object types. */
#define GOT_OBJ_TYPE_ANY 0 /* wildcard value at run-time */
#define GOT_OBJ_TYPE_COMMIT 1
blob - e330768de849dc0fa39914378bac64b6f8a09c01
blob + f666e84083ffb2ad05eab02a4fe4cc4a06853221
--- include/got_reference.h
+++ include/got_reference.h
@@ -117,6 +117,20 @@ const struct got_error *got_ref_list(struct got_reflis
/* Free all references on a ref list. */
void got_ref_list_free(struct got_reflist_head *);
+/*
+ * Insert a reference into a reference list.
+ * Return a pointer to the newly allocated list entry in *newp.
+ * If *newp is NULL and no error occured then the specified reference was
+ * already an element of the list. If *newp is not NULL then the reference
+ * was shallow-copied onto the list and should no longer be closed with
+ * got_ref_close(). Instead it will be closed along with other list
+ * elements by got_ref_list_free().
+ */
+const struct got_error *
+got_reflist_insert(struct got_reflist_entry **newp, struct got_reflist_head *refs,
+ struct got_reference *ref, struct got_repository *repo,
+ got_ref_cmp_cb cmp_cb, void *cmp_arg);
+
/* Indicate whether the provided reference is symbolic (points at another
* refernce) or not (points at an object ID). */
int got_ref_is_symbolic(struct got_reference *);
blob - /dev/null
blob + 6c27cd2e90cece2a2487a786135e4cb6b269f1ad (mode 644)
--- /dev/null
+++ include/got_repository_admin.h
@@ -0,0 +1,69 @@
+/*
+ * Copyright (c) 2021 Stefan Sperling <stsp@openbsd.org>
+ *
+ * Permission to use, copy, modify, and distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+ * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ */
+
+/* A callback function which gets invoked with progress information to print. */
+typedef const struct got_error *(*got_pack_progress_cb)(void *arg,
+ off_t packfile_size, int ncommits, int nobj_total, int obj_deltify,
+ int nobj_written);
+
+/*
+ * Attempt to pack objects reachable via 'include_refs' into a new packfile.
+ * If 'excluded_refs' is not an empty list, do not pack any objects
+ * reachable from the listed references.
+ * If loose_obj_only is zero, pack reachable objects even if they are
+ * already packed in another packfile. Otherwise, add only loose
+ * objects to the new pack file.
+ * Return an open file handle for the generated pack file.
+ * Return the SHA1 digest of the resulting pack file in pack_hash which
+ * must freed by the caller when done.
+ */
+const struct got_error *
+got_repo_pack_objects(FILE **packfile, struct got_object_id **pack_hash,
+ struct got_reflist_head *include_refs,
+ struct got_reflist_head *exclude_refs, struct got_repository *repo,
+ int loose_obj_only,
+ got_pack_progress_cb progress_cb, void *progress_arg,
+ got_cancel_cb cancel_cb, void *cancel_arg);
+
+/*
+ * Attempt to open a pack file at the specified path. Return an open
+ * file handle and the expected hash of pack file contents.
+ */
+const struct got_error *
+got_repo_find_pack(FILE **packfile, struct got_object_id **pack_hash,
+ struct got_repository *repo, const char *packfile_path);
+
+/* A callback function which gets invoked with progress information to print. */
+typedef const struct got_error *(*got_pack_index_progress_cb)(void *arg,
+ off_t packfile_size, int nobj_total, int nobj_indexed,
+ int nobj_loose, int nobj_resolved);
+
+/* (Re-)Index the pack file identified by the given hash. */
+const struct got_error *
+got_repo_index_pack(FILE *packfile, struct got_object_id *pack_hash,
+ struct got_repository *repo,
+ got_pack_index_progress_cb progress_cb, void *progress_arg,
+ got_cancel_cb cancel_cb, void *cancel_arg);
+
+typedef const struct got_error *(*got_pack_list_cb)(void *arg,
+ struct got_object_id *id, int type, off_t offset, off_t size,
+ off_t base_offset, struct got_object_id *base_id);
+
+/* List the pack file identified by the given hash. */
+const struct got_error *
+got_repo_list_pack(FILE *packfile, struct got_object_id *pack_hash,
+ struct got_repository *repo, got_pack_list_cb list_cb, void *list_arg,
+ got_cancel_cb cancel_cb, void *cancel_arg);
blob - a9a36a64dd1519966e9aa8699e80bd93c895bdbf
blob + 02f9a9d5925c144aab1275b46ce59112abff7023
--- lib/deflate.c
+++ lib/deflate.c
@@ -35,8 +35,7 @@
#endif
const struct got_error *
-got_deflate_init(struct got_deflate_buf *zb, uint8_t *outbuf, size_t bufsize,
- struct got_deflate_checksum *csum)
+got_deflate_init(struct got_deflate_buf *zb, uint8_t *outbuf, size_t bufsize)
{
const struct got_error *err = NULL;
int zerr;
@@ -74,8 +73,6 @@ got_deflate_init(struct got_deflate_buf *zb, uint8_t *
zb->flags |= GOT_DEFLATE_F_OWN_OUTBUF;
} else
zb->outbuf = outbuf;
-
- zb->csum = csum;
done:
if (err)
got_deflate_end(zb);
@@ -104,9 +101,6 @@ got_deflate_read(struct got_deflate_buf *zb, FILE *f,
*outlenp = 0;
do {
- char *csum_out = NULL;
- size_t csum_avail = 0;
-
if (z->avail_in == 0) {
size_t n = fread(zb->inbuf, 1, zb->inlen, f);
if (n == 0) {
@@ -119,15 +113,7 @@ got_deflate_read(struct got_deflate_buf *zb, FILE *f,
z->next_in = zb->inbuf;
z->avail_in = n;
}
- if (zb->csum) {
- csum_out = z->next_out;
- csum_avail = z->avail_out;
- }
ret = deflate(z, Z_NO_FLUSH);
- if (zb->csum) {
- csum_output(zb->csum, csum_out,
- csum_avail - z->avail_out);
- }
} while (ret == Z_OK && z->avail_out > 0);
if (ret == Z_OK) {
@@ -159,7 +145,7 @@ got_deflate_to_file(size_t *outlen, FILE *infile, FILE
size_t avail;
struct got_deflate_buf zb;
- err = got_deflate_init(&zb, NULL, GOT_DEFLATE_BUFSIZE, csum);
+ err = got_deflate_init(&zb, NULL, GOT_DEFLATE_BUFSIZE);
if (err)
goto done;
@@ -176,6 +162,8 @@ got_deflate_to_file(size_t *outlen, FILE *infile, FILE
err = got_ferror(outfile, GOT_ERR_IO);
goto done;
}
+ if (csum)
+ csum_output(csum, zb.outbuf, avail);
*outlen += avail;
}
} while (zb.flags & GOT_DEFLATE_F_HAVE_MORE);
blob - 0b023fb4330314246e6a110ad5aade625f8b9a0f
blob + 253ce69e255c039bca39d1cd3df3bc26c8661b7c
--- lib/deltify.c
+++ lib/deltify.c
@@ -96,10 +96,11 @@ hashblk(const unsigned char *p, off_t n)
}
static const struct got_error *
-addblk(struct got_delta_table *dt, FILE *f, off_t len, off_t offset, uint64_t h)
+addblk(struct got_delta_table *dt, FILE *f, off_t file_offset0, off_t len,
+ off_t offset, uint64_t h)
{
const struct got_error *err = NULL;
- int i, nalloc;
+ int i;
uint8_t buf[GOT_DELTIFY_MAXCHUNK];
size_t r = 0;
@@ -109,7 +110,7 @@ addblk(struct got_delta_table *dt, FILE *f, off_t len,
i = h % dt->nalloc;
while (dt->blocks[i].len != 0) {
if (r == 0) {
- if (fseeko(f, offset, SEEK_SET) == -1)
+ if (fseeko(f, file_offset0 + offset, SEEK_SET) == -1)
return got_error_from_errno("fseeko");
r = fread(buf, 1, len, f);
if (r != len) {
@@ -125,7 +126,8 @@ addblk(struct got_delta_table *dt, FILE *f, off_t len,
*/
if (len == dt->blocks[i].len) {
uint8_t buf2[GOT_DELTIFY_MAXCHUNK];
- if (fseeko(f, dt->blocks[i].offset, SEEK_SET) == -1)
+ if (fseeko(f, file_offset0 + dt->blocks[i].offset,
+ SEEK_SET) == -1)
return got_error_from_errno("fseeko");
if (fread(buf2, 1, len, f) != len)
return got_ferror(f, GOT_ERR_IO);
@@ -140,27 +142,27 @@ addblk(struct got_delta_table *dt, FILE *f, off_t len,
dt->blocks[i].offset = offset;
dt->blocks[i].hash = h;
dt->nblocks++;
- if (dt->nalloc < 2 * dt->nblocks) {
+ if (dt->nalloc < dt->nblocks + 64) {
struct got_delta_block *db;
- nalloc = dt->nalloc * 2;
+ size_t old_size = dt->nalloc;
db = dt->blocks;
- dt->blocks = calloc(nalloc, sizeof(struct got_delta_block));
+ dt->blocks = calloc(dt->nblocks + 64, sizeof(struct got_delta_block));
if (dt->blocks == NULL) {
err = got_error_from_errno("calloc");
dt->blocks = db;
return err;
}
- dt->nalloc = nalloc;
+ dt->nalloc = dt->nblocks + 64;
/*
* Recompute all block positions. Hash-based indices of blocks
* in the array depend on the allocated length of the array.
*/
dt->nblocks = 0;
- for (i = 0; i < nalloc; i++) {
+ for (i = 0; i < old_size; i++) {
if (db[i].len == 0)
continue;
- err = addblk(dt, f, db[i].len, db[i].offset,
- db[i].hash);
+ err = addblk(dt, f, file_offset0, db[i].len,
+ db[i].offset, db[i].hash);
if (err)
break;
}
@@ -172,7 +174,7 @@ addblk(struct got_delta_table *dt, FILE *f, off_t len,
static const struct got_error *
lookupblk(struct got_delta_block **block, struct got_delta_table *dt,
- unsigned char *p, off_t len, FILE *basefile)
+ unsigned char *p, off_t len, FILE *basefile, off_t basefile_offset0)
{
int i;
uint64_t h;
@@ -187,7 +189,8 @@ lookupblk(struct got_delta_block **block, struct got_d
if (dt->blocks[i].hash != h ||
dt->blocks[i].len != len)
continue;
- if (fseeko(basefile, dt->blocks[i].offset, SEEK_SET) == -1)
+ if (fseeko(basefile, basefile_offset0 + dt->blocks[i].offset,
+ SEEK_SET) == -1)
return got_error_from_errno("fseeko");
r = fread(buf, 1, len, basefile);
if (r != len)
@@ -238,6 +241,7 @@ got_deltify_init(struct got_delta_table **dt, FILE *f,
{
const struct got_error *err = NULL;
uint64_t h;
+ const off_t offset0 = fileoffset;
*dt = calloc(1, sizeof(**dt));
if (*dt == NULL)
@@ -263,10 +267,13 @@ got_deltify_init(struct got_delta_table **dt, FILE *f,
if (blocklen == 0)
break;
h = hashblk(buf, blocklen);
- err = addblk(*dt, f, blocklen, fileoffset, h);
+ err = addblk(*dt, f, offset0, blocklen,
+ fileoffset - offset0, h);
if (err)
goto done;
fileoffset += blocklen;
+ if (fseeko(f, fileoffset, SEEK_SET) == -1)
+ return got_error_from_errno("fseeko");
}
done:
if (err) {
@@ -307,16 +314,14 @@ emitdelta(struct got_delta_instruction **deltas, int *
}
static const struct got_error *
-stretchblk(FILE *basefile, struct got_delta_block *block, FILE *f,
- off_t filesize, off_t *blocklen)
+stretchblk(FILE *basefile, off_t base_offset0, struct got_delta_block *block,
+ FILE *f, off_t filesize, off_t *blocklen)
{
uint8_t basebuf[GOT_DELTIFY_MAXCHUNK], buf[GOT_DELTIFY_MAXCHUNK];
size_t base_r, r, i;
- off_t orig_blocklen = *blocklen;
- off_t pos = ftello(f);
int buf_equal = 1;
- if (fseeko(basefile, block->offset, SEEK_SET) == -1)
+ if (fseeko(basefile, base_offset0 + block->offset, SEEK_SET) == -1)
return got_error_from_errno("fseeko");
while (buf_equal && *blocklen < (1 << 24) - 1) {
@@ -341,9 +346,6 @@ stretchblk(FILE *basefile, struct got_delta_block *blo
}
}
- if (fseeko(f, pos + *blocklen - orig_blocklen, SEEK_SET) == -1)
- return got_error_from_errno("fseeko");
-
return NULL;
}
@@ -351,7 +353,7 @@ const struct got_error *
got_deltify(struct got_delta_instruction **deltas, int *ndeltas,
FILE *f, off_t fileoffset, off_t filesize,
struct got_delta_table *dt, FILE *basefile,
- off_t basefile_size)
+ off_t basefile_offset0, off_t basefile_size)
{
const struct got_error *err = NULL;
const off_t offset0 = fileoffset;
@@ -374,9 +376,17 @@ got_deltify(struct got_delta_instruction **deltas, int
err = nextblk(buf, &blocklen, f);
if (err)
break;
- if (blocklen == 0)
+ if (blocklen == 0) {
+ /* Source remainder from the file itself. */
+ if (fileoffset < filesize) {
+ err = emitdelta(deltas, ndeltas, 0,
+ fileoffset - offset0,
+ filesize - fileoffset);
+ }
break;
- err = lookupblk(&block, dt, buf, blocklen, basefile);
+ }
+ err = lookupblk(&block, dt, buf, blocklen, basefile,
+ basefile_offset0);
if (err)
break;
if (block != NULL) {
@@ -385,20 +395,27 @@ got_deltify(struct got_delta_instruction **deltas, int
* Attempt to stretch the block as far as possible and
* generate a copy instruction.
*/
- err = stretchblk(basefile, block, f, filesize,
- &blocklen);
+ err = stretchblk(basefile, basefile_offset0, block,
+ f, filesize, &blocklen);
if (err)
break;
- emitdelta(deltas, ndeltas, 1, block->offset, blocklen);
+ err = emitdelta(deltas, ndeltas, 1, block->offset, blocklen);
+ if (err)
+ break;
} else {
/*
* No match.
* This block needs to be sourced from the file itself.
*/
- emitdelta(deltas, ndeltas, 0, fileoffset - offset0,
+ err = emitdelta(deltas, ndeltas, 0, fileoffset - offset0,
blocklen);
+ if (err)
+ break;
}
fileoffset += blocklen;
+ if (fseeko(f, fileoffset, SEEK_SET) == -1)
+ return got_error_from_errno("fseeko");
+
}
if (err) {
blob - c359d7c4cfbe99e2e20a5f7798a73ce6a55257c3
blob + ed2025b8a80eacda4d360ce37c03efd2151ffa1b
--- lib/got_lib_deflate.h
+++ lib/got_lib_deflate.h
@@ -37,7 +37,7 @@ struct got_deflate_buf {
#define GOT_DEFLATE_BUFSIZE 8192
const struct got_error *got_deflate_init(struct got_deflate_buf *, uint8_t *,
- size_t, struct got_deflate_checksum *);
+ size_t);
const struct got_error *got_deflate_read(struct got_deflate_buf *, FILE *,
size_t *);
void got_deflate_end(struct got_deflate_buf *);
blob - 39069a604eca508347a2f734e6fe70185a3febea
blob + 848c59979d94aa7c6bc1d69f81419df67855c141
--- lib/got_lib_deltify.h
+++ lib/got_lib_deltify.h
@@ -44,5 +44,6 @@ const struct got_error *got_deltify_init(struct got_de
off_t fileoffset, off_t filesize);
const struct got_error *got_deltify(struct got_delta_instruction **deltas,
int *ndeltas, FILE *f, off_t fileoffset, off_t filesize,
- struct got_delta_table *dt, FILE *basefile, off_t basefile_size);
+ struct got_delta_table *dt, FILE *basefile, off_t basefile_offset0,
+ off_t basefile_size);
void got_deltify_free(struct got_delta_table *dt);
blob - 8ba2e5dff14da3efda11c6efbcaa5ba9f743c2b3
blob + 958c9a92257c3689d9d9cbd294761020330cb183
--- lib/got_lib_pack.h
+++ lib/got_lib_pack.h
@@ -24,11 +24,15 @@ struct got_pack {
struct got_delta_cache *delta_cache;
};
+struct got_packidx;
+
const struct got_error *got_pack_stop_privsep_child(struct got_pack *);
const struct got_error *got_pack_close(struct got_pack *);
const struct got_error *got_pack_parse_offset_delta(off_t *, size_t *,
struct got_pack *, off_t, int);
+const struct got_error *got_pack_parse_ref_delta(struct got_object_id *,
+ struct got_pack *, off_t, int);
const struct got_error *got_pack_resolve_delta_chain(struct got_delta_chain *,
struct got_packidx *, struct got_pack *, off_t, size_t, int, size_t,
unsigned int);
@@ -171,6 +175,7 @@ const struct got_error *got_packidx_open(struct got_pa
int, const char *, int);
const struct got_error *got_packidx_close(struct got_packidx *);
const struct got_error *got_packidx_get_packfile_path(char **, struct got_packidx *);
+off_t got_packidx_get_object_offset(struct got_packidx *, int idx);
int got_packidx_get_object_idx(struct got_packidx *, struct got_object_id *);
const struct got_error *got_packidx_match_id_str_prefix(
struct got_object_id_queue *, struct got_packidx *, const char *);
blob - /dev/null
blob + 180adaeec95923b909b3ce9e754eb670b166792d (mode 644)
--- /dev/null
+++ lib/got_lib_pack_create.h
@@ -0,0 +1,29 @@
+/*
+ * Copyright (c) 2021 Stefan Sperling <stsp@openbsd.org>
+ *
+ * Permission to use, copy, modify, and distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+ * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ */
+
+/*
+ * Write pack file data into the provided open packfile handle, for all
+ * objects reachable via the commits listed in 'ours'.
+ * Exclude any objects for commits listed in 'theirs' if 'theirs' is not NULL.
+ * Return the SHA1 digest of the resulting pack file in pack_sha1 which must
+ * be pre-allocated by the caller with at least SHA1_DIGEST_LENGTH bytes.
+ */
+const struct got_error *got_pack_create(uint8_t *pack_sha1, FILE *packfile,
+ struct got_object_id **theirs, int ntheirs,
+ struct got_object_id **ours, int nours,
+ struct got_repository *repo, int loose_obj_only,
+ got_pack_progress_cb progress_cb, void *progress_arg,
+ got_cancel_cb cancel_cb, void *cancel_arg);
blob - d599d8a2df794fac0197af296081abaca81c9eb6
blob + a207712ce41df5a11e137bb3aea65159c78ba5f6
--- lib/object.c
+++ lib/object.c
@@ -569,11 +569,6 @@ got_object_raw_open(struct got_raw_object **obj, struc
repo, fd);
}
- if (hdrlen > size) {
- err = got_error(GOT_ERR_BAD_OBJ_HDR);
- goto done;
- }
-
*obj = calloc(1, sizeof(**obj));
if (*obj == NULL) {
err = got_error_from_errno("calloc");
@@ -592,7 +587,7 @@ got_object_raw_open(struct got_raw_object **obj, struc
goto done;
}
outfd = -1;
- (*obj)->f = fmemopen(outbuf, hdrlen + size, "r");
+ (*obj)->f = fmemopen(outbuf, size + hdrlen, "r");
if ((*obj)->f == NULL) {
err = got_error_from_errno("fdopen");
goto done;
@@ -605,7 +600,7 @@ got_object_raw_open(struct got_raw_object **obj, struc
goto done;
}
- if (sb.st_size != size) {
+ if (sb.st_size != size + hdrlen) {
err = got_error(GOT_ERR_PRIVSEP_LEN);
goto done;
}
@@ -904,6 +899,30 @@ got_object_qid_alloc(struct got_object_qid **qid, stru
return NULL;
}
+const struct got_error *
+got_object_id_queue_copy(const struct got_object_id_queue *src,
+ struct got_object_id_queue *dest)
+{
+ const struct got_error *err;
+ struct got_object_qid *qid;
+
+ SIMPLEQ_FOREACH(qid, src, entry) {
+ struct got_object_qid *new;
+ /*
+ * Deep-copy the object ID only. Let the caller deal
+ * with setting up the new->data pointer if needed.
+ */
+ err = got_object_qid_alloc(&new, qid->id);
+ if (err) {
+ got_object_id_queue_free(dest);
+ return err;
+ }
+ SIMPLEQ_INSERT_TAIL(dest, new, entry);
+ }
+
+ return NULL;
+}
+
static const struct got_error *
request_packed_tree(struct got_tree_object **tree, struct got_pack *pack,
int pack_idx, struct got_object_id *id)
blob - 16b8d2d529e9ab5b19b529a8d06b9f77dc22c9b1
blob + 0b0c5c0789a60b52fbd0e7f18073f05dbf59c989
--- lib/object_parse.c
+++ lib/object_parse.c
@@ -89,6 +89,7 @@ got_object_qid_alloc_partial(struct got_object_qid **q
*qid = NULL;
return err;
}
+ (*qid)->data = NULL;
return NULL;
}
blob - 137217342b5f0eb662a63182136a6750ffa7e2af
blob + a2d46242d042d8430a350e840350199d27a6eedf
--- lib/pack.c
+++ lib/pack.c
@@ -440,8 +440,8 @@ got_packidx_get_packfile_path(char **path_packfile, st
return NULL;
}
-static off_t
-get_object_offset(struct got_packidx *packidx, int idx)
+off_t
+got_packidx_get_object_offset(struct got_packidx *packidx, int idx)
{
uint32_t offset = be32toh(packidx->hdr.offsets[idx]);
if (offset & GOT_PACKIDX_OFFSET_VAL_IS_LARGE_IDX) {
@@ -807,6 +807,25 @@ resolve_offset_delta(struct got_delta_chain *deltas,
base_tslen, base_type, base_size, recursion - 1);
}
+const struct got_error *
+got_pack_parse_ref_delta(struct got_object_id *id,
+ struct got_pack *pack, off_t delta_offset, int tslen)
+{
+ if (pack->map) {
+ size_t mapoff = delta_offset + tslen;
+ memcpy(id, pack->map + mapoff, sizeof(*id));
+ } else {
+ ssize_t n;
+ n = read(pack->fd, id, sizeof(*id));
+ if (n < 0)
+ return got_error_from_errno("read");
+ if (n != sizeof(*id))
+ return got_error(GOT_ERR_BAD_PACKFILE);
+ }
+
+ return NULL;
+}
+
static const struct got_error *
resolve_ref_delta(struct got_delta_chain *deltas, struct got_packidx *packidx,
struct got_pack *pack, off_t delta_offset, size_t tslen, int delta_type,
@@ -824,18 +843,12 @@ resolve_ref_delta(struct got_delta_chain *deltas, stru
if (delta_offset + tslen >= pack->filesize)
return got_error(GOT_ERR_PACK_OFFSET);
+ err = got_pack_parse_ref_delta(&id, pack, delta_offset, tslen);
+ if (err)
+ return err;
if (pack->map) {
- size_t mapoff = delta_offset + tslen;
- memcpy(&id, pack->map + mapoff, sizeof(id));
- mapoff += sizeof(id);
- delta_data_offset = (off_t)mapoff;
+ delta_data_offset = delta_offset + tslen + sizeof(id);
} else {
- ssize_t n;
- n = read(pack->fd, &id, sizeof(id));
- if (n < 0)
- return got_error_from_errno("read");
- if (n != sizeof(id))
- return got_error(GOT_ERR_BAD_PACKFILE);
delta_data_offset = lseek(pack->fd, 0, SEEK_CUR);
if (delta_data_offset == -1)
return got_error_from_errno("lseek");
@@ -851,7 +864,7 @@ resolve_ref_delta(struct got_delta_chain *deltas, stru
if (idx == -1)
return got_error(GOT_ERR_NO_OBJ);
- base_offset = get_object_offset(packidx, idx);
+ base_offset = got_packidx_get_object_offset(packidx, idx);
if (base_offset == (uint64_t)-1)
return got_error(GOT_ERR_BAD_PACKIDX);
@@ -954,7 +967,7 @@ got_packfile_open_object(struct got_object **obj, stru
*obj = NULL;
- offset = get_object_offset(packidx, idx);
+ offset = got_packidx_get_object_offset(packidx, idx);
if (offset == (uint64_t)-1)
return got_error(GOT_ERR_BAD_PACKIDX);
blob - 61891c554c130e0e597d44f689d9d93fec69b64d
blob + ebbe86bd40a3407e2166b0cfbe3a8490c353f30c
--- lib/privsep.c
+++ lib/privsep.c
@@ -296,8 +296,8 @@ got_privsep_send_raw_obj(struct imsgbuf *ibuf, off_t s
iobj.hdrlen = hdrlen;
iobj.size = size;
- if (data && size <= GOT_PRIVSEP_INLINE_OBJECT_DATA_MAX)
- len += (size_t)size;
+ if (data && size + hdrlen <= GOT_PRIVSEP_INLINE_OBJECT_DATA_MAX)
+ len += (size_t)size + hdrlen;
wbuf = imsg_create(ibuf, GOT_IMSG_RAW_OBJECT, 0, 0, len);
if (wbuf == NULL) {
@@ -311,8 +311,8 @@ got_privsep_send_raw_obj(struct imsgbuf *ibuf, off_t s
return err;
}
- if (data && size <= GOT_PRIVSEP_INLINE_OBJECT_DATA_MAX) {
- if (imsg_add(wbuf, data, size) == -1) {
+ if (data && size + hdrlen <= GOT_PRIVSEP_INLINE_OBJECT_DATA_MAX) {
+ if (imsg_add(wbuf, data, size + hdrlen) == -1) {
err = got_error_from_errno("imsg_add RAW_OBJECT");
ibuf_free(wbuf);
return err;
@@ -357,17 +357,17 @@ got_privsep_recv_raw_obj(uint8_t **outbuf, off_t *size
break;
}
- if (*size > GOT_PRIVSEP_INLINE_OBJECT_DATA_MAX) {
+ if (*size + *hdrlen > GOT_PRIVSEP_INLINE_OBJECT_DATA_MAX) {
err = got_error(GOT_ERR_PRIVSEP_LEN);
break;
}
- *outbuf = malloc(*size);
+ *outbuf = malloc(*size + *hdrlen);
if (*outbuf == NULL) {
err = got_error_from_errno("malloc");
break;
}
- memcpy(*outbuf, imsg.data + sizeof(*iobj), *size);
+ memcpy(*outbuf, imsg.data + sizeof(*iobj), *size + *hdrlen);
break;
default:
err = got_error(GOT_ERR_PRIVSEP_MSG);
blob - /dev/null
blob + 687269a4a6ab8472e6491b4ac5485ba24c54934c (mode 644)
--- /dev/null
+++ lib/pack_create.c
@@ -0,0 +1,1279 @@
+/*
+ * Copyright (c) 2020 Ori Bernstein
+ * Copyright (c) 2021 Stefan Sperling <stsp@openbsd.org>
+ *
+ * Permission to use, copy, modify, and distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+ * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ */
+
+#include <sys/queue.h>
+#include <sys/stat.h>
+
+#include <inttypes.h>
+#include <imsg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sha1.h>
+#include <limits.h>
+#include <zlib.h>
+
+#include "got_error.h"
+#include "got_cancel.h"
+#include "got_object.h"
+#include "got_reference.h"
+#include "got_repository_admin.h"
+
+#include "got_lib_deltify.h"
+#include "got_lib_delta.h"
+#include "got_lib_object.h"
+#include "got_lib_object_idset.h"
+#include "got_lib_object_cache.h"
+#include "got_lib_deflate.h"
+#include "got_lib_pack.h"
+#include "got_lib_privsep.h"
+#include "got_lib_repository.h"
+
+#ifndef MAX
+#define MAX(_a,_b) ((_a) > (_b) ? (_a) : (_b))
+#endif
+
+struct got_pack_meta {
+ struct got_object_id id;
+ char *path;
+ int obj_type;
+ time_t mtime;
+
+ /* The best delta we picked */
+ struct got_pack_meta *head;
+ struct got_pack_meta *prev;
+ struct got_delta_instruction *deltas;
+ int ndeltas;
+ int nchain;
+
+ /* Only used for delta window */
+ struct got_delta_table *dtab;
+
+ /* Only used for writing offset deltas */
+ off_t off;
+};
+
+struct got_pack_metavec {
+ struct got_pack_meta **meta;
+ int nmeta;
+ int metasz;
+};
+
+static const struct got_error *
+alloc_meta(struct got_pack_meta **new, struct got_object_id *id,
+ const char *path, int obj_type, time_t mtime)
+{
+ const struct got_error *err = NULL;
+ struct got_pack_meta *m;
+
+ *new = NULL;
+
+ m = calloc(1, sizeof(*m));
+ if (m == NULL)
+ return got_error_from_errno("malloc");
+
+ memcpy(&m->id, id, sizeof(m->id));
+
+ m->path = strdup(path);
+ if (m->path == NULL) {
+ err = got_error_from_errno("strdup");
+ free(m);
+ return err;
+ }
+
+ m->obj_type = obj_type;
+ m->mtime = mtime;
+ *new = m;
+ return NULL;
+}
+
+static void
+clear_meta(struct got_pack_meta *meta)
+{
+ if (meta == NULL)
+ return;
+ free(meta->deltas);
+ meta->deltas = NULL;
+ free(meta->path);
+ meta->path = NULL;
+}
+
+static void
+free_nmeta(struct got_pack_meta **meta, int nmeta)
+{
+ int i;
+
+ for (i = 0; i < nmeta; i++)
+ clear_meta(meta[i]);
+ free(meta);
+}
+
+static int
+delta_order_cmp(const void *pa, const void *pb)
+{
+ struct got_pack_meta *a, *b;
+ int cmp;
+
+ a = *(struct got_pack_meta **)pa;
+ b = *(struct got_pack_meta **)pb;
+
+ if (a->obj_type != b->obj_type)
+ return a->obj_type - b->obj_type;
+ cmp = strcmp(a->path, b->path);
+ if (cmp != 0)
+ return cmp;
+ if (a->mtime != b->mtime)
+ return a->mtime - b->mtime;
+ return got_object_id_cmp(&a->id, &b->id);
+}
+
+static int
+delta_size(struct got_delta_instruction *deltas, int ndeltas)
+{
+ int i, size = 32;
+ for (i = 0; i < ndeltas; i++) {
+ if (deltas[i].copy)
+ size += GOT_DELTA_SIZE_SHIFT;
+ else
+ size += deltas[i].len + 1;
+ }
+ return size;
+}
+
+
+static const struct got_error *
+pick_deltas(struct got_pack_meta **meta, int nmeta, int nours,
+ struct got_repository *repo,
+ got_pack_progress_cb progress_cb, void *progress_arg,
+ got_cancel_cb cancel_cb, void *cancel_arg)
+{
+ const struct got_error *err = NULL;
+ struct got_pack_meta *m = NULL, *base = NULL;
+ struct got_raw_object *raw = NULL, *base_raw = NULL;
+ struct got_delta_instruction *deltas;
+ int i, j, size, ndeltas, best;
+ const int max_base_candidates = 10;
+
+ qsort(meta, nmeta, sizeof(struct got_pack_meta *), delta_order_cmp);
+ for (i = 0; i < nmeta; i++) {
+ if (cancel_cb) {
+ err = (*cancel_cb)(cancel_arg);
+ if (err)
+ break;
+ }
+ if (progress_cb) {
+ err = progress_cb(progress_arg, 0L, nours, nmeta, i, 0);
+ if (err)
+ goto done;
+ }
+ m = meta[i];
+ m->deltas = NULL;
+ m->ndeltas = 0;
+
+ if (m->obj_type == GOT_OBJ_TYPE_COMMIT ||
+ m->obj_type == GOT_OBJ_TYPE_TAG)
+ continue;
+
+ err = got_object_raw_open(&raw, repo, &m->id, 8192);
+ if (err)
+ goto done;
+
+ err = got_deltify_init(&m->dtab, raw->f, raw->hdrlen,
+ raw->size + raw->hdrlen);
+ if (err)
+ goto done;
+
+ if (i > max_base_candidates) {
+ struct got_pack_meta *n = NULL;
+ n = meta[i - (max_base_candidates + 1)];
+ got_deltify_free(n->dtab);
+ n->dtab = NULL;
+ }
+
+ best = raw->size;
+ for (j = MAX(0, i - max_base_candidates); j < i; j++) {
+ if (cancel_cb) {
+ err = (*cancel_cb)(cancel_arg);
+ if (err)
+ goto done;
+ }
+ base = meta[j];
+ /* long chains make unpacking slow, avoid such bases */
+ if (base->nchain >= 128 ||
+ base->obj_type != m->obj_type)
+ continue;
+
+ err = got_object_raw_open(&base_raw, repo, &base->id,
+ 8192);
+ if (err)
+ goto done;
+ err = got_deltify(&deltas, &ndeltas,
+ raw->f, raw->hdrlen, raw->size + raw->hdrlen,
+ base->dtab, base_raw->f, base_raw->hdrlen,
+ base_raw->size + base_raw->hdrlen);
+ got_object_raw_close(base_raw);
+ base_raw = NULL;
+ if (err)
+ goto done;
+
+ size = delta_size(deltas, ndeltas);
+ if (size + 32 < best){
+ /*
+ * if we already picked a best delta,
+ * replace it.
+ */
+ free(m->deltas);
+ best = size;
+ m->deltas = deltas;
+ m->ndeltas = ndeltas;
+ m->nchain = base->nchain + 1;
+ m->prev = base;
+ m->head = base->head;
+ if (m->head == NULL)
+ m->head = base;
+ } else {
+ free(deltas);
+ deltas = NULL;
+ ndeltas = 0;
+ }
+ }
+
+ got_object_raw_close(raw);
+ raw = NULL;
+ }
+done:
+ for (i = MAX(0, nmeta - max_base_candidates); i < nmeta; i++) {
+ got_deltify_free(meta[i]->dtab);
+ meta[i]->dtab = NULL;
+ }
+ if (raw)
+ got_object_raw_close(raw);
+ if (base_raw)
+ got_object_raw_close(base_raw);
+ return err;
+}
+
+static const struct got_error *
+search_packidx(int *found, struct got_object_id *id,
+ struct got_repository *repo)
+{
+ const struct got_error *err = NULL;
+ struct got_packidx *packidx = NULL;
+ int idx;
+
+ *found = 0;
+
+ err = got_repo_search_packidx(&packidx, &idx, repo, id);
+ if (err == NULL)
+ *found = 1; /* object is already packed */
+ else if (err->code == GOT_ERR_NO_OBJ)
+ err = NULL;
+ return err;
+}
+
+static const struct got_error *
+add_meta(struct got_pack_metavec *v, struct got_object_idset *idset,
+ struct got_object_id *id, const char *path, int obj_type,
+ time_t mtime, int loose_obj_only, struct got_repository *repo)
+{
+ const struct got_error *err;
+ struct got_pack_meta *m;
+
+ if (loose_obj_only) {
+ int is_packed;
+ err = search_packidx(&is_packed, id, repo);
+ if (err)
+ return err;
+ if (is_packed)
+ return NULL;
+ }
+
+ err = got_object_idset_add(idset, id, NULL);
+ if (err)
+ return err;
+
+ if (v == NULL)
+ return NULL;
+
+ err = alloc_meta(&m, id, path, obj_type, mtime);
+ if (err)
+ goto done;
+
+ if (v->nmeta == v->metasz){
+ size_t newsize = 2 * v->metasz;
+ struct got_pack_meta **new;
+ new = reallocarray(v->meta, newsize, sizeof(*new));
+ if (new == NULL) {
+ err = got_error_from_errno("reallocarray");
+ goto done;
+ }
+ v->meta = new;
+ v->metasz = newsize;
+ }
+done:
+ if (err) {
+ clear_meta(m);
+ free(m);
+ } else
+ v->meta[v->nmeta++] = m;
+
+ return err;
+}
+
+static const struct got_error *
+load_tree_entries(struct got_object_id_queue *ids, struct got_pack_metavec *v,
+ struct got_object_idset *idset, struct got_object_id *tree_id,
+ const char *dpath, time_t mtime, struct got_repository *repo,
+ int loose_obj_only, got_cancel_cb cancel_cb, void *cancel_arg)
+{
+ const struct got_error *err;
+ struct got_tree_object *tree;
+ char *p = NULL;
+ int i;
+
+ err = got_object_open_as_tree(&tree, repo, tree_id);
+ if (err)
+ return err;
+
+ for (i = 0; i < got_object_tree_get_nentries(tree); i++) {
+ struct got_tree_entry *e = got_object_tree_get_entry(tree, i);
+ struct got_object_id *id = got_tree_entry_get_id(e);
+ mode_t mode = got_tree_entry_get_mode(e);
+
+ if (cancel_cb) {
+ err = (*cancel_cb)(cancel_arg);
+ if (err)
+ break;
+ }
+
+ if (got_object_tree_entry_is_symlink(e) ||
+ got_object_tree_entry_is_submodule(e) ||
+ got_object_idset_contains(idset, id))
+ continue;
+
+ if (asprintf(&p, "%s%s%s", dpath, dpath[0] != '\0' ? "/" : "",
+ got_tree_entry_get_name(e)) == -1) {
+ err = got_error_from_errno("asprintf");
+ break;
+ }
+
+ if (S_ISDIR(mode)) {
+ struct got_object_qid *qid;
+ err = got_object_qid_alloc(&qid, id);
+ if (err)
+ break;
+ SIMPLEQ_INSERT_TAIL(ids, qid, entry);
+ } else if (S_ISREG(mode)) {
+ err = add_meta(v, idset, id, p, GOT_OBJ_TYPE_BLOB,
+ mtime, loose_obj_only, repo);
+ if (err)
+ break;
+ }
+ free(p);
+ p = NULL;
+ }
+
+ got_object_tree_close(tree);
+ free(p);
+ return err;
+}
+
+static const struct got_error *
+load_tree(struct got_pack_metavec *v, struct got_object_idset *idset,
+ struct got_object_id *tree_id, const char *dpath, time_t mtime,
+ int loose_obj_only, struct got_repository *repo,
+ got_cancel_cb cancel_cb, void *cancel_arg)
+{
+ const struct got_error *err = NULL;
+ struct got_object_id_queue tree_ids;
+ struct got_object_qid *qid;
+
+ if (got_object_idset_contains(idset, tree_id))
+ return NULL;
+
+ err = got_object_qid_alloc(&qid, tree_id);
+ if (err)
+ return err;
+
+ SIMPLEQ_INIT(&tree_ids);
+ SIMPLEQ_INSERT_TAIL(&tree_ids, qid, entry);
+
+ while (!SIMPLEQ_EMPTY(&tree_ids)) {
+ if (cancel_cb) {
+ err = (*cancel_cb)(cancel_arg);
+ if (err)
+ break;
+ }
+
+ qid = SIMPLEQ_FIRST(&tree_ids);
+ SIMPLEQ_REMOVE_HEAD(&tree_ids, entry);
+
+ if (got_object_idset_contains(idset, qid->id)) {
+ got_object_qid_free(qid);
+ continue;
+ }
+
+ err = add_meta(v, idset, qid->id, dpath, GOT_OBJ_TYPE_TREE,
+ mtime, loose_obj_only, repo);
+ if (err) {
+ got_object_qid_free(qid);
+ break;
+ }
+
+ err = load_tree_entries(&tree_ids, v, idset, qid->id, dpath,
+ mtime, repo, loose_obj_only, cancel_cb, cancel_arg);
+ got_object_qid_free(qid);
+ if (err)
+ break;
+ }
+
+ got_object_id_queue_free(&tree_ids);
+ return err;
+}
+
+static const struct got_error *
+load_commit(struct got_pack_metavec *v, struct got_object_idset *idset,
+ struct got_object_id *id, struct got_repository *repo, int loose_obj_only,
+ got_cancel_cb cancel_cb, void *cancel_arg)
+{
+ const struct got_error *err;
+ struct got_commit_object *commit;
+
+ if (got_object_idset_contains(idset, id))
+ return NULL;
+
+ if (loose_obj_only) {
+ int is_packed;
+ err = search_packidx(&is_packed, id, repo);
+ if (err)
+ return err;
+ if (is_packed)
+ return NULL;
+ }
+
+ err = got_object_open_as_commit(&commit, repo, id);
+ if (err)
+ return err;
+
+ err = add_meta(v, idset, id, "", GOT_OBJ_TYPE_COMMIT,
+ got_object_commit_get_committer_time(commit),
+ loose_obj_only, repo);
+ if (err)
+ goto done;
+
+ err = load_tree(v, idset, got_object_commit_get_tree_id(commit),
+ "", got_object_commit_get_committer_time(commit),
+ loose_obj_only, repo, cancel_cb, cancel_arg);
+done:
+ got_object_commit_close(commit);
+ return err;
+}
+
+static const struct got_error *
+load_tag(struct got_pack_metavec *v, struct got_object_idset *idset,
+ struct got_object_id *id, struct got_repository *repo, int loose_obj_only,
+ got_cancel_cb cancel_cb, void *cancel_arg)
+{
+ const struct got_error *err;
+ struct got_tag_object *tag = NULL;
+
+ if (got_object_idset_contains(idset, id))
+ return NULL;
+
+ if (loose_obj_only) {
+ int is_packed;
+ err = search_packidx(&is_packed, id, repo);
+ if (err)
+ return err;
+ if (is_packed)
+ return NULL;
+ }
+
+ err = got_object_open_as_tag(&tag, repo, id);
+ if (err)
+ return err;
+
+ err = add_meta(v, idset, id, "", GOT_OBJ_TYPE_TAG,
+ got_object_tag_get_tagger_time(tag),
+ loose_obj_only, repo);
+ if (err)
+ goto done;
+
+ switch (got_object_tag_get_object_type(tag)) {
+ case GOT_OBJ_TYPE_COMMIT:
+ err = load_commit(NULL, idset,
+ got_object_tag_get_object_id(tag), repo,
+ loose_obj_only, cancel_cb, cancel_arg);
+ break;
+ case GOT_OBJ_TYPE_TREE:
+ err = load_tree(v, idset, got_object_tag_get_object_id(tag),
+ "", got_object_tag_get_tagger_time(tag),
+ loose_obj_only, repo, cancel_cb, cancel_arg);
+ break;
+ default:
+ break;
+ }
+
+done:
+ got_object_tag_close(tag);
+ return err;
+}
+
+enum findtwixt_color {
+ COLOR_KEEP = 0,
+ COLOR_DROP,
+ COLOR_BLANK,
+};
+static const int findtwixt_colors[] = {
+ COLOR_KEEP,
+ COLOR_DROP,
+ COLOR_BLANK
+};
+
+static const struct got_error *
+queue_commit_id(struct got_object_id_queue *ids, struct got_object_id *id,
+ int color, struct got_repository *repo)
+{
+ const struct got_error *err;
+ struct got_object_qid *qid;
+
+ err = got_object_qid_alloc(&qid, id);
+ if (err)
+ return err;
+
+ SIMPLEQ_INSERT_TAIL(ids, qid, entry);
+ qid->data = (void *)&findtwixt_colors[color];
+ return NULL;
+}
+
+static const struct got_error *
+drop_commit(struct got_object_idset *keep, struct got_object_idset *drop,
+ struct got_object_id *id, struct got_repository *repo,
+ got_cancel_cb cancel_cb, void *cancel_arg)
+{
+ const struct got_error *err = NULL;
+ struct got_commit_object *commit;
+ const struct got_object_id_queue *parents;
+ struct got_object_id_queue ids;
+ struct got_object_qid *qid;
+
+ SIMPLEQ_INIT(&ids);
+
+ err = got_object_qid_alloc(&qid, id);
+ if (err)
+ return err;
+ SIMPLEQ_INSERT_HEAD(&ids, qid, entry);
+
+ while (!SIMPLEQ_EMPTY(&ids)) {
+ if (cancel_cb) {
+ err = (*cancel_cb)(cancel_arg);
+ if (err)
+ break;
+ }
+
+ qid = SIMPLEQ_FIRST(&ids);
+ SIMPLEQ_REMOVE_HEAD(&ids, entry);
+
+ if (got_object_idset_contains(drop, qid->id)) {
+ got_object_qid_free(qid);
+ continue;
+ }
+
+ err = got_object_idset_add(drop, qid->id, NULL);
+ if (err) {
+ got_object_qid_free(qid);
+ break;
+ }
+
+ if (!got_object_idset_contains(keep, qid->id)) {
+ got_object_qid_free(qid);
+ continue;
+ }
+
+ err = got_object_open_as_commit(&commit, repo, qid->id);
+ got_object_qid_free(qid);
+ if (err)
+ break;
+
+ parents = got_object_commit_get_parent_ids(commit);
+ if (parents) {
+ err = got_object_id_queue_copy(parents, &ids);
+ if (err) {
+ got_object_commit_close(commit);
+ break;
+ }
+ }
+ got_object_commit_close(commit);
+ }
+
+ got_object_id_queue_free(&ids);
+ return err;
+}
+
+struct append_id_arg {
+ struct got_object_id **array;
+ int idx;
+};
+
+static const struct got_error *
+append_id(struct got_object_id *id, void *data, void *arg)
+{
+ struct append_id_arg *a = arg;
+
+ a->array[a->idx] = got_object_id_dup(id);
+ if (a->array[a->idx] == NULL)
+ return got_error_from_errno("got_object_id_dup");
+
+ a->idx++;
+ return NULL;
+}
+
+static const struct got_error *
+findtwixt(struct got_object_id ***res, int *nres,
+ struct got_object_id **head, int nhead,
+ struct got_object_id **tail, int ntail,
+ struct got_repository *repo,
+ got_cancel_cb cancel_cb, void *cancel_arg)
+{
+ const struct got_error *err = NULL;
+ struct got_object_id_queue ids;
+ struct got_object_idset *keep, *drop;
+ struct got_object_qid *qid;
+ int i, ncolor, nkeep, obj_type;
+
+ SIMPLEQ_INIT(&ids);
+ *res = NULL;
+ *nres = 0;
+
+ keep = got_object_idset_alloc();
+ if (keep == NULL)
+ return got_error_from_errno("got_object_idset_alloc");
+
+ drop = got_object_idset_alloc();
+ if (drop == NULL) {
+ err = got_error_from_errno("got_object_idset_alloc");
+ goto done;
+ }
+
+ for (i = 0; i < nhead; i++) {
+ struct got_object_id *id = head[i];
+ if (id == NULL)
+ continue;
+ err = got_object_get_type(&obj_type, repo, id);
+ if (err)
+ return err;
+ if (obj_type != GOT_OBJ_TYPE_COMMIT)
+ continue;
+ err = queue_commit_id(&ids, id, COLOR_KEEP, repo);
+ if (err)
+ goto done;
+ }
+ for (i = 0; i < ntail; i++) {
+ struct got_object_id *id = tail[i];
+ if (id == NULL)
+ continue;
+ err = got_object_get_type(&obj_type, repo, id);
+ if (err)
+ return err;
+ if (obj_type != GOT_OBJ_TYPE_COMMIT)
+ continue;
+ err = queue_commit_id(&ids, id, COLOR_DROP, repo);
+ if (err)
+ goto done;
+ }
+
+ while (!SIMPLEQ_EMPTY(&ids)) {
+ int qcolor;
+ qid = SIMPLEQ_FIRST(&ids);
+ qcolor = *((int *)qid->data);
+
+ if (got_object_idset_contains(drop, qid->id))
+ ncolor = COLOR_DROP;
+ else if (got_object_idset_contains(keep, qid->id))
+ ncolor = COLOR_KEEP;
+ else
+ ncolor = COLOR_BLANK;
+
+ if (ncolor == COLOR_DROP || (ncolor == COLOR_KEEP &&
+ qcolor == COLOR_KEEP)) {
+ SIMPLEQ_REMOVE_HEAD(&ids, entry);
+ got_object_qid_free(qid);
+ continue;
+ }
+
+ if (ncolor == COLOR_KEEP && qcolor == COLOR_DROP) {
+ err = drop_commit(keep, drop, qid->id, repo,
+ cancel_cb, cancel_arg);
+ if (err)
+ goto done;
+ } else if (ncolor == COLOR_BLANK) {
+ struct got_commit_object *commit;
+ struct got_object_id *id;
+ const struct got_object_id_queue *parents;
+ struct got_object_qid *pid;
+
+ id = got_object_id_dup(qid->id);
+ if (id == NULL) {
+ err = got_error_from_errno("got_object_id_dup");
+ goto done;
+ }
+ if (qcolor == COLOR_KEEP)
+ err = got_object_idset_add(keep, id, NULL);
+ else
+ err = got_object_idset_add(drop, id, NULL);
+ if (err) {
+ free(id);
+ goto done;
+ }
+
+ err = got_object_open_as_commit(&commit, repo, id);
+ if (err) {
+ free(id);
+ goto done;
+ }
+ parents = got_object_commit_get_parent_ids(commit);
+ if (parents) {
+ SIMPLEQ_FOREACH(pid, parents, entry) {
+ err = queue_commit_id(&ids, pid->id,
+ qcolor, repo);
+ if (err) {
+ free(id);
+ goto done;
+ }
+ }
+ }
+ got_object_commit_close(commit);
+ commit = NULL;
+ } else {
+ /* should not happen */
+ err = got_error_fmt(GOT_ERR_NOT_IMPL,
+ "%s ncolor=%d qcolor=%d", __func__, ncolor, qcolor);
+ goto done;
+ }
+
+ SIMPLEQ_REMOVE_HEAD(&ids, entry);
+ got_object_qid_free(qid);
+ }
+
+ nkeep = got_object_idset_num_elements(keep);
+ if (nkeep > 0) {
+ struct append_id_arg arg;
+ arg.array = calloc(nkeep, sizeof(struct got_object_id *));
+ if (arg.array == NULL) {
+ err = got_error_from_errno("calloc");
+ goto done;
+ }
+ arg.idx = 0;
+ err = got_object_idset_for_each(keep, append_id, &arg);
+ if (err) {
+ free(arg.array);
+ goto done;
+ }
+ *res = arg.array;
+ *nres = nkeep;
+ }
+done:
+ got_object_idset_free(keep);
+ got_object_idset_free(drop);
+ got_object_id_queue_free(&ids);
+ return err;
+}
+
+static const struct got_error *
+read_meta(struct got_pack_meta ***meta, int *nmeta,
+ struct got_object_id **theirs, int ntheirs,
+ struct got_object_id **ours, int nours, struct got_repository *repo,
+ int loose_obj_only, got_pack_progress_cb progress_cb, void *progress_arg,
+ got_cancel_cb cancel_cb, void *cancel_arg)
+{
+ const struct got_error *err = NULL;
+ struct got_object_id **ids = NULL;
+ struct got_object_idset *idset;
+ int i, nobj = 0, obj_type;
+ struct got_pack_metavec v;
+
+ *meta = NULL;
+ *nmeta = 0;
+
+ idset = got_object_idset_alloc();
+ if (idset == NULL)
+ return got_error_from_errno("got_object_idset_alloc");
+
+ v.nmeta = 0;
+ v.metasz = 64;
+ v.meta = calloc(v.metasz, sizeof(struct got_pack_meta *));
+ if (v.meta == NULL) {
+ err = got_error_from_errno("reallocarray");
+ goto done;
+ }
+
+ err = findtwixt(&ids, &nobj, ours, nours, theirs, ntheirs, repo,
+ cancel_cb, cancel_arg);
+ if (err || nobj == 0)
+ goto done;
+
+ for (i = 0; i < ntheirs; i++) {
+ struct got_object_id *id = theirs[i];
+ if (id == NULL)
+ continue;
+ err = got_object_get_type(&obj_type, repo, id);
+ if (err)
+ return err;
+ if (obj_type != GOT_OBJ_TYPE_COMMIT)
+ continue;
+ err = load_commit(NULL, idset, id, repo,
+ loose_obj_only, cancel_cb, cancel_arg);
+ if (err)
+ goto done;
+ if (progress_cb) {
+ err = progress_cb(progress_arg, 0L, nours,
+ v.nmeta, 0, 0);
+ if (err)
+ goto done;
+ }
+ }
+
+ for (i = 0; i < nobj; i++) {
+ err = load_commit(&v, idset, ids[i], repo,
+ loose_obj_only, cancel_cb, cancel_arg);
+ if (err)
+ goto done;
+ if (progress_cb) {
+ err = progress_cb(progress_arg, 0L, nours,
+ v.nmeta, 0, 0);
+ if (err)
+ goto done;
+ }
+ }
+
+ for (i = 0; i < ntheirs; i++) {
+ struct got_object_id *id = ours[i];
+ if (id == NULL)
+ continue;
+ err = got_object_get_type(&obj_type, repo, id);
+ if (err)
+ return err;
+ if (obj_type != GOT_OBJ_TYPE_TAG)
+ continue;
+ err = load_tag(NULL, idset, id, repo,
+ loose_obj_only, cancel_cb, cancel_arg);
+ if (err)
+ goto done;
+ if (progress_cb) {
+ err = progress_cb(progress_arg, 0L, nours,
+ v.nmeta, 0, 0);
+ if (err)
+ goto done;
+ }
+ }
+
+ for (i = 0; i < nours; i++) {
+ struct got_object_id *id = ours[i];
+ if (id == NULL)
+ continue;
+ err = got_object_get_type(&obj_type, repo, id);
+ if (err)
+ return err;
+ if (obj_type != GOT_OBJ_TYPE_TAG)
+ continue;
+ err = load_tag(&v, idset, id, repo,
+ loose_obj_only, cancel_cb, cancel_arg);
+ if (err)
+ goto done;
+ if (progress_cb) {
+ err = progress_cb(progress_arg, 0L, nours,
+ v.nmeta, 0, 0);
+ if (err)
+ goto done;
+ }
+ }
+
+done:
+ for (i = 0; i < nobj; i++) {
+ free(ids[i]);
+ }
+ free(ids);
+ got_object_idset_free(idset);
+ if (err == NULL) {
+ *meta = v.meta;
+ *nmeta = v.nmeta;
+ } else
+ free(v.meta);
+
+ return err;
+}
+
+const struct got_error *
+hwrite(FILE *f, void *buf, int len, SHA1_CTX *ctx)
+{
+ size_t n;
+
+ SHA1Update(ctx, buf, len);
+ n = fwrite(buf, 1, len, f);
+ if (n != len)
+ return got_ferror(f, GOT_ERR_IO);
+ return NULL;
+}
+
+static void
+putbe32(char *b, uint32_t n)
+{
+ b[0] = n >> 24;
+ b[1] = n >> 16;
+ b[2] = n >> 8;
+ b[3] = n >> 0;
+}
+
+static int
+write_order_cmp(const void *pa, const void *pb)
+{
+ struct got_pack_meta *a, *b, *ahd, *bhd;
+
+ a = *(struct got_pack_meta **)pa;
+ b = *(struct got_pack_meta **)pb;
+ ahd = (a->head == NULL) ? a : a->head;
+ bhd = (b->head == NULL) ? b : b->head;
+ if (ahd->mtime != bhd->mtime)
+ return bhd->mtime - ahd->mtime;
+ if (ahd != bhd)
+ return (uintptr_t)bhd - (uintptr_t)ahd;
+ if (a->nchain != b->nchain)
+ return a->nchain - b->nchain;
+ return a->mtime - b->mtime;
+}
+
+static const struct got_error *
+packhdr(int *hdrlen, char *hdr, size_t bufsize, int obj_type, size_t len)
+{
+ size_t i;
+
+ *hdrlen = 0;
+
+ hdr[0] = obj_type << 4;
+ hdr[0] |= len & 0xf;
+ len >>= 4;
+ for (i = 1; len != 0; i++){
+ if (i >= bufsize)
+ return got_error(GOT_ERR_NO_SPACE);
+ hdr[i - 1] |= GOT_DELTA_SIZE_MORE;
+ hdr[i] = len & GOT_DELTA_SIZE_VAL_MASK;
+ len >>= GOT_DELTA_SIZE_SHIFT;
+ }
+
+ *hdrlen = i;
+ return NULL;
+}
+
+static const struct got_error *
+append(char **p, int *len, int *sz, void *seg, int nseg)
+{
+ char *n;
+
+ if (*len + nseg >= *sz) {
+ while (*len + nseg >= *sz)
+ *sz += *sz / 2;
+ n = realloc(*p, *sz);
+ if (n == NULL)
+ return got_error_from_errno("realloc");
+ *p = n;
+ }
+ memcpy(*p + *len, seg, nseg);
+ *len += nseg;
+ return NULL;
+}
+
+
+static const struct got_error *
+encodedelta(int *nd, struct got_pack_meta *m, struct got_raw_object *o,
+ off_t base_size, char **pp)
+{
+ const struct got_error *err = NULL;
+ char *p;
+ unsigned char buf[16], *bp;
+ int len, sz, i, j;
+ off_t n;
+ struct got_delta_instruction *d;
+
+ *pp = NULL;
+ *nd = 0;
+
+ sz = 128;
+ len = 0;
+ p = malloc(sz);
+ if (p == NULL)
+ return got_error_from_errno("malloc");
+
+ /* base object size */
+ buf[0] = base_size & GOT_DELTA_SIZE_VAL_MASK;
+ n = base_size >> GOT_DELTA_SIZE_SHIFT;
+ for (i = 1; n > 0; i++) {
+ buf[i - 1] |= GOT_DELTA_SIZE_MORE;
+ buf[i] = n & GOT_DELTA_SIZE_VAL_MASK;
+ n >>= GOT_DELTA_SIZE_SHIFT;
+ }
+ err = append(&p, &len, &sz, buf, i);
+ if (err)
+ return err;
+
+ /* target object size */
+ buf[0] = o->size & GOT_DELTA_SIZE_VAL_MASK;
+ n = o->size >> GOT_DELTA_SIZE_SHIFT;
+ for (i = 1; n > 0; i++) {
+ buf[i - 1] |= GOT_DELTA_SIZE_MORE;
+ buf[i] = n & GOT_DELTA_SIZE_VAL_MASK;
+ n >>= GOT_DELTA_SIZE_SHIFT;
+ }
+ err = append(&p, &len, &sz, buf, i);
+ if (err)
+ return err;
+ for (j = 0; j < m->ndeltas; j++) {
+ d = &m->deltas[j];
+ if (d->copy) {
+ n = d->offset;
+ bp = &buf[1];
+ buf[0] = GOT_DELTA_BASE_COPY;
+ for (i = 0; i < 4; i++) {
+ /* DELTA_COPY_OFF1 ... DELTA_COPY_OFF4 */
+ buf[0] |= 1 << i;
+ *bp++ = n & 0xff;
+ n >>= 8;
+ if (n == 0)
+ break;
+ }
+
+ n = d->len;
+ if (n != GOT_DELTA_COPY_DEFAULT_LEN) {
+ /* DELTA_COPY_LEN1 ... DELTA_COPY_LEN3 */
+ for (i = 0; i < 3 && n > 0; i++) {
+ buf[0] |= 1 << (i + 4);
+ *bp++ = n & 0xff;
+ n >>= 8;
+ }
+ }
+ err = append(&p, &len, &sz, buf, bp - buf);
+ if (err)
+ return err;
+ } else {
+ char content[128];
+ size_t r;
+ if (fseeko(o->f, o->hdrlen + d->offset, SEEK_SET) == -1)
+ return got_error_from_errno("fseeko");
+ n = 0;
+ while (n != d->len) {
+ buf[0] = (d->len - n < 127) ? d->len - n : 127;
+ err = append(&p, &len, &sz, buf, 1);
+ if (err)
+ return err;
+ r = fread(content, 1, buf[0], o->f);
+ if (r != buf[0])
+ return got_ferror(o->f, GOT_ERR_IO);
+ err = append(&p, &len, &sz, content, buf[0]);
+ if (err)
+ return err;
+ n += buf[0];
+ }
+ }
+ }
+ *pp = p;
+ *nd = len;
+ return NULL;
+}
+
+static int
+packoff(char *hdr, off_t off)
+{
+ int i, j;
+ char rbuf[8];
+
+ rbuf[0] = off & GOT_DELTA_SIZE_VAL_MASK;
+ for (i = 1; (off >>= GOT_DELTA_SIZE_SHIFT) != 0; i++) {
+ rbuf[i] = (--off & GOT_DELTA_SIZE_VAL_MASK) |
+ GOT_DELTA_SIZE_MORE;
+ }
+
+ j = 0;
+ while (i > 0)
+ hdr[j++] = rbuf[--i];
+ return j;
+}
+
+static const struct got_error *
+genpack(uint8_t *pack_sha1, FILE *packfile,
+ struct got_pack_meta **meta, int nmeta, int nours,
+ int use_offset_deltas, struct got_repository *repo,
+ got_pack_progress_cb progress_cb, void *progress_arg,
+ got_cancel_cb cancel_cb, void *cancel_arg)
+{
+ const struct got_error *err = NULL;
+ int i, nh, nd;
+ SHA1_CTX ctx;
+ struct got_pack_meta *m;
+ struct got_raw_object *raw;
+ char *p = NULL, buf[32];
+ size_t outlen, n;
+ struct got_deflate_checksum csum;
+ off_t packfile_size = 0;
+
+ SHA1Init(&ctx);
+ csum.output_sha1 = &ctx;
+ csum.output_crc = NULL;
+
+ err = hwrite(packfile, "PACK", 4, &ctx);
+ if (err)
+ return err;
+ putbe32(buf, GOT_PACKFILE_VERSION);
+ err = hwrite(packfile, buf, 4, &ctx);
+ if (err)
+ goto done;
+ putbe32(buf, nmeta);
+ err = hwrite(packfile, buf, 4, &ctx);
+ if (err)
+ goto done;
+ qsort(meta, nmeta, sizeof(struct got_pack_meta *), write_order_cmp);
+ for (i = 0; i < nmeta; i++) {
+ if (progress_cb) {
+ err = progress_cb(progress_arg, packfile_size, nours,
+ nmeta, nmeta, i);
+ if (err)
+ goto done;
+ }
+ m = meta[i];
+ m->off = ftello(packfile);
+ err = got_object_raw_open(&raw, repo, &m->id, 8192);
+ if (err)
+ goto done;
+ if (m->deltas == NULL) {
+ err = packhdr(&nh, buf, sizeof(buf),
+ m->obj_type, raw->size);
+ if (err)
+ goto done;
+ err = hwrite(packfile, buf, nh, &ctx);
+ if (err)
+ goto done;
+ packfile_size += nh;
+ if (fseeko(raw->f, raw->hdrlen, SEEK_SET) == -1) {
+ err = got_error_from_errno("fseeko");
+ goto done;
+ }
+ err = got_deflate_to_file(&outlen, raw->f, packfile,
+ &csum);
+ if (err)
+ goto done;
+ packfile_size += outlen;
+ } else {
+ FILE *delta_file;
+ struct got_raw_object *base_raw;
+ err = got_object_raw_open(&base_raw, repo,
+ &m->prev->id, 8192);
+ if (err)
+ goto done;
+ err = encodedelta(&nd, m, raw, base_raw->size, &p);
+ if (err)
+ goto done;
+ got_object_raw_close(base_raw);
+ if (use_offset_deltas && m->prev->off != 0) {
+ err = packhdr(&nh, buf, sizeof(buf),
+ GOT_OBJ_TYPE_OFFSET_DELTA, nd);
+ if (err)
+ goto done;
+ nh += packoff(buf + nh,
+ m->off - m->prev->off);
+ err = hwrite(packfile, buf, nh, &ctx);
+ if (err)
+ goto done;
+ packfile_size += nh;
+ } else {
+ err = packhdr(&nh, buf, sizeof(buf),
+ GOT_OBJ_TYPE_REF_DELTA, nd);
+ err = hwrite(packfile, buf, nh, &ctx);
+ if (err)
+ goto done;
+ packfile_size += nh;
+ err = hwrite(packfile, m->prev->id.sha1,
+ sizeof(m->prev->id.sha1), &ctx);
+ packfile_size += sizeof(m->prev->id.sha1);
+ if (err)
+ goto done;
+ }
+ /* XXX need got_deflate_from_mem() */
+ delta_file = fmemopen(p, nd, "r");
+ if (delta_file == NULL) {
+ err = got_error_from_errno("fmemopen");
+ goto done;
+ }
+ err = got_deflate_to_file(&outlen, delta_file,
+ packfile, &csum);
+ fclose(delta_file);
+ if (err)
+ goto done;
+ packfile_size += outlen;
+ free(p);
+ p = NULL;
+ }
+ got_object_raw_close(raw);
+ raw = NULL;
+ }
+ SHA1Final(pack_sha1, &ctx);
+ n = fwrite(pack_sha1, 1, SHA1_DIGEST_LENGTH, packfile);
+ if (n != SHA1_DIGEST_LENGTH)
+ err = got_ferror(packfile, GOT_ERR_IO);
+ packfile_size += SHA1_DIGEST_LENGTH;
+ packfile_size += 16; /* pack file header */
+ err = progress_cb(progress_arg, packfile_size, nours,
+ nmeta, nmeta, nmeta);
+ if (err)
+ goto done;
+done:
+ free(p);
+ return err;
+}
+
+const struct got_error *
+got_pack_create(uint8_t *packsha1, FILE *packfile,
+ struct got_object_id **theirs, int ntheirs,
+ struct got_object_id **ours, int nours,
+ struct got_repository *repo, int loose_obj_only,
+ got_pack_progress_cb progress_cb, void *progress_arg,
+ got_cancel_cb cancel_cb, void *cancel_arg)
+{
+ const struct got_error *err;
+ struct got_pack_meta **meta;
+ int nmeta;
+
+ err = read_meta(&meta, &nmeta, theirs, ntheirs, ours, nours, repo,
+ loose_obj_only, progress_cb, progress_arg, cancel_cb, cancel_arg);
+ if (err)
+ return err;
+
+ if (nmeta == 0) {
+ err = got_error(GOT_ERR_CANNOT_PACK);
+ goto done;
+ }
+
+ err = pick_deltas(meta, nmeta, nours, repo,
+ progress_cb, progress_arg, cancel_cb, cancel_arg);
+ if (err)
+ goto done;
+
+ err = genpack(packsha1, packfile, meta, nmeta, nours, 1, repo,
+ progress_cb, progress_arg, cancel_cb, cancel_arg);
+ if (err)
+ goto done;
+done:
+ free_nmeta(meta, nmeta);
+ return err;
+}
blob - fcfc4cbf94642d9a6d37ac919bf5ba788dc7d9f8
blob + b89875ec9e861ec53c6a8c19a5fcbaa184ead648
--- lib/reference.c
+++ lib/reference.c
@@ -787,8 +787,8 @@ done:
return err;
}
-static const struct got_error *
-insert_ref(struct got_reflist_entry **newp, struct got_reflist_head *refs,
+const struct got_error *
+got_reflist_insert(struct got_reflist_entry **newp, struct got_reflist_head *refs,
struct got_reference *ref, struct got_repository *repo,
got_ref_cmp_cb cmp_cb, void *cmp_arg)
{
@@ -881,7 +881,7 @@ gather_on_disk_refs(struct got_reflist_head *refs, con
goto done;
if (ref) {
struct got_reflist_entry *new;
- err = insert_ref(&new, refs, ref, repo,
+ err = got_reflist_insert(&new, refs, ref, repo,
cmp_cb, cmp_arg);
if (err || new == NULL /* duplicate */)
got_ref_close(ref);
@@ -932,7 +932,7 @@ got_ref_list(struct got_reflist_head *refs, struct got
err = open_ref(&ref, path_refs, "", GOT_REF_HEAD, 0);
if (err)
goto done;
- err = insert_ref(&new, refs, ref, repo,
+ err = got_reflist_insert(&new, refs, ref, repo,
cmp_cb, cmp_arg);
if (err || new == NULL /* duplicate */)
got_ref_close(ref);
@@ -952,7 +952,7 @@ got_ref_list(struct got_reflist_head *refs, struct got
goto done;
/* Try to look up references in a given namespace. */
} else {
- err = insert_ref(&new, refs, ref, repo,
+ err = got_reflist_insert(&new, refs, ref, repo,
cmp_cb, cmp_arg);
if (err || new == NULL /* duplicate */)
got_ref_close(ref);
@@ -1037,7 +1037,7 @@ got_ref_list(struct got_reflist_head *refs, struct got
continue;
}
}
- err = insert_ref(&new, refs, ref, repo,
+ err = got_reflist_insert(&new, refs, ref, repo,
cmp_cb, cmp_arg);
if (err || new == NULL /* duplicate */)
got_ref_close(ref);
@@ -1282,7 +1282,7 @@ delete_packed_ref(struct got_reference *delref, struct
continue;
}
- err = insert_ref(&new, &refs, ref, repo,
+ err = got_reflist_insert(&new, &refs, ref, repo,
got_ref_cmp_by_name, NULL);
if (err || new == NULL /* duplicate */)
got_ref_close(ref);
blob - fd7075fb6429bbf5503805287967e8c9fd51c710
blob + a97375aa5712bae416359100f0059fa17f6cfc46
--- lib/repository.c
+++ lib/repository.c
@@ -1912,7 +1912,7 @@ const struct got_error *
got_repo_get_packfile_info(int *npackfiles, int *nobjects,
off_t *total_packsize, struct got_repository *repo)
{
- const struct got_error *err;
+ const struct got_error *err = NULL;
DIR *packdir = NULL;
struct dirent *dent;
struct got_packidx *packidx = NULL;
blob - /dev/null
blob + 95151516f8146f021c5f8f3a919b810499c016e3 (mode 644)
--- /dev/null
+++ lib/repository_admin.c
@@ -0,0 +1,594 @@
+/*
+ * Copyright (c) 2020 Stefan Sperling <stsp@openbsd.org>
+ *
+ * Permission to use, copy, modify, and distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+ * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ */
+
+#include <sys/types.h>
+#include <sys/queue.h>
+#include <sys/uio.h>
+#include <sys/stat.h>
+#include <sys/socket.h>
+#include <sys/wait.h>
+
+#include <dirent.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <stdint.h>
+#include <sha1.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <limits.h>
+#include <unistd.h>
+#include <imsg.h>
+
+#include "got_error.h"
+#include "got_cancel.h"
+#include "got_object.h"
+#include "got_reference.h"
+#include "got_repository.h"
+#include "got_repository_admin.h"
+#include "got_opentemp.h"
+#include "got_path.h"
+
+#include "got_lib_delta.h"
+#include "got_lib_object.h"
+#include "got_lib_object_cache.h"
+#include "got_lib_pack.h"
+#include "got_lib_privsep.h"
+#include "got_lib_repository.h"
+#include "got_lib_pack_create.h"
+#include "got_lib_sha1.h"
+
+#ifndef nitems
+#define nitems(_a) (sizeof((_a)) / sizeof((_a)[0]))
+#endif
+
+static const struct got_error *
+get_reflist_object_ids(struct got_object_id ***ids, int *nobjects,
+ unsigned int wanted_obj_type_mask, struct got_reflist_head *refs,
+ struct got_repository *repo,
+ got_cancel_cb cancel_cb, void *cancel_arg)
+{
+ const struct got_error *err = NULL;
+ const size_t alloc_chunksz = 256;
+ size_t nalloc;
+ struct got_reflist_entry *re;
+ int i;
+
+ *ids = NULL;
+ *nobjects = 0;
+
+ *ids = reallocarray(NULL, alloc_chunksz, sizeof(struct got_object_id *));
+ if (*ids == NULL)
+ return got_error_from_errno("reallocarray");
+ nalloc = alloc_chunksz;
+
+ TAILQ_FOREACH(re, refs, entry) {
+ struct got_object_id *id;
+
+ if (cancel_cb) {
+ err = cancel_cb(cancel_arg);
+ if (err)
+ goto done;
+ }
+
+ err = got_ref_resolve(&id, repo, re->ref);
+ if (err)
+ goto done;
+
+ if (wanted_obj_type_mask != GOT_OBJ_TYPE_ANY) {
+ int obj_type;
+ err = got_object_get_type(&obj_type, repo, id);
+ if (err)
+ goto done;
+ if ((wanted_obj_type_mask & (1 << obj_type)) == 0) {
+ free(id);
+ id = NULL;
+ continue;
+ }
+ }
+
+ if (nalloc >= *nobjects) {
+ struct got_object_id **new;
+ new = recallocarray(*ids, nalloc,
+ nalloc + alloc_chunksz,
+ sizeof(struct got_object_id *));
+ if (new == NULL) {
+ err = got_error_from_errno(
+ "recallocarray");
+ goto done;
+ }
+ *ids = new;
+ nalloc += alloc_chunksz;
+ }
+ (*ids)[*nobjects] = id;
+ if ((*ids)[*nobjects] == NULL) {
+ err = got_error_from_errno("got_object_id_dup");
+ goto done;
+ }
+ (*nobjects)++;
+ }
+done:
+ if (err) {
+ for (i = 0; i < *nobjects; i++)
+ free((*ids)[i]);
+ free(*ids);
+ *ids = NULL;
+ *nobjects = 0;
+ }
+ return err;
+}
+
+const struct got_error *
+got_repo_pack_objects(FILE **packfile, struct got_object_id **pack_hash,
+ struct got_reflist_head *include_refs,
+ struct got_reflist_head *exclude_refs, struct got_repository *repo,
+ int loose_obj_only, got_pack_progress_cb progress_cb, void *progress_arg,
+ got_cancel_cb cancel_cb, void *cancel_arg)
+{
+ const struct got_error *err = NULL;
+ struct got_object_id **ours = NULL, **theirs = NULL;
+ int nours = 0, ntheirs = 0, packfd = -1, i;
+ char *tmpfile_path = NULL, *path = NULL, *packfile_path = NULL;
+ char *sha1_str = NULL;
+
+ *packfile = NULL;
+ *pack_hash = NULL;
+
+ if (asprintf(&path, "%s/%s/packing.pack",
+ got_repo_get_path_git_dir(repo), GOT_OBJECTS_PACK_DIR) == -1) {
+ err = got_error_from_errno("asprintf");
+ goto done;
+ }
+ err = got_opentemp_named_fd(&tmpfile_path, &packfd, path);
+ if (err)
+ goto done;
+
+ if (fchmod(packfd, GOT_DEFAULT_FILE_MODE) != 0) {
+ err = got_error_from_errno2("fchmod", tmpfile_path);
+ goto done;
+ }
+
+ *packfile = fdopen(packfd, "w");
+ if (*packfile == NULL) {
+ err = got_error_from_errno2("fdopen", tmpfile_path);
+ goto done;
+ }
+ packfd = -1;
+
+ err = get_reflist_object_ids(&ours, &nours,
+ (1 << GOT_OBJ_TYPE_COMMIT) | (1 << GOT_OBJ_TYPE_TAG),
+ include_refs, repo, cancel_cb, cancel_arg);
+ if (err)
+ goto done;
+
+ if (nours == 0) {
+ err = got_error(GOT_ERR_CANNOT_PACK);
+ goto done;
+ }
+
+ if (!TAILQ_EMPTY(exclude_refs)) {
+ err = get_reflist_object_ids(&theirs, &ntheirs,
+ (1 << GOT_OBJ_TYPE_COMMIT) | (1 << GOT_OBJ_TYPE_TAG),
+ exclude_refs, repo,
+ cancel_cb, cancel_arg);
+ if (err)
+ goto done;
+ }
+
+ *pack_hash = calloc(1, sizeof(**pack_hash));
+ if (*pack_hash == NULL) {
+ err = got_error_from_errno("calloc");
+ goto done;
+ }
+
+ err = got_pack_create((*pack_hash)->sha1, *packfile, theirs, ntheirs,
+ ours, nours, repo, loose_obj_only, progress_cb, progress_arg,
+ cancel_cb, cancel_arg);
+ if (err)
+ goto done;
+
+ err = got_object_id_str(&sha1_str, *pack_hash);
+ if (err)
+ goto done;
+ if (asprintf(&packfile_path, "%s/%s/pack-%s.pack",
+ got_repo_get_path_git_dir(repo), GOT_OBJECTS_PACK_DIR,
+ sha1_str) == -1) {
+ err = got_error_from_errno("asprintf");
+ goto done;
+ }
+
+ if (fflush(*packfile) == -1) {
+ err = got_error_from_errno("fflush");
+ goto done;
+ }
+ if (fseek(*packfile, 0L, SEEK_SET) == -1) {
+ err = got_error_from_errno("fseek");
+ goto done;
+ }
+ if (rename(tmpfile_path, packfile_path) == -1) {
+ err = got_error_from_errno3("rename", tmpfile_path,
+ packfile_path);
+ goto done;
+ }
+ free(tmpfile_path);
+ tmpfile_path = NULL;
+done:
+ for (i = 0; i < nours; i++)
+ free(ours[i]);
+ free(ours);
+ for (i = 0; i < ntheirs; i++)
+ free(theirs[i]);
+ free(theirs);
+ if (packfd != -1 && close(packfd) == -1 && err == NULL)
+ err = got_error_from_errno2("close", packfile_path);
+ if (tmpfile_path && unlink(tmpfile_path) == -1 && err == NULL)
+ err = got_error_from_errno2("unlink", tmpfile_path);
+ free(tmpfile_path);
+ free(packfile_path);
+ free(sha1_str);
+ free(path);
+ if (err) {
+ free(*pack_hash);
+ *pack_hash = NULL;
+ if (*packfile)
+ fclose(*packfile);
+ *packfile = NULL;
+ }
+ return err;
+}
+
+const struct got_error *
+got_repo_index_pack(FILE *packfile, struct got_object_id *pack_hash,
+ struct got_repository *repo,
+ got_pack_index_progress_cb progress_cb, void *progress_arg,
+ got_cancel_cb cancel_cb, void *cancel_arg)
+{
+ size_t i;
+ char *path;
+ int imsg_idxfds[2];
+ int npackfd = -1, idxfd = -1, nidxfd = -1;
+ int tmpfds[3];
+ int idxstatus, done = 0;
+ const struct got_error *err;
+ struct imsgbuf idxibuf;
+ pid_t idxpid;
+ char *tmpidxpath = NULL;
+ char *packfile_path = NULL, *idxpath = NULL, *id_str = NULL;
+ const char *repo_path = got_repo_get_path_git_dir(repo);
+ struct stat sb;
+
+ for (i = 0; i < nitems(tmpfds); i++)
+ tmpfds[i] = -1;
+
+ if (asprintf(&path, "%s/%s/indexing.idx",
+ repo_path, GOT_OBJECTS_PACK_DIR) == -1) {
+ err = got_error_from_errno("asprintf");
+ goto done;
+ }
+ err = got_opentemp_named_fd(&tmpidxpath, &idxfd, path);
+ free(path);
+ if (err)
+ goto done;
+ if (fchmod(idxfd, GOT_DEFAULT_FILE_MODE) != 0) {
+ err = got_error_from_errno2("fchmod", tmpidxpath);
+ goto done;
+ }
+
+ nidxfd = dup(idxfd);
+ if (nidxfd == -1) {
+ err = got_error_from_errno("dup");
+ goto done;
+ }
+
+ for (i = 0; i < nitems(tmpfds); i++) {
+ tmpfds[i] = got_opentempfd();
+ if (tmpfds[i] == -1) {
+ err = got_error_from_errno("got_opentempfd");
+ goto done;
+ }
+ }
+
+ err = got_object_id_str(&id_str, pack_hash);
+ if (err)
+ goto done;
+
+ if (asprintf(&packfile_path, "%s/%s/pack-%s.pack",
+ repo_path, GOT_OBJECTS_PACK_DIR, id_str) == -1) {
+ err = got_error_from_errno("asprintf");
+ goto done;
+ }
+
+ if (fstat(fileno(packfile), &sb) == -1) {
+ err = got_error_from_errno2("fstat", packfile_path);
+ goto done;
+ }
+
+ if (asprintf(&idxpath, "%s/%s/pack-%s.idx",
+ repo_path, GOT_OBJECTS_PACK_DIR, id_str) == -1) {
+ err = got_error_from_errno("asprintf");
+ goto done;
+ }
+
+ if (socketpair(AF_UNIX, SOCK_STREAM, PF_UNSPEC, imsg_idxfds) == -1) {
+ err = got_error_from_errno("socketpair");
+ goto done;
+ }
+ idxpid = fork();
+ if (idxpid == -1) {
+ err= got_error_from_errno("fork");
+ goto done;
+ } else if (idxpid == 0)
+ got_privsep_exec_child(imsg_idxfds,
+ GOT_PATH_PROG_INDEX_PACK, packfile_path);
+ if (close(imsg_idxfds[1]) == -1) {
+ err = got_error_from_errno("close");
+ goto done;
+ }
+ imsg_init(&idxibuf, imsg_idxfds[0]);
+
+ npackfd = dup(fileno(packfile));
+ if (npackfd == -1) {
+ err = got_error_from_errno("dup");
+ goto done;
+ }
+ err = got_privsep_send_index_pack_req(&idxibuf, pack_hash->sha1,
+ npackfd);
+ if (err != NULL)
+ goto done;
+ npackfd = -1;
+ err = got_privsep_send_index_pack_outfd(&idxibuf, nidxfd);
+ if (err != NULL)
+ goto done;
+ nidxfd = -1;
+ for (i = 0; i < nitems(tmpfds); i++) {
+ err = got_privsep_send_tmpfd(&idxibuf, tmpfds[i]);
+ if (err != NULL)
+ goto done;
+ tmpfds[i] = -1;
+ }
+ done = 0;
+ while (!done) {
+ int nobj_total, nobj_indexed, nobj_loose, nobj_resolved;
+
+ if (cancel_cb) {
+ err = cancel_cb(cancel_arg);
+ if (err)
+ goto done;
+ }
+
+ err = got_privsep_recv_index_progress(&done, &nobj_total,
+ &nobj_indexed, &nobj_loose, &nobj_resolved,
+ &idxibuf);
+ if (err != NULL)
+ goto done;
+ if (nobj_indexed != 0) {
+ err = progress_cb(progress_arg, sb.st_size,
+ nobj_total, nobj_indexed, nobj_loose,
+ nobj_resolved);
+ if (err)
+ break;
+ }
+ imsg_clear(&idxibuf);
+ }
+ if (close(imsg_idxfds[0]) == -1) {
+ err = got_error_from_errno("close");
+ goto done;
+ }
+ if (waitpid(idxpid, &idxstatus, 0) == -1) {
+ err = got_error_from_errno("waitpid");
+ goto done;
+ }
+
+ if (rename(tmpidxpath, idxpath) == -1) {
+ err = got_error_from_errno3("rename", tmpidxpath, idxpath);
+ goto done;
+ }
+ free(tmpidxpath);
+ tmpidxpath = NULL;
+
+done:
+ if (tmpidxpath && unlink(tmpidxpath) == -1 && err == NULL)
+ err = got_error_from_errno2("unlink", tmpidxpath);
+ if (npackfd != -1 && close(npackfd) == -1 && err == NULL)
+ err = got_error_from_errno("close");
+ if (idxfd != -1 && close(idxfd) == -1 && err == NULL)
+ err = got_error_from_errno("close");
+ for (i = 0; i < nitems(tmpfds); i++) {
+ if (tmpfds[i] != -1 && close(tmpfds[i]) == -1 && err == NULL)
+ err = got_error_from_errno("close");
+ }
+ free(tmpidxpath);
+ free(idxpath);
+ free(packfile_path);
+ return err;
+}
+
+const struct got_error *
+got_repo_find_pack(FILE **packfile, struct got_object_id **pack_hash,
+ struct got_repository *repo, const char *packfile_path)
+{
+ const struct got_error *err = NULL;
+ const char *packdir_path = NULL;
+ char *packfile_name = NULL, *p, *dot;
+ struct got_object_id id;
+ int packfd = -1;
+
+ *packfile = NULL;
+ *pack_hash = NULL;
+
+ packdir_path = got_repo_get_path_objects_pack(repo);
+ if (packdir_path == NULL)
+ return got_error_from_errno("got_repo_get_path_objects_pack");
+
+ if (!got_path_is_child(packfile_path, packdir_path,
+ strlen(packdir_path))) {
+ err = got_error_path(packfile_path, GOT_ERR_BAD_PATH);
+ goto done;
+
+ }
+
+ err = got_path_basename(&packfile_name, packfile_path);
+ if (err)
+ goto done;
+ p = packfile_name;
+
+ if (strncmp(p, "pack-", 5) != 0) {
+ err = got_error_fmt(GOT_ERR_BAD_PATH,
+ "'%s' is not a valid pack file name",
+ packfile_name);
+ goto done;
+ }
+ p += 5;
+ dot = strchr(p, '.');
+ if (dot == NULL) {
+ err = got_error_fmt(GOT_ERR_BAD_PATH,
+ "'%s' is not a valid pack file name",
+ packfile_name);
+ goto done;
+ }
+ if (strcmp(dot + 1, "pack") != 0) {
+ err = got_error_fmt(GOT_ERR_BAD_PATH,
+ "'%s' is not a valid pack file name",
+ packfile_name);
+ goto done;
+ }
+ *dot = '\0';
+ if (!got_parse_sha1_digest(id.sha1, p)) {
+ err = got_error_fmt(GOT_ERR_BAD_PATH,
+ "'%s' is not a valid pack file name",
+ packfile_name);
+ goto done;
+ }
+
+ *pack_hash = got_object_id_dup(&id);
+ if (*pack_hash == NULL) {
+ err = got_error_from_errno("got_object_id_dup");
+ goto done;
+ }
+
+ packfd = open(packfile_path, O_RDONLY | O_NOFOLLOW);
+ if (packfd == -1) {
+ err = got_error_from_errno2("open", packfile_path);
+ goto done;
+ }
+
+ *packfile = fdopen(packfd, "r");
+ if (*packfile == NULL) {
+ err = got_error_from_errno2("fdopen", packfile_path);
+ goto done;
+ }
+ packfd = -1;
+done:
+ if (packfd != -1 && close(packfd) == -1 && err == NULL)
+ err = got_error_from_errno2("close", packfile_path);
+ free(packfile_name);
+ if (err) {
+ free(*pack_hash);
+ *pack_hash = NULL;
+ }
+ return err;
+}
+
+const struct got_error *
+got_repo_list_pack(FILE *packfile, struct got_object_id *pack_hash,
+ struct got_repository *repo, got_pack_list_cb list_cb, void *list_arg,
+ got_cancel_cb cancel_cb, void *cancel_arg)
+{
+ const struct got_error *err = NULL;
+ char *id_str = NULL, *idxpath = NULL, *packpath = NULL;
+ struct got_packidx *packidx = NULL;
+ struct got_pack *pack = NULL;
+ uint32_t nobj, i;
+
+ err = got_object_id_str(&id_str, pack_hash);
+ if (err)
+ goto done;
+
+ if (asprintf(&packpath, "%s/pack-%s.pack",
+ GOT_OBJECTS_PACK_DIR, id_str) == -1) {
+ err = got_error_from_errno("asprintf");
+ goto done;
+ }
+ if (asprintf(&idxpath, "%s/pack-%s.idx",
+ GOT_OBJECTS_PACK_DIR, id_str) == -1) {
+ err = got_error_from_errno("asprintf");
+ goto done;
+ }
+
+ err = got_packidx_open(&packidx, got_repo_get_fd(repo), idxpath, 1);
+ if (err)
+ goto done;
+
+ err = got_repo_cache_pack(&pack, repo, packpath, packidx);
+ if (err)
+ goto done;
+
+ nobj = be32toh(packidx->hdr.fanout_table[0xff]);
+ for (i = 0; i < nobj; i++) {
+ struct got_packidx_object_id *oid;
+ struct got_object_id id, base_id;
+ off_t offset, base_offset = 0;
+ uint8_t type;
+ uint64_t size;
+ size_t tslen, len;
+
+ if (cancel_cb) {
+ err = cancel_cb(cancel_arg);
+ if (err)
+ break;
+ }
+ oid = &packidx->hdr.sorted_ids[i];
+ memcpy(id.sha1, oid->sha1, SHA1_DIGEST_LENGTH);
+
+ offset = got_packidx_get_object_offset(packidx, i);
+ if (offset == -1) {
+ err = got_error(GOT_ERR_BAD_PACKIDX);
+ goto done;
+ }
+
+ err = got_pack_parse_object_type_and_size(&type, &size, &tslen,
+ pack, offset);
+ if (err)
+ goto done;
+
+ switch (type) {
+ case GOT_OBJ_TYPE_OFFSET_DELTA:
+ err = got_pack_parse_offset_delta(&base_offset, &len,
+ pack, offset, tslen);
+ if (err)
+ goto done;
+ break;
+ case GOT_OBJ_TYPE_REF_DELTA:
+ err = got_pack_parse_ref_delta(&base_id,
+ pack, offset, tslen);
+ if (err)
+ goto done;
+ break;
+ }
+ err = (*list_cb)(list_arg, &id, type, offset, size,
+ base_offset, &base_id);
+ if (err)
+ goto done;
+ }
+
+done:
+ free(id_str);
+ free(idxpath);
+ free(packpath);
+ if (packidx)
+ got_packidx_close(packidx);
+ return err;
+}
blob - e11b7d2292e10741d6a883a408cbd72d61137d6f
blob + 8cb2bc37152c96e55501d1e2f7927e159f2ca29f
--- libexec/got-read-object/got-read-object.c
+++ libexec/got-read-object/got-read-object.c
@@ -77,7 +77,7 @@ send_raw_obj(struct imsgbuf *ibuf, struct got_object *
return err;
}
- if (obj->size <= GOT_PRIVSEP_INLINE_OBJECT_DATA_MAX)
+ if (obj->size + obj->hdrlen <= GOT_PRIVSEP_INLINE_OBJECT_DATA_MAX)
err = got_inflate_to_mem(&data, &len, &consumed, f);
else
err = got_inflate_to_fd(&len, f, outfd);
@@ -85,12 +85,12 @@ send_raw_obj(struct imsgbuf *ibuf, struct got_object *
goto done;
if (len < obj->hdrlen || len != obj->hdrlen + obj->size) {
- fprintf(stderr, "len=%zd obj->hdrlen=%zd obj->size=%zd\n", len, obj->hdrlen, obj->size);
err = got_error(GOT_ERR_BAD_OBJ_HDR);
goto done;
}
- err = got_privsep_send_raw_obj(ibuf, len, obj->hdrlen, data);
+ err = got_privsep_send_raw_obj(ibuf, obj->size, obj->hdrlen, data);
+
done:
free(data);
if (fclose(f) == EOF && err == NULL)
blob - 30756b439f8433c9f8b8f391bd93db8ddd6feb9f
blob + 612026c469e1b0b43b39ffebdb8592d4b22c474f
--- libexec/got-read-pack/got-read-pack.c
+++ libexec/got-read-pack/got-read-pack.c
@@ -827,7 +827,7 @@ raw_object_request(struct imsg *imsg, struct imsgbuf *
if (err)
goto done;
- err = got_privsep_send_raw_obj(ibuf, size, obj->hdrlen, buf);
+ err = got_privsep_send_raw_obj(ibuf, obj->size, obj->hdrlen, buf);
done:
free(buf);
if (outfile && fclose(outfile) == EOF && err == NULL)
blob - e32b5a142455da777776713187383ad1166a01d4
blob + 994b3c728968b83c996b8eb7fd40f01354a38383
--- regress/cmdline/Makefile
+++ regress/cmdline/Makefile
@@ -1,6 +1,6 @@
REGRESS_TARGETS=checkout update status log add rm diff blame branch tag \
ref commit revert cherrypick backout rebase import histedit \
- integrate stage unstage cat clone fetch tree
+ integrate stage unstage cat clone fetch tree pack
NOOBJ=Yes
GOT_TEST_ROOT=/tmp
@@ -80,4 +80,7 @@ fetch:
tree:
./tree.sh -q -r "$(GOT_TEST_ROOT)"
+pack:
+ ./pack.sh -q -r "$(GOT_TEST_ROOT)"
+
.include <bsd.regress.mk>
blob - /dev/null
blob + b6d15ae7a39b4f755d66d6ea89e268ee45b22005 (mode 755)
--- /dev/null
+++ regress/cmdline/pack.sh
@@ -0,0 +1,589 @@
+#!/bin/sh
+#
+# Copyright (c) 2021 Stefan Sperling <stsp@openbsd.org>
+#
+# Permission to use, copy, modify, and distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+. ./common.sh
+
+# disable automatic packing for these tests
+export GOT_TEST_PACK=""
+
+test_pack_all_loose_objects() {
+ local testroot=`test_init pack_all_loose_objects`
+
+ # tags should also be packed
+ got tag -r $testroot/repo -m 1.0 1.0 >/dev/null
+
+ # no pack files should exist yet
+ ls $testroot/repo/.git/objects/pack/ > $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+ echo -n > $testroot/stdout.expected
+ cmp -s $testroot/stdout.expected $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ diff -u $testroot/stdout.expected $testroot/stdout
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ gotadmin pack -r $testroot/repo > $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ echo "gotadmin pack failed unexpectedly" >&2
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+ packname=`grep ^Wrote $testroot/stdout | cut -d ' ' -f2`
+ gotadmin listpack $testroot/repo/.git/objects/pack/pack-$packname \
+ > $testroot/stdout
+
+ for d in $testroot/repo/.git/objects/[0-9a-f][0-9a-f]; do
+ id0=`basename $d`
+ ret=0
+ for e in `ls $d`; do
+ obj_id=${id0}${e}
+ if grep -q ^$obj_id $testroot/stdout; then
+ continue
+ fi
+ echo "loose object $obj_id was not packed" >&2
+ ret=1
+ break
+ done
+ if [ "$ret" == "1" ]; then
+ break
+ fi
+ done
+
+ test_done "$testroot" "$ret"
+}
+
+test_pack_exclude() {
+ local testroot=`test_init pack_exclude`
+ local commit0=`git_show_head $testroot/repo`
+
+ # no pack files should exist yet
+ ls $testroot/repo/.git/objects/pack/ > $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+ echo -n > $testroot/stdout.expected
+ cmp -s $testroot/stdout.expected $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ diff -u $testroot/stdout.expected $testroot/stdout
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ got branch -r $testroot/repo mybranch
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ got checkout -b mybranch $testroot/repo $testroot/wt > /dev/null
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ echo a new line >> $testroot/wt/alpha
+ (cd $testroot/wt && got commit -m "edit alpha" >/dev/null)
+
+ gotadmin pack -r $testroot/repo -x master > $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ echo "gotadmin pack failed unexpectedly" >&2
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+ packname=`grep ^Wrote $testroot/stdout | cut -d ' ' -f2`
+ gotadmin listpack $testroot/repo/.git/objects/pack/pack-$packname \
+ > $testroot/stdout
+
+ tree0=`got cat -r $testroot/repo $commit0 | grep ^tree | \
+ cut -d ' ' -f2`
+ excluded_ids=`got tree -r $testroot/repo -c $commit0 -R -i | \
+ cut -d ' ' -f 1`
+ excluded_ids="$excluded_ids $commit0 $tree0"
+ for id in $excluded_ids; do
+ ret=0
+ if grep -q ^$id $testroot/stdout; then
+ echo "found excluded object $id in pack file" >&2
+ ret=1
+ fi
+ if [ "$ret" == "1" ]; then
+ break
+ fi
+ done
+ if [ "$ret" == "1" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ for d in $testroot/repo/.git/objects/[0-9a-f][0-9a-f]; do
+ id0=`basename $d`
+ ret=0
+ for e in `ls $d`; do
+ obj_id=${id0}${e}
+ excluded=0
+ for id in $excluded_ids; do
+ if [ "$obj_id" == "$id" ]; then
+ excluded=1
+ break
+ fi
+ done
+ if [ "$excluded" == "1" ]; then
+ continue
+ fi
+ if grep -q ^$obj_id $testroot/stdout; then
+ continue
+ fi
+ echo "loose object $obj_id was not packed" >&2
+ ret=1
+ break
+ done
+ if [ "$ret" == "1" ]; then
+ break
+ fi
+ done
+
+ test_done "$testroot" "$ret"
+}
+
+test_pack_include() {
+ local testroot=`test_init pack_include`
+ local commit0=`git_show_head $testroot/repo`
+
+ # no pack files should exist yet
+ ls $testroot/repo/.git/objects/pack/ > $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+ echo -n > $testroot/stdout.expected
+ cmp -s $testroot/stdout.expected $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ diff -u $testroot/stdout.expected $testroot/stdout
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ got branch -r $testroot/repo mybranch
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ got checkout -b mybranch $testroot/repo $testroot/wt > /dev/null
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ echo a new line >> $testroot/wt/alpha
+ (cd $testroot/wt && got commit -m "edit alpha" >/dev/null)
+ local commit1=`git_show_branch_head $testroot/repo mybranch`
+
+ gotadmin pack -r $testroot/repo master > $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ echo "gotadmin pack failed unexpectedly" >&2
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+ packname=`grep ^Wrote $testroot/stdout | cut -d ' ' -f2`
+ gotadmin listpack $testroot/repo/.git/objects/pack/pack-$packname \
+ > $testroot/stdout
+
+ tree1=`got cat -r $testroot/repo $commit1 | grep ^tree | \
+ cut -d ' ' -f2`
+ alpha1=`got tree -r $testroot/repo -i -c $commit1 | \
+ grep "[0-9a-f] alpha$" | cut -d' ' -f 1`
+ excluded_ids="$alpha1 $commit1 $tree1"
+ for id in $excluded_ids; do
+ ret=0
+ if grep -q ^$id $testroot/stdout; then
+ echo "found excluded object $id in pack file" >&2
+ ret=1
+ fi
+ if [ "$ret" == "1" ]; then
+ break
+ fi
+ done
+ if [ "$ret" == "1" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ tree0=`got cat -r $testroot/repo $commit0 | grep ^tree | \
+ cut -d ' ' -f2`
+ included_ids=`got tree -r $testroot/repo -c $commit0 -R -i | \
+ cut -d ' ' -f 1`
+ included_ids="$included_ids $commit0 $tree0"
+ for obj_id in $included_ids; do
+ for id in $excluded_ids; do
+ if [ "$obj_id" == "$id" ]; then
+ excluded=1
+ break
+ fi
+ done
+ if [ "$excluded" == "1" ]; then
+ continue
+ fi
+ if grep -q ^$obj_id $testroot/stdout; then
+ continue
+ fi
+ echo "included object $obj_id was not packed" >&2
+ ret=1
+ break
+ done
+
+ test_done "$testroot" "$ret"
+}
+
+test_pack_ambiguous_arg() {
+ local testroot=`test_init pack_ambiguous_arg`
+ local commit0=`git_show_head $testroot/repo`
+
+ # no pack files should exist yet
+ ls $testroot/repo/.git/objects/pack/ > $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+ echo -n > $testroot/stdout.expected
+ cmp -s $testroot/stdout.expected $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ diff -u $testroot/stdout.expected $testroot/stdout
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ got branch -r $testroot/repo mybranch
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ got checkout -b mybranch $testroot/repo $testroot/wt > /dev/null
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ echo a new line >> $testroot/wt/alpha
+ (cd $testroot/wt && got commit -m "edit alpha" >/dev/null)
+ local commit1=`git_show_branch_head $testroot/repo mybranch`
+
+ gotadmin pack -r $testroot/repo -x master master \
+ > $testroot/stdout 2> $testroot/stderr
+ ret="$?"
+ if [ "$ret" == "0" ]; then
+ echo "gotadmin pack succeeded unexpectedly" >&2
+ test_done "$testroot" "1"
+ return 1
+ fi
+
+ printf "\rpacking 1 reference\n" > $testroot/stdout.expected
+ cmp -s $testroot/stdout.expected $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ diff -u $testroot/stdout.expected $testroot/stdout
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ echo "gotadmin: not enough objects to pack" > $testroot/stderr.expected
+ cmp -s $testroot/stderr.expected $testroot/stderr
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ diff -u $testroot/stderr.expected $testroot/stderr
+ fi
+ test_done "$testroot" "$ret"
+}
+
+test_pack_loose_only() {
+ local testroot=`test_init pack_loose_only`
+ local commit0=`git_show_head $testroot/repo`
+
+ # no pack files should exist yet
+ ls $testroot/repo/.git/objects/pack/ > $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+ echo -n > $testroot/stdout.expected
+ cmp -s $testroot/stdout.expected $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ diff -u $testroot/stdout.expected $testroot/stdout
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ got branch -r $testroot/repo mybranch
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ got checkout -b mybranch $testroot/repo $testroot/wt > /dev/null
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ echo a new line >> $testroot/wt/alpha
+ (cd $testroot/wt && got commit -m "edit alpha" >/dev/null)
+
+ # pack objects belonging to the 'master' branch; its objects
+ # should then be excluded while packing 'mybranch' since they
+ # are already packed
+ gotadmin pack -r $testroot/repo master > /dev/null
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ echo "gotadmin pack failed unexpectedly" >&2
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ gotadmin pack -r $testroot/repo mybranch > $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ echo "gotadmin pack failed unexpectedly" >&2
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+ packname=`grep ^Wrote $testroot/stdout | cut -d ' ' -f2`
+ gotadmin listpack $testroot/repo/.git/objects/pack/pack-$packname \
+ > $testroot/stdout
+
+ tree0=`got cat -r $testroot/repo $commit0 | grep ^tree | \
+ cut -d ' ' -f2`
+ excluded_ids=`got tree -r $testroot/repo -c $commit0 -R -i | \
+ cut -d ' ' -f 1`
+ excluded_ids="$excluded_ids $commit0 $tree0"
+ for id in $excluded_ids; do
+ ret=0
+ if grep -q ^$id $testroot/stdout; then
+ echo "found excluded object $id in pack file" >&2
+ ret=1
+ fi
+ if [ "$ret" == "1" ]; then
+ break
+ fi
+ done
+ if [ "$ret" == "1" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ for d in $testroot/repo/.git/objects/[0-9a-f][0-9a-f]; do
+ id0=`basename $d`
+ ret=0
+ for e in `ls $d`; do
+ obj_id=${id0}${e}
+ excluded=0
+ for id in $excluded_ids; do
+ if [ "$obj_id" == "$id" ]; then
+ excluded=1
+ break
+ fi
+ done
+ if [ "$excluded" == "1" ]; then
+ continue
+ fi
+ if grep -q ^$obj_id $testroot/stdout; then
+ continue
+ fi
+ echo "loose object $obj_id was not packed" >&2
+ ret=1
+ break
+ done
+ if [ "$ret" == "1" ]; then
+ break
+ fi
+ done
+
+ test_done "$testroot" "$ret"
+}
+
+test_pack_all_objects() {
+ local testroot=`test_init pack_all_objects`
+ local commit0=`git_show_head $testroot/repo`
+
+ # no pack files should exist yet
+ ls $testroot/repo/.git/objects/pack/ > $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+ echo -n > $testroot/stdout.expected
+ cmp -s $testroot/stdout.expected $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ diff -u $testroot/stdout.expected $testroot/stdout
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ got branch -r $testroot/repo mybranch
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ got checkout -b mybranch $testroot/repo $testroot/wt > /dev/null
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ echo a new line >> $testroot/wt/alpha
+ (cd $testroot/wt && got commit -m "edit alpha" >/dev/null)
+
+ # pack objects belonging to the 'master' branch
+ gotadmin pack -r $testroot/repo master > /dev/null
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ echo "gotadmin pack failed unexpectedly" >&2
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ # pack mybranch, including already packed objects on the
+ # 'master' branch which are reachable from mybranch
+ gotadmin pack -r $testroot/repo -a mybranch > $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ echo "gotadmin pack failed unexpectedly" >&2
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+ packname=`grep ^Wrote $testroot/stdout | cut -d ' ' -f2`
+ gotadmin listpack $testroot/repo/.git/objects/pack/pack-$packname \
+ > $testroot/stdout
+
+ for d in $testroot/repo/.git/objects/[0-9a-f][0-9a-f]; do
+ id0=`basename $d`
+ ret=0
+ for e in `ls $d`; do
+ obj_id=${id0}${e}
+ if grep -q ^$obj_id $testroot/stdout; then
+ continue
+ fi
+ echo "loose object $obj_id was not packed" >&2
+ ret=1
+ break
+ done
+ if [ "$ret" == "1" ]; then
+ break
+ fi
+ done
+
+ test_done "$testroot" "$ret"
+}
+
+test_pack_bad_ref() {
+ local testroot=`test_init pack_bad_ref`
+ local commit0=`git_show_head $testroot/repo`
+
+ # no pack files should exist yet
+ ls $testroot/repo/.git/objects/pack/ > $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+ echo -n > $testroot/stdout.expected
+ cmp -s $testroot/stdout.expected $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ diff -u $testroot/stdout.expected $testroot/stdout
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ got branch -r $testroot/repo mybranch
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ got checkout -b mybranch $testroot/repo $testroot/wt > /dev/null
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ gotadmin pack -r $testroot/repo refs/got/worktree/ \
+ > $testroot/stdout 2> $testroot/stderr
+ ret="$?"
+ if [ "$ret" == "0" ]; then
+ echo "gotadmin pack succeeded unexpectedly" >&2
+ test_done "$testroot" "1"
+ return 1
+ fi
+
+ echo -n > $testroot/stdout.expected
+ cmp -s $testroot/stdout.expected $testroot/stdout
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ diff -u $testroot/stdout.expected $testroot/stdout
+ test_done "$testroot" "$ret"
+ return 1
+ fi
+
+ echo "gotadmin: not enough objects to pack" > $testroot/stderr.expected
+ cmp -s $testroot/stderr.expected $testroot/stderr
+ ret="$?"
+ if [ "$ret" != "0" ]; then
+ diff -u $testroot/stderr.expected $testroot/stderr
+ fi
+ test_done "$testroot" "$ret"
+}
+
+test_parseargs "$@"
+run_test test_pack_all_loose_objects
+run_test test_pack_exclude
+run_test test_pack_include
+run_test test_pack_ambiguous_arg
+run_test test_pack_loose_only
+run_test test_pack_all_objects
+run_test test_pack_bad_ref
blob - 5fa89910676821b58b5c98fe4828ced3e50f3e97
blob + 872d88379087aa49d66ebbc780ae19c92d4637e9
--- regress/deltify/deltify_test.c
+++ regress/deltify/deltify_test.c
@@ -85,7 +85,8 @@ deltify_abc_axc(void)
}
err = got_deltify(&deltas, &ndeltas, derived_file, 0,
- 3 * GOT_DELTIFY_MAXCHUNK, dt, base_file, 3 * GOT_DELTIFY_MAXCHUNK);
+ 3 * GOT_DELTIFY_MAXCHUNK, dt, base_file, 0,
+ 3 * GOT_DELTIFY_MAXCHUNK);
if (err)
goto done;
creating pack files