From: Stefan Sperling Subject: Re: optimise reading the file index To: "Todd C. Miller" Cc: Christian Weisgerber , gameoftrees@openbsd.org Date: Thu, 27 Jul 2023 18:13:00 +0200 On Thu, Jul 27, 2023 at 09:07:13AM -0600, Todd C. Miller wrote: > On Thu, 27 Jul 2023 15:19:47 +0200, Christian Weisgerber wrote: > > > The current code uses fread(3), i.e., buffered stdio, which translates > > to a long series of 16 kB read(2) system calls. > > stdio uses the "optimal" buffer size reported for the device, > st_blksize from struct stat. The buffer size can be modified via > a call to setvbuf(). The defaults were created when memory was > more dear so you may be able to improve things by just bumping the > buffer size. > > > For a got status of /usr/src, 786 read(2) calls look rather negligible > > compared to 92,300+ stat(2) calls. > > Yikes, that is a lot. It seems to be about one stat call per file which can't get much better? $ find /usr/src | wc -l 99998 In any case, this fileindex optimization isn't about making 'got status' run faster. We noticed a visible delay when starting up tog, where the newly added base-commit marker takes a short time to appear on some systems. I determined that the time was spent reading the file index, which prompted Mark to write this patch.