From: Mark Jamsek Subject: Re: optimise reading the file index To: Stefan Sperling Cc: "Todd C. Miller" , Christian Weisgerber , gameoftrees@openbsd.org Date: Sun, 30 Jul 2023 13:42:26 +1000 Stefan Sperling wrote: > On Fri, Jul 28, 2023 at 03:14:43PM +1000, Mark Jamsek wrote: > > As op taught me on irc, 'got info ' is good for this case, and > > while we're seeing some improvement there, it doesn't seem to be showing > > any improvement in the time it takes to draw the base commit marker in > > tog. However, it is also making the new 'got branch -l' about 30% faster > > and that takes the base commit of each file in the index into account > > the same way tog's new base commit marker does, so there could be some > > other reason in tog for the delay. > > Right. Then let's use branch -l as the benchmark and forget about tog for now. > It's not even very important to fix the commit-marker delay in tog, just > something I noticed. And it seems Omar has now figured out why the routine > which is reading the file index is so slow (tons of realloc, making the > hundreds of fread calls a secondary issue). Yes, as mentioned in the OP, those multiple small reallocs when building the entry’s path were the motivation behind the original diff, but from our chat on irc about not assuming a fixed size path buffer, I opted to memory map the file, which had the added benefit of mitigating the repetitive small reads too. However, as you point out in another thread, paths longer than PATH_MAX would be most impractical, so this yields much of the performance gain with a net code reduction :) > Overall, I am very happy with the progress that's being made on this :) Likewise! While I don’t really have any issues with got's performance on my new daily driver, I sometimes take my x1 when out and about and that is much slower! Besides which, I think many small optimisations will eventually add up in this area so if we can obtain them without complicating things it will be worthwhile. And from chatting with op off-list, he’s got a good plan for further related optimisations that I'm really looking forward to; I think it'll be great! btw I’ve got an idea as to why tog is a little delayed, which I’ll investigate tonight; but, I agree, it’s really not a big deal. -- Mark Jamsek GPG: F2FF 13DE 6A06 C471 CA80 E6E2 2930 DC66 86EE CF68