report CI test failures, stats, history

[justinmk]

Problem

Flakey tests are painful to track and it's difficult to correlate when a test started becoming unreliable. Also hard to correlate increases in CI time-cost.

Proposal

Automate storage of a json file with CI + test stats.
Display it here?

Or maybe this can be done in a branch on the main neovim repo, as a jupyter notebook or something like that.

[yochem]

Probably the easiest approach to this would be:

• In neovim/neovim have a script that generates the plots and tables (e.g. in markdown format)
• That script is called from the gendoc.lua script in neovim/neovim.github.io and gets build to a page /ci-stats or whatever.

Note: if going the jupyter notebook route, do note these can be exported to html. Otherwise pd.to_markdown() and plt.savefig('name.svg') are really useful.

[justinmk]

if going the jupyter notebook route, do note these can be exported to html.

yeah, also I think github can display them "natively", which is very nice and saves a lot of work.

That script is called from the gendoc.lua script in neovim/neovim.github.io and gets build to a page /ci-stats or whatever.

That reminds me, one reason we had neovim/doc was to avoid clobbering "real" repos with many GB of data. However, we can avoid blowing up the neovim/neovim.github.io repo size by (1) committing to a separate branch and (2) force-pushing to delete old history.

[yochem]

That reminds me, one reason we had neovim/doc was to avoid clobbering "real" repos with many GB of data. However, we can avoid blowing up the neovim/neovim.github.io repo size by (1) committing to a separate branch and (2) force-pushing to delete old history.

I don't see how this would be a problem if the files are never commited to the repo: current /doc/user/ pages are also generated on the fly in CI and deployed to GH pages. Not included in the repo.

EDIT: ah okay if you want to display the notebooks on GitHub then yes a separate branch might be a good idea.

[justinmk]

normally, or at least previously (with jekyll), html was committed to gh-pages branch. but that is apparently not required with the "CI-driven pages".

[yochem]

but that is apparently not required with the "CI-driven pages".

Yep, not required :) CI just zips the public/ folder Hugo spits out and deploys that to pages. Everything in public/ is uploaded and nothing is committed back to the source repo.

[justinmk]

Nice. I suppose we don't need to store the "data" anywhere if it can be regenerated. OTOH, regenerating CI history isn't really feasible because github does not keep CI logs (and maybe other things) around forever. So we probably need to store the data file in a branch somewhere.