I'm not quite sure how this will behave on windows, and we can't really
create a tempdir (as we don't want to mess with path on home.
One possibility would be to mock/monkeypatch scandir, listdir and
os.path.expanduser or set $HOME in env; but I'm quite unsure we want to
go that route.