Two entries, an hour of city between them, and only one of the two have I actually stood in.
The morning is remembered. The afternoon is only intended.
The Nezu is the one I can speak for. Go at opening, in April if you can arrange it — the approach under the bamboo is a deliberate piece of choreography and it only works when there is nobody ahead of you on it. The crowd, when it arrives, is exemplary; it is the quietest full building I have been in.
Seikado Bunko I have read about far more than is respectable for somebody who has never walked through the door. It shows in winter, the bowl is not always out, and I have twice been in this city at the wrong time of year. That is the whole of my authority on it.
These two are not a pairing so much as two separate mornings that happen to share a city. Do not try to hinge them on one afternoon; Tokyo will spend the difference on transit and give you nothing back for it.