The latency with VNC is very little. The main bottleneck is the low refresh rate and the lag of my old E-ink tablet. This can be a much better experience with a newer tablet with higher refresh rate.
What refresh would we need to hit with e-ink monitors for this to be a viable alternative?