lethjakman: idk details, but something along the lines of everytime you move to a different line (or even same line? idk) it has to redraw a lot of the screen.
lethjakman: linum in general has this problem though, so if you find it acceptable maybe relative won't be that bad. nlinum fixes this problem of linum but can't do relative numbers.
lethjakman: there are two relative linum packages on melpa. idk if either have the format you want. I made https://github.com/scottjad/linum-relativenumber
but it displays absolute and relative for all lines.