Actually the reason seems to be, that when scrolling down the address bar at the top disappears. Then seemingly randomly the scrolling readjustment happens. Now that I turned off this behavior in Firefox, things seem just fine. It also never happened to me when scrolling from bottom to top - only from top to bottom.
I got my address/navigation bar at the top.