Would be good to see your specs.Our systems sound very similar. I'm not experiencing any trouble. In fact,, I am amazed at how smooth everything is running using the 4k material.
I have found that scrubbing over a 5 minute shot in the source monitor and a five second shot on the timeline have always been two different things--1080p or 4k.
As scrubbing is very disk intensive, I'm wondering if I need to look at that first.