If it is an Android TV I am afraid that the volume bar behaves by itself. The only way I have got to make it working as it must is connecting the Apple TV 4K straight to the TV. At that point the change in volume (from the TV remote) is smooth and synchronised. A bit slow, maybe. Connected to my AVR it jumps as it likes.
I have never had big volume steps casting anything to the TV, though. Although I don't understand the need to cast YouTube. If it's just to use the phone as a remote you may try the apps "Video & TV SideView" from Sony or Google's "Android TV" (stupid name for a stupid TV remote app...). They are both available for Android and iOS.
