[Custom bot] Servos as robot wheel motor

I am building a robot and need to be able to controll the speed and direction of the wheels very detailed. I have experiemnted some with GPIOZERO and motor and robot, but I cannot seem to get a good controll of the speed and with the robot library it is only limited to left, right, back and forward.

I started to think about using servos as the wheel motors instead. I tried using GPIOZERO with Servo and AngleServo libraries, but it seems the accuracy is off when controlling the servos.

Anyone having some experience thoughts on this? Should I continue to explore the servos or is there a better way with servo motors? Should I go for a servo controller like the Adafruit servo HAT?