Baviaanskloof is not a place to traverse only. I would recommend a two nights stay. Although the Willowmore appears more attractive and offers a variety of smaller and bigger valleys within the Kloof the Cambria area offers good 4x4 side trips and Patensie has more infrastructure.
The main attraction is the Baviaanskloof trail area which in hindsight is fairly easy to drive and which offers several spectacular views. Due to many drifts and mainly green vegetation it is a bit more varied than "The Hell" between Oudshoorn and Prince Albert.
Apart from beautiful views you can do hiking as well as kloofing. You could stay on a bigger farm or self-catering cottage or camp and will be rewarded with fresh air, very little noise, a very starry night sky and get a good nights sleep. Without cellphone reception you can ignore the world outside and totally relax.
If in a hurry it would be possible to do a round trip from any place between Knysna and Jeffrey's Bay in a day but than you would want to return to Baviaanskloof to linger. To capture the beautiful impressions you get in morning and afternoon light require at least one overnight stay.