One I enjoy for swimming is the Plage d'Arene Grosse. Actually, it's just beyond that beach, immediately after the Port of Boulouris as you head east.
In general, it's far less crowded than the better-known public beaches nearby such as the beaches of Saint-Raphael, Frejus or even Agay. Few visitors know about it, unless they're staying in Boulouris right next door.
Because the beach faces the open sea rather than a sheltered cove, the water is usually very clean. You can check the current water quality status at the entrance by the port of Boulouris as water samples are taken and analysed regularly all summer long.
The other thing that may keep some beach-lovers out are the rocks. Most of the beach is sandy, but under the sand lays layers of rocks stretched flat. They create a gentle surf as the sea breaks on the rocks.
While they're not sharp, you'll probably be more confortable wearing water shoes. Personally, I wear fins as I love to swim beyond the reef to the open waters.
Where is the beach?
If you don't have a car, it's a walk from St Raphael to reach it, though a very pleasant one if you decide to take the coastal path instead of the main road. That probably keeps many folks away.
Interested in local hikes on the Western side of the French Riviera from Cannes to Hyeres? Take a look at this hiking guide available on Amazon: