The drawbacks of bidets are minor enough to pass unnoticed by the masses, so that should give you an idea of how convenient they are. That said, there are drawbacks to them and they can cause you temporary setbacks.
For a first-time user, a bidet is a foreign experience and it can be tricky for them to switch from toilet paper to bidets completely. Some people will require a brief adjustment period with bidets before they get accustomed to them.
As for the other drawback, you may see concerns of bacterial infections when an unclean bidet seat or attachment is involved. The nozzle or shower head can become a home for bacteria and not cleaning the bidet for a long time can result in contamination.