You could build one hell of a nice site without ever hosting these libraries on your server. But you shouldn’t.
Why? Put simply, it’s bad form to put your trust in an unknown entity. And while I’m sure the creator(s) of jsDelivr are the nicest, most trustworthy people in the world, outside of a personal Twitter account link buried in the footer, there’s practically no info about them. That doesn’t instill a lot of trust in the product. So for me, the trade off just isn’t worth it. At least not right now.
Honestly, if I were someone who was considering jsDelivr, I’d probably just quit being cheap and pay for a proper CDN account somewhere. Especially if you’re getting to the point where you’re worried about your site’s performance.