Everyone is panicking and banning it... why?

Bad reputation

Like all applications (Webex, Skype, etc.) Zoom also has bugs, which it's now scrambling to fix as fast as possible — see the plan published on their blog.

Facebook data sharing

This issue has already been resolved and only affected iOS users (Apple iPhone/iPad). Data sharing is a big topic right now, but this isn't the first or last app that's been caught doing it — see Facebook Off.

Zoombombing

This problem is a configuration issue. If a video conference is insufficiently secured (no password, no access control) this can happen. In my opinion this is not primarily a Zoom problem but a user problem — people who can't set it up properly and assume the default settings are secure (which turns out to be a wrong assumption — and this applies generally, not just to Zoom).

Encryption

Zoom, like other platforms, uses encryption. The problem is that Zoom was marketing end-to-end encryption but that wasn't true.
Like others (e.g., Skype or Webex) it uses TLS — the same protocol responsible for securing the transport layer of HTTPS.
Using end-to-end encryption typically limits what the service can do (Google Hangouts/Duo don't support more than 12 participants, Webex won't allow the web app, recording, etc.).
In my opinion there's nothing to worry about if you're using the latest version of the app and following the recommended settings.