In fact, the Prismic toolbar script is added by you in your applications (and so you can remove it when it is not necessary) knowing that it is optimized based on the context, and so some modules don't load if not required.
As far as I remember, the Prismic prismic.js is not only used to support the toolbar and previews, but shareable links as well where public users can preview a draft document even if they aren't Prismic users