The client framework would be much more light weight when only the widgets needed are loaded. This would be a big improvement for especially devices with less resources.
One solution would be to create a widget bundle per user role, which includes only the widgets on pages reachable for that user role. For example, when an anonymous user only needs some basics to reset a password; it only loads the dataview widget in the bundle; while the functional administrator also gets charting widgets and rich text editor widgets in the bundle.
Another solution would be to load the widget's resource when needed the first time. For example, the charting widget is loaded when you go to a dashboard page.