How to customize or make new widgets?

Hello, this is a quite general question, but the answer may be quite useful to most people.

How can I customize or make new widgets, so that they can be added to different sidebars on different posts or pages? I am using a plugin to create many more sidebars, and the content on these sidebars can be anything I like. The default widgets are quite limited. In order to add a new functionality to a sidebar, my questions are:
1) Is the sidebar the only or best place to host the new functionality?
2) Is the widget the only way to add new functionality to sidebar?
3) How can this process can be easier? For instance, I　need to extract some data from web and then fill it in a table which will be displayed on a sidebar? Any existing plugins to help with this?

Any advice, suggestion, or pointers to resources are highly appreciated!