I’v had a similar issue — the add image dialog would load as a normal page instead of in a overlay/popup.
This was caused by a javascript error in a plugin called “HeadSpace 2”.
You can use the Web-Inspector in Safari/Chrome or Firebug in Firefox to find these javascript errors, and then disable the offending plugin.