Whilst working out a problem with how to return values from a SharePoint Modal Dialog I found an interesting link to a page in the his book online explaining the works.
Firstly here is how values are returned from a SPModalDialog to its calling page
http://www.learningsharepoint.com/2010/07/22/programming-with-dialog-in-sharepoint-2010/
and here is the page with stuff on the javascript functions within the popup frame
link
You can find the book here