Virtual time travel from existing remains of a heritage site to its previous states and original condition is an educational and interesting experience and can provide better understanding of history. However, digitally reconstructing non-existing objects is a challenge. The interaction and navigation within virtual 4D worlds (adding time to 3D worlds) is also problematical due to the time dimension. In this paper we developed an approach to modelling of heritage sites that has undergone changes over the years. The method creates independent models from different types of data, such as frescos and paintings, drawings, old photos, historic descriptions, and digitization of remains, then assembles and integrates these models for an interactive presentation. Several research issues had to be addressed: (1) Modelling from frescos and drawings with incorrect perspective, (2) modelling from paintings and old photos including fine geometric details from shading (3) colouring models from old photos and drawings to match the colours of existing elements, (4) the seamless and accurate integration of models created independently from different sets of data, and (5) the creation of intuitive interactive presentation that combines all the models and other useful information. We provide contributions to these issues, including our own advanced model viewer, and apply them to modelling of: destroyed Haida house of Chief Weah (Masset, Canada), the demolished and partially relocated Rideau Chapel (Ottawa, Canada), and the Stenico castle (Trentino, Italy) which undergone many changes over several centuries. Each of these diverse examples illustrates different approach for reconstructing heritage sites that changed through time.