I will present recent results on characterizing the infrared spectral energy distributions (SEDs) of 191 mid-IR selected z~0.3-3.0 and LIR~1011-1013Lsun galaxies and AGN, and study how their SEDs differ from those of local and high-z analogs. Infrared SEDs depend both on the power source (AGN or star-formation) and the dust distribution. Therefore, differences in the SEDs of high-z and local galaxies provide clues as to differences in their physical conditions. I will show that there is strong evolution in the SEDs between local and z~2 IR-luminous galaxies, as well as that there is a wide range of SEDs among high redshift IR-luminous sources. Lastly, I will discuss possible explanations for this SED evolution as revealed in the morphology (based on HST NICMOS images) of our sources, as well as based on theoretical SED models (based on GADGET+SUNRISE simulations).