LISA, the space-based gravitational wave detector, is due to fly in the mid 2030s. An entire new frequency range will be opened up for discovering gravitational wave sources, including intermediate and extreme mass ratio black hole binaries which will remain in band for up to weeks, months or even years. This offers an exciting new avenue for fundamental physics discoveries because the environment of the binaries will have an effect on the gravitational waveform over this long period of time, and we will be able to measure the properties of the environments from the gravitational wave observations alone. I will show that we can measure the parameters of not only baryonic environments such as accretion disks, but also the properties of dark matter spikes or clouds of scalar fields if they are present around the binaries. I will demonstrate that we can distinguish between different environments with a Bayesian model comparison approach and argue the importance of including environmental effects in waveform modelling. This is so that we don't miss the opportunity to learn about the nature of dark matter or the structure of accretion disks, but crucially also so that we don't infer biased parameters by assuming that the system is inspiralling through vacuum.