Clearing out unused tracks

I've googled this and found lots of posts but nothing that really addresses my issue. So I have about 10 tracks in use for a song I've been working on but in the tracks list on the left of the mix window there are probably 40-50 tracks there. I've tried downloading, adding them and then deleting them but they never go away. How can I clean up this list to only include what's current?