Have alook on the Performance tab of task manager and post the 3 physical memory figures.
I'd be surprised if you have any 'free' memory. I'm running Vista x64 with 4GB also, currently I have 0MB free and 1.6GB cached. SuperFetch will fill all the RAM it can with data it thinks you will use - it learns which applications you use frequently and when you use them. If you open Outlook at 9am every morning Windows will try and make sure its in RAM before you need it.
It makes making assumpsions about performance difficult.
If you have no free or cached RAM then your running at the limit and could probably benifit from more RAM.