It is safe to assume that this is some kind of caching gone wrong. I reverted the settings only to experience the slow loading folder contents once again. The folder contents where displayed right away without loading time.
If you are using Windows 8 or Windows 10, do the following instead:
Select Organize from the top menu and select Properties from the context menu that opens up.Open Windows Explorer and browse to the folder that you experience the slow loading times in.