I just wanted to tell you about a common, and as far as I know unavoidable, problem that occurs in all Linux distributions I have worked with on machines that have large memory. The RAM cache fills up and is not emptied properly.
The RAM cache is used extensively when large files are read and written, and for some reason the RAM cache gradually fills up due to file transfers and is not cleared very quickly. In some cases when another program requires memory the memory just isn’t there – you can see now on hulk no memory is being used by programs, yet over 100 Gb is being used by the cache, so only about half of the RAM is listed as available. In many cases this can use up so much RAM that the computer has to switch to using SWAP space, which happens regularly on the zoology cluster and all of our high-memory machines. I have seen this on RedHat and related, Ubuntu, and OpenSuse, among others. The only way I have found to get rid of this problem is to manually clear the cache, which only works on our machines as you need root access. Although this is not an ideal solution it hasn’t caused me problems so far. When using high-RAM programs I frequently clear the RAM cache to make sure the Free Memory stays at the max. To do that:
sync; echo 3 > /proc/sys/vm/drop_caches
Don’t worry if a file transfer is occurring, the sync command protects it. But, don’t alter these commands unless you are sure you know what you are doing. I have talked to many very smart people about this and nobody has come up with a better solution. Let me know, though, if any of you have better ideas.