Stumbled upon this useful article: http://www.mfoot.com/blog/2012/01/23/choosing-a-maths-library-for-cross-platform-c-game-development/
I've been using OpenCV for generic linear algebra, and then some others for fixed size 3D matrices and vectors. However, I've grown tired of OpenCV's quirks, especially the difficulty of debugging due to its internals, where void pointers are passed around and a custom RTTI system tries to make sense of the data (legacy of the first OpenCV versions that were C only). It seems to be fairly easy to cause an exception for which one doesn't even get a stack trace. Maybe it's time to switch to Eigen, which is apparently equally useful for large and small matrices, and also performs faster than GLM.
Edit May 27: Eigen definitely seems to work well and is easy to use. Creating math-heavy applications in C++ just got so much easier.
Edit June 12: I'm a full Eigen convert now, preaching to everybody who cares to listen =) Many matrix size and alignment errors are catched at compile time, and the API is flexible - for example, one can declare a matrix m and then say, e.g., Vector3f v=m.rowwise().sum(). More in this API showcase.