Metadata Compression

One simple alternative to consider, if metadata file size is causing metadata distribution to become excessively slow, might be to compress the metadata before transfering it.

For example, the current InCommon metadata file is 7,139,737 octets in length

If compressed with bzip2 -9 (see http://en.wikipedia.org/wiki/Bzip2 ), the current file drops in size to 1,238,006 octets, just 17.3% of the size of the original file. Everything else being unchanged, I'd therefore expect the file transfer time to be proportionately less.

If compressed with xz -9 (see http://en.wikipedia.org/wiki/Xz ), the current file drops in size still further, to just 1,027,212 octets, just 14.4% of the size of the original file

While manual compression could easily be built into the process of preparing the metadata file, another option to consider might be mod_deflate, as discussed at
http://www.devside.net/articles/apache-performance-tuning , which would enable compression to be negotiated between client and the web server on the fly