GB and TB-class Metadata Distribution Options

In addition to the metadata distribution options I've outlined in other child pages, I wanted to also note a couple of potential GB and TB-class distribution options, if metadata volumes end up growing from the single digit MB level to gigabyte or terabyte levels:

One option would be to use NNTP (http://en.wikipedia.org/wiki/Network_News_Transfer_Protocol), as underlays what's normally referred to as Usenet News (see https://en.wikipedia.org/wiki/Usenet). Approximately ten years ago, I worked with Ann Wilson of UNIDATA (at NCAR/UCAR) to trial distributing high volume weather data via NNTP. It worked quite well. See for example: http://www.unidata.ucar.edu/Presentations/UPCsemseries/nldm.pps

Another option to consider for GB or TB class metadata distribution might be a BitTorrent-based solution (see http://en.wikipedia.org/wiki/BitTorrent ).

Given the decentralized nature of these file distribution protocols, content distributed via these channels, to be trustworthy, will obviously need to be cryptographically signed in a verifiable way.

Another challenge to be aware of, assuming a flooding-based solution (ala NNTP) is deployed, is that reliable delivery isn't guaranteed, although in practice NNTP (with sufficient feed redundancy) can be quite efficient and complete, see for example Unidata's report on delivery statistics at http://www.unidata.ucar.edu/projects/nldm/

My belief, however, is that we don't/won't need GB or TB class metadata distribution options for the forseeable future.