Usenet Provider vs. Usenet Indexer Explained
Everyday I read new posts on a few forums I lurk on about how bad an index is because the user cannot download the newest release of their favorite televsion program. Typically, this does not have anything to do with the Usenet Indexer at all but everything to do with your Usenet service provider and both their retention and DMCA takedown policies. When a release is posted to an Usenet newsgroup by a scene group, two different processes take place. First, it is immediately available as a binary download. These binaries are stored on your Usenet providers servers and depending upon their retention will be available for years to come. Next, the Usenet index retrieves the latest headers or parts from the newsgroups they index and log them. They will recieve all the binary parts associated with a release to ensure it is uploaded completely. At that point a NZB file is generated and stored within its library for users to download and use. If your index has the NZB and only displays completed releases, then at the time the index updated that group it successfully fetched all of the binaries associated with the release. When you use that NZB and try to download the release, if you get corrupted binaries, this post is out of your servers retention, it is because it was likely DMCA'd by the copyright holder and your provider made the binaries unavailable. This process has absolutely nothing to do with your indexer. Specifically, about our indexer we scan all of our newsgroups for changes every 10-35 seconds. Coupled with the fact that we utilize two 1Gbps connections and are just a few hops from our primary provider means we get releases extremely fast. Typically, we get releases faster than the automatic DMCA tools get them to remove them. Astraweb is notorious for having tons of takedowns. It had become an unstable provider for newer releases, but is still an absolutely great provider for older releases and as a backup. Many users feel as though they didn't have these issues with NZBmatrix and Newzbin. This is true. I am not saying it didn't happen, it just didn't happen as often. Those providers both shutdown because of varying circumstsnces. Both believed that too much material was being DMCA'd, leaving very little to index. Without content to index, their model of business and service offering became irrelevant to them. There is still plenty of room for adequate indexing services and we all have what we do better than the others. So, next time you find yourself wishing to blame your indexer for the issues you experience while downloading, remember it is only one piece of the puzzle.
