>> Also: we recently contributed a new URIBL plugin to the Qpsmtpd >> project, >> which makes use of our "pruned TLD" lists that use your datafeed data. >> They had some question of whether this was something you would be >> comfortable with having distributed publicly. Note no actual URIBL data >> is distributed, just a list of TLDs that happens to *not* include >> top-level TLDs that would be extremely unlikely to generate hits against >> your service. This is used to limit the number of extraneous queries to >> your public mirrors, which I'm guessing you would consider beneficial. >> Could you verify whether we have your permission/blessing to distribute >> such a list gleaned from your data? >> > > Ya, fine. It doesnt sound like it would have significant impact on > volume to me, as the top 25 tlds (including ipv4 volume) that are > queried represent 91% of the total query volume, and the top 100 tld > represent 99% of the volume. > > If you tell me which tlds are suppressed, I can give you an idea of > query volume savings according to mirror traffic. > > For example, suppression of .mil and .int would result save 4/100th of a > percent (0.00039). Now, if there was a hacked webserver in .mil and > spammers used it as a drop page or redirector, our temporary listing of > it would never hit for you if you suppress them. I guess you have to > weigh the savings versus the potential for abuse. You wouldnt want to > supress a TLD that becomes the next spammer haven and have to scramble > to release an update. Thinking about recent history such as .tk > (Tokelau), .st (Sao Tome), .im (Isle of Man), and others.
There are 135 excluded tlds: ac ad af ag ai al an ao aq arpa asia aw ax bb bf bi bj bm bn bo bt bw cd cf cg ci ck coop cr cu cv dj dm do dz edu er et fj fk fm fo ga gf gh gi gl gm gn gov gp gq gt gu gw gy ht int iq jm jo jobs kh ki km kn kp kw ky lb lc lk lr ls lu ly mc mg mh mil ml mm mo mp mq mr mt museum mv mw mz na nc nf ng ni nr om pa pf pg pn ps pw qa re rw sb sd sh sl sm sn sr sv sy sz td tel tf tg tj tm tn travel ug va vg vi vu wf ye yu zm zw ... I also used the feed data to prune two and three level TLDs from the SURBL list, which is pretty obviously not based on any data: coop.br coop.tt gov.ae gov.am gov.ar gov.as gov.au gov.az gov.ba gov.bd gov.bh gov.bs gov.by gov.bz gov.ch gov.cl gov.cm gov.co gov.cx gov.cy gov.ec gov.ee gov.eg gov.ge gov.gg gov.gr gov.hk gov.hu gov.ie gov.il gov.im gov.in gov.io gov.ir gov.is gov.it gov.je gov.jp gov.kg gov.kz gov.la gov.li gov.lt gov.lv gov.ma gov.me gov.mk gov.mn gov.mu gov.my gov.np gov.ph gov.pk gov.pl gov.pr gov.pt gov.py gov.rs gov.ru gov.sa gov.sc gov.sg gov.sk gov.st gov.tl gov.to gov.tp gov.tt gov.tv gov.tw gov.ua gov.uk gov.vc gov.ve gov.vn gov.ws gov.za jobs.tt tel.no tel.tr act.gov.au nsw.gov.au nt.gov.au pa.gov.pl po.gov.pl qld.gov.au sa.gov.au so.gov.pl sr.gov.pl starostwo.gov.pl tas.gov.au ug.gov.pl um.gov.pl upow.gov.pl uw.gov.pl vic.gov.au wa.gov.au There were also about _1400_ reserved .us TLDs prune from the list, IIRC You make some pretty good points. It may well not be worth the trouble, at least for one-level TLDs. Thanks, Jared Johnson Software Developer DoubleCheck Email Manager
