Script I used back in the day to do what you are looking for: #!/bin/bash for i in $(seq -f "%03g" 2 999) do wget http://downloads.digitalcorpora.org/corpora/files/govdocs1/zipfiles/$i.zip -O $i.zip unzip $i.zip rm $i.zip done
not sure if it still works On Tue, Jul 26, 2022 at 1:59 PM Tim Allison <talli...@apache.org> wrote: > As a warning, tho, Common Crawl truncates files at 1MB, so we have a bunch > of truncated files. We refetched some and put those under > commoncrawl3_refetched. > > On Tue, Jul 26, 2022 at 2:58 PM Tim Allison <talli...@apache.org> wrote: > >> We have ~1.9TB. But I'd skip cc_large because that's just a copy of some >> directories under commoncrawl3_refetched. >> >> If you want to pull fresher data out of CommonCrawl, I have undocumented >> scripts to do that. I could add documentation. >> >> These are the top 100 mime types and counts. This db was generated on a >> slightly earlier version of the corpus/corpora, but it should be close >> enough. >> >> MIME_STRING cnt >> application/pdf 768490 >> text/plain 472041 >> text/html 429707 >> application/x-tika-msoffice 297990 >> image/png 190815 >> application/octet-stream 190645 >> image/jpeg 179533 >> application/xhtml+xml 151830 >> application/x-bzip2 124204 >> application/x-tika-ooxml 122523 >> application/x-bzip 107435 >> application/xml 107003 >> application/zip 93467 >> application/x-sh 88712 >> application/gzip 73535 >> image/gif 66713 >> application/zlib 46483 >> text/calendar 40385 >> application/postscript 35526 >> application/rss+xml 34428 >> application/atom+xml 28950 >> multipart/appledouble 27602 >> image/svg+xml 25771 >> application/vnd.oasis.opendocument.text 25753 >> application/rdf+xml 24890 >> application/vnd.google-earth.kml+xml 24049 >> application/rtf 23915 >> application/x-matroska 19437 >> application/x-shockwave-flash 18879 >> video/quicktime 18546 >> application/epub+zip 18205 >> application/vnd.ms-excel 17465 >> application/x-xz 16869 >> text/x-vcard 16772 >> application/java-vm 16761 >> audio/mpeg 15534 >> message/rfc822 14405 >> application/vnd.oasis.opendocument.spreadsheet 12659 >> application/x-bibtex-text-file 12261 >> application/x-rar-compressed; version=4 12123 >> text/x-php 10870 >> text/x-diff 10080 >> video/mp4 8281 >> audio/mp4 8221 >> application/x-msdownload 8019 >> application/x-bittorrent 7964 >> image/vnd.microsoft.icon 7382 >> application/mbox 6799 >> application/x-x509-cert; format=der 6597 >> audio/vnd.wave 6550 >> image/bmp 6411 >> application/x-endnote-refer 5922 >> image/vnd.djvu 5874 >> text/x-matlab 5734 >> application/vnd.apple.mpegurl 5511 >> image/tiff 5430 >> image/webp 4972 >> application/vnd.oasis.opendocument.presentation 3989 >> text/x-jsp 3973 >> text/x-csrc 3555 >> video/x-ms-wmv 3453 >> video/x-m4v 3443 >> application/x-dbf 3381 >> text/x-chdr 3263 >> text/x-perl 3124 >> application/x-rpm 3023 >> application/x-mobipocket-ebook 2726 >> audio/midi 2697 >> application/vnd.oasis.opendocument.graphics 2675 >> application/vnd.ms-excel.sheet.4 2591 >> application/x-font-ttf 2575 >> application/xspf+xml 2557 >> text/x-python 2416 >> audio/vorbis 2354 >> application/msword 2223 >> application/ogg 2222 >> application/x-gtar 2181 >> audio/x-mpegurl 2067 >> video/x-flv 1969 >> audio/x-ms-wma 1874 >> image/icns 1857 >> application/x-object 1823 >> application/x-7z-compressed 1795 >> application/x-msdownload; format=pe32 1784 >> application/x-debian-package 1700 >> application/x-mysql-table-definition 1669 >> image/vnd.dxf; format=ascii 1664 >> application/x-sqlite3 1606 >> application/x-berkeley-db; format=hash 1457 >> application/x-executable 1455 >> video/mpeg 1366 >> application/pkcs7-signature 1359 >> application/x-ms-asx 1266 >> image/vnd.zbrush.pcx 1247 >> image/vnd.dwg 1243 >> application/fits 1217 >> application/xslfo+xml 1206 >> application/x-sharedlib 1185 >> audio/prs.sid 1173 >> text/x-vcalendar 1156 >> >> >> >> >> On Tue, Jul 26, 2022 at 2:12 PM Oscar Rieken Jr < >> oscar.rieke...@cofense.com> wrote: >> >>> We were thinking something around 2TB of data with a good mix of excel, >>> images, pdfs, text and powerpoints. So I guess a mix of everything. >>> >>> >>> >>> *From: *Tim Allison <talli...@apache.org> >>> *Date: *Tuesday, July 26, 2022 at 9:19 AM >>> *To: *u...@tika.apache.org <u...@tika.apache.org> >>> *Cc: *Oscar Rieken Jr <oscar.rieke...@cofense.com>, >>> corpora-dev@tika.apache.org <corpora-dev@tika.apache.org> >>> *Subject: *Re: Datasets for testing large number of attachments >>> >>> External Email >>> >>> What Nick said... >>> >>> >>> >>> cc_large is a sample of some of the larger documents from >>> commoncrawl3_refetched. >>> >>> >>> >>> If you want to give your pipeline a workout, I also recommend using the >>> MockParser that is available in the tika-core tests jar. That allows you >>> to instrument an OOM and timeouts and system exits and all sorts of other >>> mayhem. Obv, don't put the tika-core tests jar on your class path in >>> production. See the files in >>> https://github.com/apache/tika/tree/main/tika-parsers/tika-parsers-standard/tika-parsers-standard-package/src/test/resources/test-documents/mock >>> for examples of how to trigger bad behavior with the MockParser. >>> >>> >>> >>> On the corpora, as Nick said, let us know what you want and we can help >>> you select files. >>> >>> >>> >>> Cheers, >>> >>> >>> >>> Tim >>> >>> >>> >>> >>> >>> On Tue, Jul 26, 2022 at 7:06 AM Nick Burch <apa...@gagravarr.org> wrote: >>> >>> On Mon, 25 Jul 2022, Oscar Rieken Jr via user wrote: >>> > I am currently trying to validate our Tika setup and was looking for a >>> > set of example data I could use >>> >>> If you want a small number of files of lots of different types, the test >>> files in the Tika source tree will work. Main set are in >>> tika-parsers/src/test/resources/test-documents/ >>> >>> If you want a very large number of files, then the Tika Corpora >>> collection >>> is a good source. We have a few different collections, including stuff >>> from common crawl, govdocs and bug trackers. If you can let us know what >>> sort of file types and how many, we can suggest the best corpora >>> collection >>> >>> Nick >>> >>>
