Solr 8.6.2 - download full data

[Anuj Bhargava]

I need to download the data in solr on my laptop. The data has around
950 records

The following command shows the result in the browser. As the file is very
large I want to download it.

I tried with 1 records. Need to download all
http://xx.xxx.xxx.xxx:8983/solr/data_2019/select?q=*%3A*&rows=1&wt=csv

Regards,

Anuj

Re: Solr 8.6.2 - download full data

[Jan Høydahl]

Try the export command

https://solr.apache.org/guide/8_6/solr-control-script-reference.html#exporting-documents-to-a-file

Jan

> 25. feb. 2022 kl. 15:19 skrev Anuj Bhargava :
> 
> I need to download the data in solr on my laptop. The data has around
> 950 records
> 
> The following command shows the result in the browser. As the file is very
> large I want to download it.
> 
> I tried with 1 records. Need to download all
> http://xx.xxx.xxx.xxx:8983/solr/data_2019/select?q=*%3A*&rows=1&wt=csv
> 
> Regards,
> 
> Anuj

Re: Solr 8.6.2 - download full data

[Andy Lester]

> On Feb 25, 2022, at 8:19 AM, Anuj Bhargava  wrote:
> 
> I tried with 1 records. Need to download all
> http://xx.xxx.xxx.xxx:8983/solr/data_2019/select?q=*%3A*&rows=1&wt=csv 
> 
Use a command-line utility like curl or wget to pull it down.

Re: Solr 8.6.2 - download full data

[Eric Pugh]

https://github.com/o19s/solr_dump

> On Feb 25, 2022, at 11:15 AM, Andy Lester  wrote:
> 
> 
> 
>> On Feb 25, 2022, at 8:19 AM, Anuj Bhargava  wrote:
>> 
>> I tried with 1 records. Need to download all
>> http://xx.xxx.xxx.xxx:8983/solr/data_2019/select?q=*%3A*&rows=1&wt=csv 
>> 
> Use a command-line utility like curl or wget to pull it down.

___
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
http://www.opensourceconnections.com  | 
My Free/Busy   
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed 


This e-mail and all contents, including attachments, is considered to be 
Company Confidential unless explicitly stated otherwise, regardless of whether 
attachments are marked as such.

Re: Solr 8.6.2 - download full data

[Anuj Bhargava]

Tried the following
wget -O 2019.csv
http://xx.xxx.xxx.xxx:8983/solr/data_2019/select?q=*%3A*&rows=2&wt=csv

It is not saving the total rows specified but just a few - saved [9042/9042]

And the result is not in csv format
{
  "responseHeader":{
"status":0,
"QTime":0,
"params":{
  "q":"*:*"}},
  "response":{"numFound":8634628,"start":0,"numFoundExact":true,"docs":[
  {
"country":["IN"],
"date_c":"2019-01-01T18:30:00Z",
"cpv":["29124000"],
"_version_":1693668431515090947},
  {
"country":["PT"],
"date_c":"2018-03-14T18:30:00Z",

On Fri, 25 Feb 2022 at 21:46, Andy Lester  wrote:

>
>
> > On Feb 25, 2022, at 8:19 AM, Anuj Bhargava  wrote:
> >
> > I tried with 1 records. Need to download all
> >
> http://xx.xxx.xxx.xxx:8983/solr/data_2019/select?q=*%3A*&rows=1&wt=csv
> <
> http://xx.xxx.xxx.xxx:8983/solr/data_2019/select?q=*%3A*&rows=1&wt=csv
> >
> Use a command-line utility like curl or wget to pull it down.

Re: Solr 8.6.2 - download full data

[Shawn Heisey]

On 2/25/2022 11:36 PM, Anuj Bhargava wrote:

Tried the following
wget -O 2019.csv
http://xx.xxx.xxx.xxx:8983/solr/data_2019/select?q=*%3A*&rows=2&wt=csv

It is not saving the total rows specified but just a few - saved [9042/9042]


Try putting the URL in quotes:

wget -O 2019.csv 
"http://xx.xxx.xxx.xxx:8983/solr/data_2019/select?q=*%3A*&rows=2&wt=csv";


It behaved that way because the & is a special character in a unix shell 
(it will cause what precedes the & to run in the background), and 
without the quotes, it will be handled by the shell, not sent to the 
wget command.


This is why whenever I am using curl or wget, I type the command, a 
space, then two quotes, and use the left arrow to put the cursor between 
the quotes, then I paste the URL in.


Thanks,
Shawn