Re: Uninstall Solr 9.1

[Shawn Heisey]

On 1/13/23 01:05, Anuj Bhargava wrote:
> On Centos
>
> wget https://dlcdn.apache.org/solr/solr/9.1.0/solr-9.1.0.tgz

I accidentally replied directly to the sender to ask how they installed 
Solr, above is the direct reply I got.


The wget command does not install Solr.  It only downloads it.  You had 
to have done additional steps after the wget.


Thanks,
Shawn

Re: Uninstall Solr 9.1

[Anuj Bhargava]

Installed java - sudo dnf install java-11-openjdk




*$java -versionopenjdk version "11.0.17" 2022-10-18 LTSOpenJDK Runtime
Environment (Red_Hat-11.0.17.0.8-2.el7_9) (build 11.0.17+8-LTS)OpenJDK
64-Bit Server VM (Red_Hat-11.0.17.0.8-2.el7_9) (build 11.0.17+8-LTS, mixed
mode, sharing)*

Downloaded solr wget https://dlcdn.apache.org/solr/solr/9.1.0/solr-9.1.0.tgz

tar xzf solr-9.1.0.tgz solr-9.1.0/bin/install_solr_service.sh
--strip-components=2

sudo bash ./install_solr_service.sh solr-9.1.0.tgz



*$ service solr startWaiting up to 180 seconds to see Solr running on port
8983 [\]Started Solr server on port 8983 (pid=11859). Happy searching!*

On Fri, 13 Jan 2023 at 18:37, Shawn Heisey  wrote:

> On 1/13/23 01:05, Anuj Bhargava wrote:
>  > On Centos
>  >
>  > wget https://dlcdn.apache.org/solr/solr/9.1.0/solr-9.1.0.tgz
>
> I accidentally replied directly to the sender to ask how they installed
> Solr, above is the direct reply I got.
>
> The wget command does not install Solr.  It only downloads it.  You had
> to have done additional steps after the wget.
>
> Thanks,
> Shawn
>

Re: Uninstall Solr 9.1

[Shawn Heisey]

On 1/13/23 06:44, Anuj Bhargava wrote:

Installed java - sudo dnf install java-11-openjdk

*$java -versionopenjdk version "11.0.17" 2022-10-18 LTSOpenJDK Runtime
Environment (Red_Hat-11.0.17.0.8-2.el7_9) (build 11.0.17+8-LTS)OpenJDK
64-Bit Server VM (Red_Hat-11.0.17.0.8-2.el7_9) (build 11.0.17+8-LTS, mixed
mode, sharing)*

Downloaded solr wget https://dlcdn.apache.org/solr/solr/9.1.0/solr-9.1.0.tgz

tar xzf solr-9.1.0.tgz solr-9.1.0/bin/install_solr_service.sh
--strip-components=2

sudo bash ./install_solr_service.sh solr-9.1.0.tgz


To completely remove all traces from a service install with default options:

sudo rm -rf /opt/solr /opt/solr-9.1.0
sudo rm -rf /var/solr
sudo find /etc | grep solr | sudo xargs rm -f
sudo userdel -r solr

If you were upgrading to a newer version, I would suggest different 
steps.  I think I saw somewhere that you were wanting to install an 8.x 
version instead.  You would not be able to use existing index data from 
9.x in 8.x.


Thanks,
Shawn

Re: Uninstall Solr 9.1

[Anuj Bhargava]

Thanks. Got Solr 9.1 working

Regards,

Anuj

On Fri, 13 Jan 2023 at 19:32, Shawn Heisey  wrote:

> On 1/13/23 06:44, Anuj Bhargava wrote:
> > Installed java - sudo dnf install java-11-openjdk
> >
> > *$java -versionopenjdk version "11.0.17" 2022-10-18 LTSOpenJDK Runtime
> > Environment (Red_Hat-11.0.17.0.8-2.el7_9) (build 11.0.17+8-LTS)OpenJDK
> > 64-Bit Server VM (Red_Hat-11.0.17.0.8-2.el7_9) (build 11.0.17+8-LTS,
> mixed
> > mode, sharing)*
> >
> > Downloaded solr wget
> https://dlcdn.apache.org/solr/solr/9.1.0/solr-9.1.0.tgz
> >
> > tar xzf solr-9.1.0.tgz solr-9.1.0/bin/install_solr_service.sh
> > --strip-components=2
> >
> > sudo bash ./install_solr_service.sh solr-9.1.0.tgz
>
> To completely remove all traces from a service install with default
> options:
>
> sudo rm -rf /opt/solr /opt/solr-9.1.0
> sudo rm -rf /var/solr
> sudo find /etc | grep solr | sudo xargs rm -f
> sudo userdel -r solr
>
> If you were upgrading to a newer version, I would suggest different
> steps.  I think I saw somewhere that you were wanting to install an 8.x
> version instead.  You would not be able to use existing index data from
> 9.x in 8.x.
>
> Thanks,
> Shawn
>

Importing Data from MySql

[Anuj Bhargava]

How do I import data from MySql database in Solr 9.1

Earlier I was using DIH in version 8.6.2

Regards,

Anuj

Re: Importing Data from MySql

[Markus Jelsma]

Hello,

DIH was removed in Solr 9.x. You can either build your own importer, or use
the, now 3rd party, DIH [1] that is available as a package.

Regards,
Markus

[1] https://github.com/rohitbemax/dataimporthandler

Op vr 13 jan. 2023 om 14:47 schreef Anuj Bhargava :

> How do I import data from MySql database in Solr 9.1
>
> Earlier I was using DIH in version 8.6.2
>
> Regards,
>
> Anuj
>

Re: Importing Data from MySql

[Jan Høydahl]

I don't think the 3rd party DIH is maintained.

Other options are using other 3rd party frameworks like
ManifoldCF 

 or Apache NiFI 
 
which is a more visual approach.
Personally I'd write a small Python program.

When asking ChatGPT to sketch such a program, this is what I got :) 

Sure, here is a sample Python program that demonstrates how to import data from 
a MySQL database and index it into Solr using the SolrPy library:

import mysql.connector
from solr import Solr

# Connect to MySQL database
cnx = mysql.connector.connect(user='username',
  password='password',
  host='hostname',
  database='database_name')
cursor = cnx.cursor()

# Execute MySQL query
query = "SELECT * FROM table_name"
cursor.execute(query)

# Fetch all data
data = cursor.fetchall()

# Connect to Solr
solr = Solr('http://localhost:8983/solr/core_name')

# Index data into Solr
for row in data:
solr.add(row)

# Commit changes
solr.commit()

# Close database and Solr connection
cursor.close()
cnx.close()
solr.close()


I have not tested it, but with some massage you should be able to fly

Jan

> 13. jan. 2023 kl. 16:02 skrev Markus Jelsma :
> 
> Hello,
> 
> DIH was removed in Solr 9.x. You can either build your own importer, or use
> the, now 3rd party, DIH [1] that is available as a package.
> 
> Regards,
> Markus
> 
> [1] https://github.com/rohitbemax/dataimporthandler
> 
> Op vr 13 jan. 2023 om 14:47 schreef Anuj Bhargava :
> 
>> How do I import data from MySql database in Solr 9.1
>> 
>> Earlier I was using DIH in version 8.6.2
>> 
>> Regards,
>> 
>> Anuj
>> 

Re: Importing Data from MySql

[Dave]

Yeah, it’s trivial building your own indexer in any language that can read a 
db. Also I wouldn’t trust the dih on its own even when supported 

> On Jan 13, 2023, at 10:17 AM, Jan Høydahl  wrote:
> 
> I don't think the 3rd party DIH is maintained.
> 
> Other options are using other 3rd party frameworks like
> ManifoldCF 
> 
>  or Apache NiFI 
> 
>  which is a more visual approach.
> Personally I'd write a small Python program.
> 
> When asking ChatGPT to sketch such a program, this is what I got :) 
> 
> Sure, here is a sample Python program that demonstrates how to import data 
> from a MySQL database and index it into Solr using the SolrPy library:
> 
> import mysql.connector
> from solr import Solr
> 
> # Connect to MySQL database
> cnx = mysql.connector.connect(user='username',
>  password='password',
>  host='hostname',
>  database='database_name')
> cursor = cnx.cursor()
> 
> # Execute MySQL query
> query = "SELECT * FROM table_name"
> cursor.execute(query)
> 
> # Fetch all data
> data = cursor.fetchall()
> 
> # Connect to Solr
> solr = Solr('http://localhost:8983/solr/core_name')
> 
> # Index data into Solr
> for row in data:
>solr.add(row)
> 
> # Commit changes
> solr.commit()
> 
> # Close database and Solr connection
> cursor.close()
> cnx.close()
> solr.close()
> 
> 
> I have not tested it, but with some massage you should be able to fly
> 
> Jan
> 
>> 13. jan. 2023 kl. 16:02 skrev Markus Jelsma :
>> 
>> Hello,
>> 
>> DIH was removed in Solr 9.x. You can either build your own importer, or use
>> the, now 3rd party, DIH [1] that is available as a package.
>> 
>> Regards,
>> Markus
>> 
>> [1] https://github.com/rohitbemax/dataimporthandler
>> 
>>> Op vr 13 jan. 2023 om 14:47 schreef Anuj Bhargava :
>>> 
>>> How do I import data from MySql database in Solr 9.1
>>> 
>>> Earlier I was using DIH in version 8.6.2
>>> 
>>> Regards,
>>> 
>>> Anuj
>>> 
> 

Re: Logging correlation id

[Jan Høydahl]

Have you considered enabling tracing, (header trace-parent) and get that Id 
added to the logs?

Jan Høydahl

> 13. jan. 2023 kl. 02:42 skrev Shawn Heisey :
> 
> On 1/12/23 12:19, Srijan wrote:
>> I am looking to log additional info to my Solr log. Specifically, I am
>> looking to log a "correlation-id" provided by my client as part of the HTTP
>> header field "x-correlation-id" and put that in my log to relate various
>> client sessions and their actions with Solr. Is there a codeless way of
>> doing this? Has anyone done something similar?
> 
> If you make it a URL parameter instead of a header, Solr will log it 
> automatically.
> 
> Arbitrary URL parameters that do not match anything Solr expects tend to be 
> ignored, so it is a good way to log something that doesn't affect Solr.
> 
> If it must be a header, then you're going to need to write some custom code 
> as suggesteed by Markus.
> 
> Thanks,
> Shawn

Re: Importing Data from MySql

[Thomas Corthals]

Build an indexer that can send updates in batches. It'll be faster than
sending each document in a separate request.

Op vr 13 jan. 2023 om 16:41 schreef Dave :

> Yeah, it’s trivial building your own indexer in any language that can read
> a db. Also I wouldn’t trust the dih on its own even when supported
>
> > On Jan 13, 2023, at 10:17 AM, Jan Høydahl  wrote:
> >
> > I don't think the 3rd party DIH is maintained.
> >
> > Other options are using other 3rd party frameworks like
> > ManifoldCF <
> https://datafari.atlassian.net/wiki/spaces/DATAFARI/pages/2718007297/How+to+crawl+a+database+on+Solr+without+DIH+-+DIH+alternative>
> or Apache NiFI <
> https://www.datainmotion.dev/2020/04/building-search-indexes-with-apache.html>
> which is a more visual approach.
> > Personally I'd write a small Python program.
> >
> > When asking ChatGPT to sketch such a program, this is what I got :)
> >
> > Sure, here is a sample Python program that demonstrates how to import
> data from a MySQL database and index it into Solr using the SolrPy library:
> >
> > import mysql.connector
> > from solr import Solr
> >
> > # Connect to MySQL database
> > cnx = mysql.connector.connect(user='username',
> >  password='password',
> >  host='hostname',
> >  database='database_name')
> > cursor = cnx.cursor()
> >
> > # Execute MySQL query
> > query = "SELECT * FROM table_name"
> > cursor.execute(query)
> >
> > # Fetch all data
> > data = cursor.fetchall()
> >
> > # Connect to Solr
> > solr = Solr('http://localhost:8983/solr/core_name')
> >
> > # Index data into Solr
> > for row in data:
> >solr.add(row)
> >
> > # Commit changes
> > solr.commit()
> >
> > # Close database and Solr connection
> > cursor.close()
> > cnx.close()
> > solr.close()
> >
> >
> > I have not tested it, but with some massage you should be able to fly
> >
> > Jan
> >
> >> 13. jan. 2023 kl. 16:02 skrev Markus Jelsma  >:
> >>
> >> Hello,
> >>
> >> DIH was removed in Solr 9.x. You can either build your own importer, or
> use
> >> the, now 3rd party, DIH [1] that is available as a package.
> >>
> >> Regards,
> >> Markus
> >>
> >> [1] https://github.com/rohitbemax/dataimporthandler
> >>
> >>> Op vr 13 jan. 2023 om 14:47 schreef Anuj Bhargava  >:
> >>>
> >>> How do I import data from MySql database in Solr 9.1
> >>>
> >>> Earlier I was using DIH in version 8.6.2
> >>>
> >>> Regards,
> >>>
> >>> Anuj
> >>>
> >
>

Re: Importing Data from MySql

[Dave]

Agreed. I like either waiting until the entire thing is done or a modulus of 
like 1000 if you’re worried about it failing part way through and it’s big. The 
less commits the better

> On Jan 13, 2023, at 10:49 AM, Thomas Corthals  wrote:
> 
> Build an indexer that can send updates in batches. It'll be faster than
> sending each document in a separate request.
> 
> Op vr 13 jan. 2023 om 16:41 schreef Dave :
> 
>> Yeah, it’s trivial building your own indexer in any language that can read
>> a db. Also I wouldn’t trust the dih on its own even when supported
>> 
 On Jan 13, 2023, at 10:17 AM, Jan Høydahl  wrote:
>>> 
>>> I don't think the 3rd party DIH is maintained.
>>> 
>>> Other options are using other 3rd party frameworks like
>>> ManifoldCF <
>> https://datafari.atlassian.net/wiki/spaces/DATAFARI/pages/2718007297/How+to+crawl+a+database+on+Solr+without+DIH+-+DIH+alternative>
>> or Apache NiFI <
>> https://www.datainmotion.dev/2020/04/building-search-indexes-with-apache.html>
>> which is a more visual approach.
>>> Personally I'd write a small Python program.
>>> 
>>> When asking ChatGPT to sketch such a program, this is what I got :)
>>> 
>>> Sure, here is a sample Python program that demonstrates how to import
>> data from a MySQL database and index it into Solr using the SolrPy library:
>>> 
>>> import mysql.connector
>>> from solr import Solr
>>> 
>>> # Connect to MySQL database
>>> cnx = mysql.connector.connect(user='username',
>>> password='password',
>>> host='hostname',
>>> database='database_name')
>>> cursor = cnx.cursor()
>>> 
>>> # Execute MySQL query
>>> query = "SELECT * FROM table_name"
>>> cursor.execute(query)
>>> 
>>> # Fetch all data
>>> data = cursor.fetchall()
>>> 
>>> # Connect to Solr
>>> solr = Solr('http://localhost:8983/solr/core_name')
>>> 
>>> # Index data into Solr
>>> for row in data:
>>>   solr.add(row)
>>> 
>>> # Commit changes
>>> solr.commit()
>>> 
>>> # Close database and Solr connection
>>> cursor.close()
>>> cnx.close()
>>> solr.close()
>>> 
>>> 
>>> I have not tested it, but with some massage you should be able to fly
>>> 
>>> Jan
>>> 
 13. jan. 2023 kl. 16:02 skrev Markus Jelsma >> :
 
 Hello,
 
 DIH was removed in Solr 9.x. You can either build your own importer, or
>> use
 the, now 3rd party, DIH [1] that is available as a package.
 
 Regards,
 Markus
 
 [1] https://github.com/rohitbemax/dataimporthandler
 
> Op vr 13 jan. 2023 om 14:47 schreef Anuj Bhargava >> :
> 
> How do I import data from MySql database in Solr 9.1
> 
> Earlier I was using DIH in version 8.6.2
> 
> Regards,
> 
> Anuj
> 
>>> 
>> 

Re: Importing Data from MySql

[Gus Heck]

Not sure I'd say it's trivial. But there are lots of folks who've done it
successfully. As noted, batching is important, depending on the nature of
the data fault tolerance can be important too. Daily data loads are a bit
different than continuous feeds of data. Also depends on to what extent
one  wants to massage the data going in to reduce load on the server from
indexing. Size of the dataset can imply performance needs, and scalability
etc.

-Gus

On Fri, Jan 13, 2023 at 10:41 AM Dave  wrote:

> Yeah, it’s trivial building your own indexer in any language that can read
> a db. Also I wouldn’t trust the dih on its own even when supported
>
> > On Jan 13, 2023, at 10:17 AM, Jan Høydahl  wrote:
> >
> > I don't think the 3rd party DIH is maintained.
> >
> > Other options are using other 3rd party frameworks like
> > ManifoldCF <
> https://datafari.atlassian.net/wiki/spaces/DATAFARI/pages/2718007297/How+to+crawl+a+database+on+Solr+without+DIH+-+DIH+alternative>
> or Apache NiFI <
> https://www.datainmotion.dev/2020/04/building-search-indexes-with-apache.html>
> which is a more visual approach.
> > Personally I'd write a small Python program.
> >
> > When asking ChatGPT to sketch such a program, this is what I got :)
> >
> > Sure, here is a sample Python program that demonstrates how to import
> data from a MySQL database and index it into Solr using the SolrPy library:
> >
> > import mysql.connector
> > from solr import Solr
> >
> > # Connect to MySQL database
> > cnx = mysql.connector.connect(user='username',
> >  password='password',
> >  host='hostname',
> >  database='database_name')
> > cursor = cnx.cursor()
> >
> > # Execute MySQL query
> > query = "SELECT * FROM table_name"
> > cursor.execute(query)
> >
> > # Fetch all data
> > data = cursor.fetchall()
> >
> > # Connect to Solr
> > solr = Solr('http://localhost:8983/solr/core_name')
> >
> > # Index data into Solr
> > for row in data:
> >solr.add(row)
> >
> > # Commit changes
> > solr.commit()
> >
> > # Close database and Solr connection
> > cursor.close()
> > cnx.close()
> > solr.close()
> >
> >
> > I have not tested it, but with some massage you should be able to fly
> >
> > Jan
> >
> >> 13. jan. 2023 kl. 16:02 skrev Markus Jelsma  >:
> >>
> >> Hello,
> >>
> >> DIH was removed in Solr 9.x. You can either build your own importer, or
> use
> >> the, now 3rd party, DIH [1] that is available as a package.
> >>
> >> Regards,
> >> Markus
> >>
> >> [1] https://github.com/rohitbemax/dataimporthandler
> >>
> >>> Op vr 13 jan. 2023 om 14:47 schreef Anuj Bhargava  >:
> >>>
> >>> How do I import data from MySql database in Solr 9.1
> >>>
> >>> Earlier I was using DIH in version 8.6.2
> >>>
> >>> Regards,
> >>>
> >>> Anuj
> >>>
> >
>


-- 
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)

Re: Importing Data from MySql

[Anuj Bhargava]

So what would be the advice to a layman like me?

1. Use Solr 8.x with builtin DIH
or
2. Use a 3rd part DIH with Solr 9.x

I have more than 20 cores, updating data from mysql daily through cron.

Regards,

Anuj

On Fri, 13 Jan 2023 at 20:33, Markus Jelsma 
wrote:

> Hello,
>
> DIH was removed in Solr 9.x. You can either build your own importer, or use
> the, now 3rd party, DIH [1] that is available as a package.
>
> Regards,
> Markus
>
> [1] https://github.com/rohitbemax/dataimporthandler
>
> Op vr 13 jan. 2023 om 14:47 schreef Anuj Bhargava :
>
> > How do I import data from MySql database in Solr 9.1
> >
> > Earlier I was using DIH in version 8.6.2
> >
> > Regards,
> >
> > Anuj
> >
>

Re: Importing Data from MySql

[dmitri maziuk]

On 2023-01-13 11:12 PM, Anuj Bhargava wrote:

So what would be the advice to a layman like me?

1. Use Solr 8.x with builtin DIH
or
2. Use a 3rd part DIH with Solr 9.x


Export your data as JSON documents (look it up on Google) and POST them 
to Solr with curl (look it up in Solr documentation). Most of us will do 
that in a script because we can, but in the simple case the above should 
work too.


Dima