Interestingly enough the next issue we hit is the `fl=750 fields` == too big,
so we switched to using POST/x-url-form-encoded for REINDEXCOLLECTION which
accepts the request, but then it silently fails (no logs in solr, just doesn't
work).
I can only assume this is because behind the scenes the daemon is using GET and
hitting the same url limit (but the error is being swallowed).
Significantly increasing the max http header length in jetty resolved the issue
so this feels like a bit of a bug?
On 22/03/2021, 15:37, "Karl Stoney" <karl.sto...@autotrader.co.uk.INVALID>
wrote:
So for context we have 900x fields on Collection one and have removed some
250 fields from the schema and want to reindex into collection2. We're trying
to have a process where we can easily remove fields and reindex without too
much coding overhead. Therefore, we were simply using the default `fl=*:*`
from the first collection as the assumption was that when we try and save the
document to collection2, the additional fields would just be ignored. This
wasn't the case.
Subsequently to work around this now we read the schema.xml for collection2
and build up a `fl` to pass to REINDEXCOLLECTION which only includes the fields
in collection2's schema. Which works.
On 22/03/2021, 13:20, "David Hastings" <hastings.recurs...@gmail.com> wrote:
>Surely this field should simply just be ignored?
why would solr ignore this field if you're trying to index to it? can't
you change your indexer to remove these fields as well? solr will try
to
do what its told, and if its told to do something bad it will simply
fail,
you dont want it to ignore errors or bad indexing
On Mon, Mar 22, 2021 at 9:15 AM David Smiley <dsmi...@apache.org> wrote:
>
>
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fguide%2F8_8%2Fcollection-management.html%23reindexcollection&data=04%7C01%7CKarl.Stoney%40autotrader.co.uk%7C16d7a53cccdf4bb545ac08d8ed484cf1%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637520242513742368%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5c6Do5%2FRPNVFdgdgm7kKw97O3xgvyBiMOhJcXDupP9o%3D&reserved=0
>
> See the "fl" param
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
>
https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fdavidwsmiley&data=04%7C01%7CKarl.Stoney%40autotrader.co.uk%7C16d7a53cccdf4bb545ac08d8ed484cf1%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637520242513752345%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ATyR1bkLE8Mt9F3oreaVmToKGC6Hzl6RvolmbJA997w%3D&reserved=0
>
>
> On Mon, Mar 22, 2021 at 9:01 AM Karl Stoney
> <karl.sto...@autotrader.co.uk.invalid> wrote:
>
> > Hi,
> > Sorry for all the questions recently…
> >
> > So as per
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsolr.apache.org%2Fguide%2F8_0%2Freindexing.html&data=04%7C01%7CKarl.Stoney%40autotrader.co.uk%7C16d7a53cccdf4bb545ac08d8ed484cf1%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637520242513752345%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tJ74flsz5IxGN72KI9A4Z%2BurJHRz1QXSpENBlanh%2Fv4%3D&reserved=0;
we’re
> trying
> > to remove a load of fields. Subsequently we’ve created a new
collection
> > with the new schema and we’re attempting to reindex from old to new.
> > There’s about 216 fields in total being removed…
> >
> > The REINDEX fails though, because the field has been removed:
> >
> > 2:52:31.508
> >
>
[DaemonStream-at-uk-003-12889-thread-1-processing-n:solr-0.search-solr.svc.cluster.local:80_solr
> > x:at-uk-002_shard1_replica_n1 c:at-uk-002 s:shard1 r:core_node2]
ERROR
> > org.apache.solr.client.solrj.io.stream.DaemonStream - Fatal Error in
> > DaemonStream: at-uk-003
> >
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> > Error from server at
> >
https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsolr-0.search-solr.svc.cluster.local%2Fsolr%2Fat-uk-003&data=04%7C01%7CKarl.Stoney%40autotrader.co.uk%7C16d7a53cccdf4bb545ac08d8ed484cf1%7C926f3743f3d24b8a816818cfcbe776fe%7C0%7C0%7C637520242513752345%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=eQIWOrab8PPiOQSXvcOMPMJavUCm9KFVEHaKcbQ7v9M%3D&reserved=0:
ERROR:
> > [doc=PVc175f0f12f8f43789a7c1863e10229cd] unknown field
> > 'OEM_ENGINE_MARKETING'
> >
> > Surely this field should simply just be ignored?
> >
> > I can’t see any way of using REINDEXCOLLECTION to reindex data from
one
> > collection to another where we have removed fields.
> >
> > Any input would be appreciated.
> > Unless expressly stated otherwise in this email, this e-mail is
sent on
> > behalf of Auto Trader Limited Registered Office: 1 Tony Wilson
Place,
> > Manchester, Lancashire, M15 4FN (Registered in England No.
03909628).
> Auto
> > Trader Limited is part of the Auto Trader Group Plc group. This
email and
> > any files transmitted with it are confidential and may be legally
> > privileged, and intended solely for the use of the individual or
entity
> to
> > whom they are addressed. If you have received this email in error
please
> > notify the sender. This email message has been swept for the
presence of
> > computer viruses.
> >
>
Unless expressly stated otherwise in this email, this e-mail is sent on
behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place,
Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto
Trader Limited is part of the Auto Trader Group Plc group. This email and any
files transmitted with it are confidential and may be legally privileged, and
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this email in error please notify the sender.
This email message has been swept for the presence of computer viruses.
Unless expressly stated otherwise in this email, this e-mail is sent on behalf
of Auto Trader Limited Registered Office: 1 Tony Wilson Place, Manchester,
Lancashire, M15 4FN (Registered in England No. 03909628). Auto Trader Limited
is part of the Auto Trader Group Plc group. This email and any files
transmitted with it are confidential and may be legally privileged, and
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this email in error please notify the sender.
This email message has been swept for the presence of computer viruses.
