Hi,
Could someone please help me with the what Solr is looking in deltaQuery of
Delta Import ?
I have got 18 rows fetched from the database, but none of the row is
processed as document.
There are the "WARN" In the solr.log, but, what attribute is not specified?
It seems important because the Raw Debug-Response shows 18 rows fetched and
are empty/no processed documents. Thanks.
"verbose-output": [
"entity:<cowboysEntity>",
[
"document#1",
[]
],
"entity:<cowboysEntity>",
[
"document#1",
[]
],
...
solr.log
2023-12-01 09:24:38.325 INFO (qtp548482954-31) [ x:p6]
o.a.s.h.d.DataImporter Loading DIH Configuration: db-data-config.xml
2023-12-01 09:24:38.328 INFO (qtp548482954-31) [ x:p6]
o.a.s.h.d.DataImporter Data Configuration loaded successfully
2023-12-01 09:24:38.328 INFO (qtp548482954-31) [ x:p6]
o.a.s.h.d.DataImporter Starting Delta Import
2023-12-01 09:24:38.335 INFO (qtp548482954-31) [ x:p6]
o.a.s.h.d.SimplePropertiesWriter Read dataimport.properties
2023-12-01 09:24:38.336 INFO (qtp548482954-31) [ x:p6]
o.a.s.h.d.DocBuilder Starting delta collection.
2023-12-01 09:24:38.338 INFO (qtp548482954-31) [ x:p6]
o.a.s.h.d.DocBuilder Running ModifiedRowKey() for Entity: <cowboysEntity>
2023-12-01 09:24:38.339 INFO (qtp548482954-31) [ x:p6]
o.a.s.h.d.JdbcDataSource Creating a connection for entity <cowboysEntity>
with URL: jdbc:teradata://<teradataServer>/database
=Name1,tmode=TERA,charset=UTF8
2023-12-01 09:24:39.101 INFO (qtp548482954-31) [ x:p6]
o.a.s.h.d.JdbcDataSource Time taken for getConnection(): 762
2023-12-01 09:24:39.695 INFO (qtp548482954-31) [ x:p6]
o.a.s.h.d.DocBuilder Completed ModifiedRowKey for Entity: <cowboysEntity>
rows obtained: 18
2023-12-01 09:24:39.695 INFO (qtp548482954-31) [ x:p6]
o.a.s.h.d.DocBuilder Completed DeletedRowKey for Entity: <cowboysEntity>
rows obtained : 0
2023-12-01 09:24:39.695 INFO (qtp548482954-31) [ x:p6]
o.a.s.h.d.DocBuilder Completed parentDeltaQuery for Entity: <cowboysEntity>
2023-12-01 09:24:39.696 WARN (qtp548482954-31) [ x:p6]
o.a.s.h.d.SqlEntityProcessor 'deltaImportQuery' attribute is not specified
for entity : <cowboysEntity>
2023-12-01 09:24:39.940 WARN (qtp548482954-31) [ x:p6]
o.a.s.h.d.SqlEntityProcessor 'deltaImportQuery' attribute is not specified
for entity : <cowboysEntity>
2023-12-01 09:24:40.138 WARN (qtp548482954-31) [ x:p6]
o.a.s.h.d.SqlEntityProcessor 'deltaImportQuery' attribute is not specified
for entity : <cowboysEntity>
2023-12-01 09:24:40.337 WARN (qtp548482954-31) [ x:p6]
o.a.s.h.d.SqlEntityProcessor 'deltaImportQuery' attribute is not specified
for entity : <cowboysEntity>
2023-12-01 09:24:40.532 WARN (qtp548482954-31) [ x:p6]
o.a.s.h.d.SqlEntityProcessor 'deltaImportQuery' attribute is not specified
for entity : <cowboysEntity>
2023-12-01 09:24:40.728 WARN (qtp548482954-31) [ x:p6]
o.a.s.h.d.SqlEntityProcessor 'deltaImportQuery' attribute is not specified
for entity : <cowboysEntity>
2023-12-01 09:24:40.924 WARN (qtp548482954-31) [ x:p6]
o.a.s.h.d.SqlEntityProcessor 'deltaImportQuery' attribute is not specified
for entity : <cowboysEntity>
2023-12-01 09:24:41.121 WARN (qtp548482954-31) [ x:p6]
o.a.s.h.d.SqlEntityProcessor 'deltaImportQuery' attribute is not specified
for entity : <cowboysEntity>
2023-12-01 09:24:41.322 WARN (qtp548482954-31) [ x:p6]
o.a.s.h.d.SqlEntityProcessor 'deltaImportQuery' attribute is not specified
for entity : <cowboysEntity>
2023-12-01 09:24:41.524 WARN (qtp548482954-31) [ x:p6]
o.a.s.h.d.SqlEntityProcessor 'deltaImportQuery' attribute is not specified
for entity : <cowboysEntity>
2023-12-01 09:24:41.724 WARN (qtp548482954-31) [ x:p6]
o.a.s.h.d.SqlEntityProcessor 'deltaImportQuery' attribute is not specified
for entity : <cowboysEntity>
2023-12-01 09:24:41.928 WARN (qtp548482954-31) [ x:p6]
o.a.s.h.d.SqlEntityProcessor 'deltaImportQuery' attribute is not specified
for entity : <cowboysEntity>
2023-12-01 09:24:42.129 WARN (qtp548482954-31) [ x:p6]
o.a.s.h.d.SqlEntityProcessor 'deltaImportQuery' attribute is not specified
for entity : <cowboysEntity>
2023-12-01 09:24:42.350 WARN (qtp548482954-31) [ x:p6]
o.a.s.h.d.SqlEntityProcessor 'deltaImportQuery' attribute is not specified
for entity : <cowboysEntity>
2023-12-01 09:24:42.551 WARN (qtp548482954-31) [ x:p6]
o.a.s.h.d.SqlEntityProcessor 'deltaImportQuery' attribute is not specified
for entity : <cowboysEntity>
2023-12-01 09:24:42.758 WARN (qtp548482954-31) [ x:p6]
o.a.s.h.d.SqlEntityProcessor 'deltaImportQuery' attribute is not specified
for entity : <cowboysEntity>
2023-12-01 09:24:42.955 WARN (qtp548482954-31) [ x:p6]
o.a.s.h.d.SqlEntityProcessor 'deltaImportQuery' attribute is not specified
for entity : <cowboysEntity>
2023-12-01 09:24:43.155 WARN (qtp548482954-31) [ x:p6]
o.a.s.h.d.SqlEntityProcessor 'deltaImportQuery' attribute is not specified
for entity : <cowboysEntity>
2023-12-01 09:24:43.361 INFO (qtp548482954-31) [ x:p6]
o.a.s.h.d.DocBuilder Delta Import completed successfully
2023-12-01 09:24:43.361 INFO (qtp548482954-31) [ x:p6]
o.a.s.h.d.DocBuilder Time taken = 0:0:5.26
2023-12-01 09:24:43.361 INFO (qtp548482954-31) [ x:p6]
o.a.s.u.p.LogUpdateProcessorFactory [p6] webapp=/solr path=/dataimport
params={core=p6&debug=true&indent=on&commit=true&name
=dataimport&clean=false&wt=json&command=delta-import&_=1701422662358&verbose=true}{}
0 5036
2023-12-01 09:24:43.605 INFO (qtp548482954-30) [ x:p6] o.a.s.c.S.Request
[p6] webapp=/solr path=/dataimport params={indent=on&wt=json&command=status
&_=1701422662358} status=0 QTime=0
Configuration (db-data-config.xml)
<dataConfig>
<dataSource type="JdbcDataSource"
driver="com.teradata.jdbc.TeraDriver"
url=
"jdbc:teradata://<teradataServer>/database=Name1,tmode=TERA,charset=UTF8"
user="<user>"
password="<pwd>"/>
<document>
<entity name ="<cowboysEntity>"
pk ="id"
query ="SELECT id, updated_at FROM
Name1.<cowboyTable> WHERE updated_at between '2023-01-01 00:00:00' and
'2023-01-01 00:30:00' "
deltaQuery ="SELECT id, updated_at FROM
Name1.<cowboyTable> WHERE updated_at >= cast('${dih.last_index_time}' AS
TIMESTAMP(0) FORMAT 'YYYY-MM-DDbHH:MI:SS') - INTERVAL '8' DAY ">
<field column="id" name="id"/>
<field column="updated_at" name="updated_at"/>
</entity>
</document>
</dataConfig>
In manage-schema file I have
<field name="id" type="string" indexed="true" stored="true" required=
"true" multiValued="false" />
<field name="update_at" type="string" indexed="true" stored="true"
multiValued="false"/>
Raw Debug-Response
{
"responseHeader": {
"status": 0,
"QTime": 5266
},
"initArgs": [
"defaults",
[
"config",
"db-data-config.xml"
]
],
"command": "delta-import",
"mode": "debug",
"documents": [],
"verbose-output": [
"entity:<cowboysEntity>",
[
"document#1",
[]
],
"entity:<cowboysEntity>",
[
"document#1",
[]
],
"entity:<cowboysEntity>",
[
"document#1",
[]
],
"entity:<cowboysEntity>",
[
"document#1",
[]
],
"entity:<cowboysEntity>",
[
"document#1",
[]
],
"entity:<cowboysEntity>",
[
"document#1",
[]
],
"entity:<cowboysEntity>",
[
"document#1",
[]
],
"entity:<cowboysEntity>",
[
"document#1",
[]
],
"entity:<cowboysEntity>",
[
"document#1",
[]
],
"entity:<cowboysEntity>",
[
"document#1",
[]
],
"entity:<cowboysEntity>",
[
"document#1",
[]
],
"entity:<cowboysEntity>",
[
"document#1",
[]
],
"entity:<cowboysEntity>",
[
"document#1",
[]
],
"entity:<cowboysEntity>",
[
"document#1",
[]
],
"entity:<cowboysEntity>",
[
"document#1",
[]
],
"entity:<cowboysEntity>",
[
"document#1",
[]
],
"entity:<cowboysEntity>",
[
"document#1",
[]
],
"entity:<cowboysEntity>",
[
"document#1",
[]
]
],
"status": "idle",
"importResponse": "",
"statusMessages": {
"Total Requests made to DataSource": "19",
"Total Rows Fetched": "18",
"Total Documents Processed": "0",
"Total Documents Skipped": "0",
"Delta Dump started": "2023-12-01 09:24:38",
"Identifying Delta": "2023-12-01 09:24:38",
"Deltas Obtained": "2023-12-01 09:24:39",
"Building documents": "2023-12-01 09:24:39",
"Total Changed Documents": "18",
"Time taken": "0:0:5.26"
}
}
