Hello,
I have a route that triggers a SELECT in a database and creates a JSON file
in a folder (this file which on average generates something around 100k to
150k of records).
Then another route that sends the data of this file to an endpoint using the
HTTP4 component and performing a POST and REST API (the endpoint is already
prepared to perform batch processing).
I noticed something curious, that in certain cases Camel sends the data, Web
Services is processing correctly, but after a few seconds Camel returns me a
DeadLetterChannel, with some exceptions:
- "org.apache.http.wire" "http-outgoing-5" "end of stream"
- "org.apache.http.nohttpresponseexception" "failed to respond"
- "h.i.c.DefaultManagedHttpClientConnection" "http-outgoing-5" "Close
connection"
- "h.i.c.DefaultManagedHttpClientConnection" "http-outgoing-5" "Shutdown
connection"
- "o.a.http.impl.execchain.MainClientExec" "Connection discarded"
"Connection released"
But the process continues normally, it tries to resend again, but gives the
error again, but in AWS web services it is processing normally and I have
verified that it returns the normal status codes.
Searching the Camel documentation and found httpClient. I made the
configuration by increasing the timeout, but nothing worked, it seems that
it ignores these settings (httpClient.socketTimeout = 300000).
I'm thinking of splitting the JSON file, with some rule like every 1000
records generating 1 JSON file. So I will have several smaller files that
will be processed faster and thus get the response from the server, avoiding
this error.
My actual route thats find all records in database and create a json file
is:
from("timer://query?fixedRate=true&delay=5s&period=10800s")
.routeId("route-product")
.setBody(constant(new ProductQuery().getProductSelect()))
.to("jdbc:myDataSource")
.marshal()
.json(JsonLibrary.Gson)
.setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
.setHeader(Exchange.HTTP_CHARACTER_ENCODING, constant("UTF-8"))
.setHeader("CamelFileName", constant("products.json"))
.to("file:products");
However, searching the documentation, I was able to perform the split:
from("timer://query?fixedRate=true&delay=5s&period=10800s")
.routeId("route-product")
.setBody(constant(new ProductQuery().getProductSelect()))
.to("jdbc:myDataSource?outputType=StreamList")
.split(body()).streaming()
.marshal()
.json(JsonLibrary.Gson)
.setHeader(Exchange.CONTENT_TYPE, constant("application/json"))
.setHeader("CamelFileName", simple("${id}.json"))
.to("file:product");
In this way each record creates a JSON file, but my case, I want to group
these records, again in the documentation is suggested as follows:
.split().tokenize("\n", 1000).streaming()
But in this way the output of the log shows only 1 file with the following
content:
"org.apache.camel.component.jdbc.ResultSetIterator@7be7d81f"
If I remove the option ?outputType=StreamList from the JDBC component's URI,
then the following exception is thrown:
No converter found capable of converting from type [java.util.ArrayList<?>]
to type [java.lang.String]
--
View this message in context:
http://camel.465427.n5.nabble.com/Using-Split-with-JSON-File-tp5797508.html
Sent from the Camel - Users mailing list archive at Nabble.com.
