I've had similar issues with the different processing done by
JoltTransformJSON and JoltTransformRecord and threw my hands in the air and
just ExcuteScript out to some Python that transforms the data correctly.
Try to minimise content transformations so the content repository doesn't
bloat with in
Chris,
I guess I was so wrapped up in the ConvertRecord, it didn't occur to me to
have the JOLT processor just emit a different format; not sure how I missed
that.
So, I re-rigged my flow to have ReplaceText -> JOLTTransform ->
UpdateAttribute -> etc. Unfortunately, I got the attached error.
T
You could use an AvroWriter to output the results of the JoltTransformRecord -
it doesn’t need to be JSON (in or out), that’s one of the great things of
NiFi’s Record processors - if there’s a Reader and Writer in the format you
want, you can use that data and the Writer doesn’t need to be the s
Chris,
1) well, reassuring to learn that I've found an actual bug; and pleasing to
know that I constructed an effective and illuminating test. hurrah.
2) So, I can certainly use the ReplaceText (is there a better choice?)
processor to ditch any field that looks like "whatever": "", (and I
succes
FYI - original thread in the archives for reference [1].
Thanks for your more complete example, this does indeed fail with the error you
indicate. I think it’s related to NIFI-8135 [2], which identified a deficiency
in the way Records are converted to Java Maps, particularly where CHOICE types
Matt,
1) Yea, I definitely muddled what you meant by "just JSON" before. I'm
definitely looking to get a bunch of records out...because I want to AVRO
each of them later.
2) I'm sure I've also messed up the threading in the issues. Apparently, I
should have subscribed before today (which I've
Chris,
1) I've attached a file with a total of four records. 3 with no complex
inner fields, and the last one has the complex inner fields. I get the
error on the last record: complaining about the Dimensions field.
2) The data definitely seems to be nested objects (at least as I understand
the
For some reason I don't have the original thread, I must've
inadvertently deleted it. IIRC your example input was a single JSON
object and I said if that were the case you could use
JoltTransformJSON instead. However if that is NOT the case (which is
your point c above) then you have a couple of op
Mew,
Are you able to provide examples of the additional data?
Are you able to (easily) see whether any of your data has these inner
fields as anything other than nested objects, e.g. maybe the field is a
string in one of your top-level objects?
On Tue, 10 Oct 2023, 21:47 Mark Woodcock, wrote:
Chris,
1) I've upgraded to 1.23.2 (which appears to be the latest and greatest).
2) I've tested the JoltTransformRecord with
a) JsonTreeReader w/ InferredSchema
b) JsonRecordSetWriter w/ InheritsSchema
c) a GetFile processor which grabs a text file with the various bits of
test data
It appears
Using your example (single JSON Object and Jolt Spec) seems to work fine in
both JoltTransformJSON and JoltTransformRecord when run on the current main
branch (which is for the upcoming 2.0.0 release).
To test, I setup a GenerateFlowFile processor to output the example JSON you
gave, then sent
H,
One small problem: While JOLTTransformJSON is quite lovely (a) it has a
great "advanced" interface that allows one to test their spec and json
inputs and (b) it actually works for the cases that I noted...it treats the
input a single blob of JSON. Unfortunately, my input files are collect
confirmed: version 1.21.
How recent is the fix?
thx,
mew
On Sun, Oct 8, 2023 at 11:39 PM Mark Woodcock wrote:
> Matt,
>
> Unfortunately (at home now) the details are all at work at the moment, but
> I know that I didn't start this work until April (at the earliest), so I'm
> surely using at
Matt,
Unfortunately (at home now) the details are all at work at the moment, but
I know that I didn't start this work until April (at the earliest), so I'm
surely using at least 1.21; is the fix more recent than that? {If so,
perhaps there is a bug.}
Fortunately, yea, JSON out is the intent; I
mew,
What version of NiFi are you using? We may have fixed the bug in newer
versions but if you are using the latest, this is bug. We shouldn't
infer CHOICE[STRING,RECORD] when the only entry is a RECORD.
Since you only have one top-level record in your FlowFile you can use
JoltTransformJSON inst
Howdy,
Given the jolt spec which I arrived at with the help I received in the
previous issue (flattening record schemas):
[
{
"operation": "shift",
"spec": { "*": "record.&" }
},
{
"operation": "shift",
"spec": {
"record": {
"*": {
"$": "TValue[#2].na
16 matches
Mail list logo
