[DISCUSS] KIP-301 Schema Inferencing in JsonConverters

2018-05-31 Thread Andrew Otto
Hiya, I’m late to this thread, and was just referred to the KIP from the confluent mailing list. Nice idea! I like the idea of being able to provide a Schema fallback for JsonConverter. Perhaps an interface that could return the schema given the JsonNode?For my use case, I have JSONSchemas r

Re: [DISCUSS] KIP-301 Schema Inferencing in JsonConverters

2018-05-18 Thread Randall Hauch
There are lots of potential details/gotchas here, with no obvious defaults. Even if we did have optional configs for the type(s) to use for empty arrays and null values, how likely is it that those would apply for all occurrences? Two fields might have empty arrays, but the best schema for each mig

Re: [DISCUSS] KIP-301 Schema Inferencing in JsonConverters

2018-05-15 Thread Allen Tang
I've went through several iterations of back-and-forth with @rhauch on the PR and on Confluent's Slack Community. The current thinking is that assuming an empty array is a String array is not necessarily the best option, nor is assuming that all null values in a JSON node is a String. We might be

[DISCUSS] KIP-301 Schema Inferencing in JsonConverters

2018-05-14 Thread Allen Tang
Hi, I just opened a KIP to add Schema Inferencing in JsonConverters for Kafka Connect. The details of the proposal can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-301%3A+Schema+Inferencing+for+JsonConverter Also, I have created a - 1.) JIRA ticket: https://issues.apache