Hello kafka community, We use Apache Kafka and microservices publishing and subscribing to the data exposed by other microservices. We use AVRO. Different microservices publish one or more topics like events topics or facts/entities/current state topics. And consume from others.
Currently the data is a bit heterogenous and a lot of guesswork happens in terms of "is there any microservice publishing the data i need?", "where to find a data source for customer like entities?", "what data does team x publish?". We are exploring semantic web & ontologies for: 1. linking different topics AVRO schemas with OWL/RDFS ontologies 2. marking metadata on topics to be searched for 3. giving more semantics to data and allow more ways to navigate it for users across the enterprise It seems to me that the problem starts perhaps with proper modelling. Since AVRO is already a schema (for JSON documents), I would like to explore ways to go into the semantic layer of relating various concepts exposed by these microservices so that later on people and microservices can query this set of relations and find data sources that they would not normally be aware of, thus increasing productivity. Do you know of any project that integrated Kafka or AVRO with OWL Ontologies? Or more generally in the land of microservices do you know of pragmatic and nice application of ontologies to help offer visibility on data and various parties can search for "customer like entities" for instance, across the organisation? Do you see other technologies for achieving our goals of increasing transparency of domains and their interconnections? Thank you very much, for more context: https://www.quora.com/unanswered/How-can-AVRO-schemas-used-in-Kafka-topics-be-reused-to-generate-OWL-ontologies-and-how-can-OWL-ontologies-be-used-to-generate-AVRO-schemas-What-are-other-ways-to-inter-op https://www.quora.com/unanswered/How-would-you-mix-OWL-ontologies-with-AVRO-schemas-needed-in-Apache-Kafka-to-describe-the-same-domain-without-much-duplication https://www.quora.com/unanswered/What-are-the-ways-to-define-a-domain-model-that-can-be-freely-shared-across-organisations-and-be-laterally-developed-in-a-decentralised-fashion-allow-for-different-ways-of-modelling-the-Customer-Is-OWL-ontologies-a https://www.quora.com/What-kind-of-metadata-would-you-use-to-associate-data-sources-like-Kafka-topics-or-db-tables-that-they-represent-Entity-X-and-they-are-owned-by-team-Y-Would-OWL-Ontologies-be-ok-to-model-entities-across Thank you very much, Nicu -- Thank you, Nicolae Marasoiu Scala Engineer Orion, OVO Group
