fuweng11 commented on code in PR #965: URL: https://github.com/apache/inlong-website/pull/965#discussion_r1699780045
########## blog/2024-07-18-release-1.13.0.md: ########## @@ -0,0 +1,142 @@ +--- +title: Release 1.13.0 +author: Wenkai Fu +author_url: https://github.com/fuweng11 +author_image_url: https://avatars.githubusercontent.com/u/8108604?s=400&v=4 +tags: [Apache InLong, Version] +--- + +Apache InLong recently released version 1.13.0, which closed about 275+ issues, including 6+ major features and 100+ optimizations. The main features include Manager supports for agent install package management and it's self-upgrading processe, Agent ability for self-upgrading process, Agent ability for collecting data from Kafka、Pulsar and MongoDB, Support for Redis connector in Sort module, Optimization for Audit and enhancement of its capabilities +. After the release of 1.13.0, Apache InLong has enriched and optimized Agent function scenarios, enhanced the accuracy of Audit data measurement, and enriched the capabilities and applicable scenarios of Sort, solved the demand for quick troubleshooting in development and operation, and optimized the user experience of Apache InLong operation and maintenance. +<!--truncate--> + +## About Apache InLong + +As the industry's first one-stop, full-scenario, open-source massive data integration framework, Apache InLong provides automatic, safe, reliable, and high-performance data transmission capabilities to facilitate businesses to build stream-based data analysis, modeling, and applications quickly. At present, InLong is widely used in various industries such as advertising, payment, social networking, games, artificial intelligence, etc., serving thousands of businesses, among which the scale of high-performance scene data exceeds 1 trillion lines per day, and the scale of high-reliability scene data exceeds 10 trillion lines per day. + +The core keywords of InLong project positioning are "one-stop" and "massive data". For "one-stop", we hope to shield technical details, provide complete data integration and support services, and implement out-of-the-box; With its advantages, such as multi-cluster management, it can stably support larger-scale data volumes based on trillions of lines per day. + +## 1.13.0 Version Overview + +Apache InLong recently released version 1.13.0, which closed about 140+ issues, including 7+ major features and 90+ optimizations. The main features include Manager supports for agent install package management and it's self-upgrading processe, Agent ability for self-upgrading process, Agent ability for collecting data from Kafka、Pulsar and MongoDB, Support for Redis connector in Sort module, Optimization for Audit and enhancement of its capabilities +. After the release of 1.13.0, Apache InLong has enriched and optimized Agent function scenarios, enhanced the accuracy of Audit data measurement, and enriched the capabilities and applicable scenarios of Sort, solved the demand for quick troubleshooting in development and operation, and optimized the user experience of Apache InLong operation and maintenance. In Apache InLong 1.13.0 version, a large number of other features have also been completed, mainly including: + +### Agent Module +- Support data version numbers to distinguish between normal data and supplementary data +- Location storage supports plugins, currently supporting Rocksdb and Zookeeper +- Support configuration version number comparison to prevent repeated configuration +- Support minute level file collection +- Add PostgreSQL and MongoDB data source collection + +### Manager Module +- Support installing agents through SSH +- Switch audit ID query from direct interaction with database to Audit SDK +- Offline synchronization supports Pulsar -> MySQL +- Support offline synchronous scheduling information management +- File collection supports multi IP collection +- Support obtaining Agent configuration information +- Support automatic synchronization to Sink after modifying Stream field information +- Support field template management +- Data preview supports KV format +- Data preview supports querying based on field filtering criteria + +### Dashboard Module +- Add Source Data Field Template Page +- Add monitoring and auditing page +- Support installing agents through SSH key based authentication +- Audit supports displaying total and variance audit data +- File type data stream supports minute level cycles + +### Audit Module +- Unified allocation and management of audit items using the Audit SDK +- The Audit SDK supports automatic management of Audit Proxy addresses +- Audit SDK optimization of TCP packet sticking leads to inaccurate audit reconciliation +- Optimization of Audit SDK for Audit Item and Indicator Management Issues +- Audit Store supports the universal JDBC protocol +- Restarting the Audit Store optimization process may lead to data loss issues +- Audit Store cleans up historical useless code +- Audit Service optimizes thread pool management +- Audit Service compatible with historical audit data with empty Audit Tag +- Audit Service Optimization for OpenAPI Audit Transmission Delay Calculation +- Audit Service OpenAPI supports querying historical hourly audit data from one day ago +- Audit Service supports automatic management of partitions +- Optimizing container environment variable conflicts + +### Sort Module +- Supports using state key during StarRocks connector sinitialization +- Supports parsing KV and CSV data containing split symbols +- Using ZLIB as the default compression type for Pulsar Sink +- Pulsar Connector supports authentication configuration +- Pulsar Sink supports authentication configuration +- Redis Source supports String, Hash, and ZSet data types +- Redis Sink supports Bitmap, Hash, and String data types + +## 1.13.0 Version Feature Introduction + +### Manager supports installing Agent by SSH +Through this feature, operation and maintenance personnel can install agents through the Dashboard, which currently supports SSH and manual installation methods. Users can create a new Agent cluster on the cluster management page. + +Afterwards, enter the node, select the new node and configure the SSH username and password to achieve SSH installation agent capability. Thanks to @haifxu and @fuweng11. For more information, please refer to INLONG-10409. + + +### Manager supports field template management +Through this feature, users can pre configure field templates, and when creating a new Stream, they can select the already configured field template, thereby achieving the goal of repeatedly configuring multiple Streams. +Thanks to @kamianlaida and @fuweng11. For more information, please refer to: INLONG-10330. +! [1.13.0-create-template.png](img%2F1.13.0-create-template.png) +! [1.13.0-select-template.png](img%2F1.13.0-select-template.png) +! [1.13.0-import-template.png](img%2F1.13.0-import-template.png) + +### InLong supports configuring offline synchronization tasks Pulsar > MySQL +In version 1.13.0, Manager supports the configuration of offline synchronization tasks. Compared to real-time synchronization, offline data synchronization(not supported yet) pays more attention to synchronization throughput and efficiency. +Real-time synchronization tasks run in the manner of Flink stream tasks, while offline synchronization runs in the manner of Flink batch tasks. This approach can ensure the consistency of real-time and offline synchronization tasks' code as much as possible, reducing maintenance costs. +The offline synchronization function of InLong will be combined with the scheduling system to synchronize the complete or incremental data of the data source information to the data target. The offline synchronization task is created by InLong Manager (including scheduling information), and the specific data synchronization logic is implemented through the InLong Sort module. + Review Comment: Fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@inlong.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
