在线时间:8:00-16:00
迪恩网络APP
随时随地掌握行业动态
扫描二维码
关注迪恩网络微信公众号
开源软件名称:KSQL开源软件地址:https://gitee.com/mirrors/KSQL开源软件介绍:ksqlDBThe database purpose-built for stream processing applicationsOverviewksqlDB is a database for building stream processing applications on top of Apache Kafka. It is distributed, scalable, reliable, and real-time. ksqlDB combines the power of real-time stream processing with the approachable feel of a relational database through a familiar, lightweight SQL syntax. ksqlDB offers these core primitives:
Composing these powerful primitives enables you to build a complete streaming app with just SQL statements, minimizing complexity and operational overhead. ksqlDB supports a wide range of operations including aggregations, joins, windowing, sessionization, and much more. You can find more ksqlDB tutorials and resources here. Getting Started
DocumentationSee the ksqlDB documentation for the latest stable release. Use Cases and ExamplesMaterialized viewsksqlDB allows you to define materialized views over your streams and tables. Materialized views are defined by what is known as a "persistent query". These queries are known as persistent because they maintain their incrementally updated results using a table. CREATE TABLE hourly_metrics AS SELECT url, COUNT(*) FROM page_views WINDOW TUMBLING (SIZE 1 HOUR) GROUP BY url EMIT CHANGES; Results may be "pulled" from materialized views on demand via SELECT * FROM hourly_metrics WHERE url = 'http://myurl.com' AND WINDOWSTART = '2019-11-20T19:00'; Results may also be continuously "pushed" to clients via streaming SELECT * FROM hourly_metrics EMIT CHANGES; Streaming queries will run perpetually until they are explicitly terminated. Streaming ETLApache Kafka is a popular choice for powering data pipelines. ksqlDB makes it simple to transform data within the pipeline, readying messages to cleanly land in another system. CREATE STREAM vip_actions AS SELECT userid, page, action FROM clickstream c LEFT JOIN users u ON c.userid = u.user_id WHERE u.level = 'Platinum' EMIT CHANGES; Anomaly DetectionksqlDB is a good fit for identifying patterns or anomalies on real-time data. By processing the stream as data arrives you can identify and properly surface out of the ordinary events with millisecond latency. CREATE TABLE possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 SECONDS) GROUP BY card_number HAVING count(*) > 3 EMIT CHANGES; MonitoringKafka's ability to provide scalable ordered records with stream processing make it a common solution for log data monitoring and alerting. ksqlDB lends a familiar syntax for tracking, understanding, and managing alerts. CREATE TABLE error_counts AS SELECT error_code, count(*) FROM monitoring_stream WINDOW TUMBLING (SIZE 1 MINUTE) WHERE type = 'ERROR' GROUP BY error_code EMIT CHANGES; Integration with External Data Sources and SinksksqlDB includes native integration with Kafka Connect data sources and sinks, effectively providing a unified SQL interface over a broad variety of external systems. The following query is a simple persistent streaming query that will produce all of its output into a topic named CREATE STREAM clicks_transformed AS SELECT userid, page, action FROM clickstream c LEFT JOIN users u ON c.userid = u.user_id EMIT CHANGES; Rather than simply send all continuous query output into a Kafka topic, it is often very useful to route the output into another datastore. ksqlDB's Kafka Connect integration makes this pattern very easy. The following statement will create a Kafka Connect sink connector that continuously sends all output from the above streaming ETL query directly into Elasticsearch: CREATE SINK CONNECTOR es_sink WITH ( 'connector.class' = 'io.confluent.connect.elasticsearch.ElasticsearchSinkConnector', 'key.converter' = 'org.apache.kafka.connect.storage.StringConverter', 'topics' = 'clicks_transformed', 'key.ignore' = 'true', 'schema.ignore' = 'true', 'type.name' = '', 'connection.url' = 'http://elasticsearch:9200'); Join the CommunityFor user help, questions or queries about ksqlDB please use our user Google Groupor our public Slack channel #ksqldb in Confluent Community Slack. Everyone is welcome! You can get help, learn how to contribute to ksqlDB, and find the latest news by connecting with the Confluent community. For more general questions about the Confluent Platform please post in the Confluent Google group. Contributing and building from sourceContributions to the code, examples, documentation, etc. are very much appreciated.
LicenseThe project is licensed under the Confluent Community License. Apache, Apache Kafka, Kafka, and associated open source project names are trademarks of the Apache Software Foundation. |
请发表评论