在线时间:8:00-16:00
迪恩网络APP
随时随地掌握行业动态
扫描二维码
关注迪恩网络微信公众号
开源软件名称:linkedin/goavro开源软件地址:https://github.com/linkedin/goavro开源编程语言:Go 100.0%开源软件介绍:goavroGoavro is a library that encodes and decodes Avro data. Description
With the exception of features not yet supported, goavro attempts to be fully compliant with the most recent version of the Avro specification. Dependency NoticeAll usage of
[[constraint]]
name = "github.com/linkedin/goavro"
version = "=1.0.5" Major Improvements in v2 over v1Avro namespacesThe original version of this library was written prior to my really understanding how Avro namespaces ought to work. After using Avro for a long time now, and after a lot of research, I think I grok Avro namespaces properly, and the library now correctly handles every test case the Apache Avro distribution has for namespaces, including being able to refer to a previously defined data type later on in the same schema. Getting Data into and out of RecordsThe original version of this library required creating The new version of this library eliminates the Furthermore, there is never a reason to ever have to break your schema
down into record schemas. Merely feed the entire schema into the
3x--4x Performance ImprovementThe original version of this library was truly written with Go's idea
of Avro JSON SupportThe original version of this library did not support JSON encoding or decoding, because it wasn't deemed useful for our internal use at the time. When writing the new version of the library I decided to tackle this issue once and for all, because so many engineers needed this functionality for their work. Better Handling of Record Field Default ValuesThe original version of this library did not well handle default values for record fields. This version of the library uses a default value of a record field when encoding from native Go data to Avro data and the record field is not specified. Additionally, when decoding from Avro JSON data to native Go data, and a field is not specified, the default value will be used to populate the field. Contrast With Code Generation ToolsIf you have the ability to rebuild and redeploy your software whenever data schemas change, code generation tools might be the best solution for your application. There are numerous excellent tools for generating source code to
translate data between native and Avro binary or textual data. One
such tool is linked below. If a particular application is designed to
work with a rarely changing schema, programs that use code generated
functions can potentially be more performant than a program that uses
goavro to create a I recommend benchmarking the resultant programs using typical data using both the code generated functions and using goavro to see which performs better. Not all code generated functions will out perform goavro for all data corpuses. If you don't have the ability to rebuild and redeploy software updates
whenever a data schema change occurs, goavro could be a great fit for
your needs. With goavro, your program can be given a new schema while
running, compile it into a ResourcesUsageDocumentation is available via . package main
import (
"fmt"
"github.com/linkedin/goavro/v2"
)
func main() {
codec, err := goavro.NewCodec(`
{
"type": "record",
"name": "LongList",
"fields" : [
{"name": "next", "type": ["null", "LongList"], "default": null}
]
}`)
if err != nil {
fmt.Println(err)
}
// NOTE: May omit fields when using default value
textual := []byte(`{"next":{"LongList":{}}}`)
// Convert textual Avro data (in Avro JSON format) to native Go form
native, _, err := codec.NativeFromTextual(textual)
if err != nil {
fmt.Println(err)
}
// Convert native Go form to binary Avro data
binary, err := codec.BinaryFromNative(nil, native)
if err != nil {
fmt.Println(err)
}
// Convert binary Avro data back to native Go form
native, _, err = codec.NativeFromBinary(binary)
if err != nil {
fmt.Println(err)
}
// Convert native Go form to textual Avro data
textual, err = codec.TextualFromNative(nil, native)
if err != nil {
fmt.Println(err)
}
// NOTE: Textual encoding will show all fields, even those with values that
// match their default values
fmt.Println(string(textual))
// Output: {"next":{"LongList":{"next":null}}}
} Also please see the example programs in the ab2tThe arwThe Avro-ReWrite program, avroheaderThe Avro Header program, spliceThe Translating DataA The following methods convert data between native Go data and byte slices of the binary Avro representation:
The following methods convert data between native Go data and byte slices of the textual Avro representation:
Each Translating From Avro to Go DataGoavro does not use Go's structure tags to translate data between native Go types and Avro encoded data. When translating from either binary or textual Avro to native Go data, goavro returns primitive Go data values for corresponding Avro data values. The table below shows how goavro translates Avro types to Go types.
Because of encoding rules for Avro unions, when an union's value is
Translating From Go to Avro DataGoavro does not use Go's structure tags to translate data between native Go types and Avro encoded data. When translating from native Go to either binary or textual Avro data,
goavro generally requires the same native Go data types as the decoder
would provide, with some exceptions for programmer convenience. Goavro
will accept any numerical data type provided there is no precision
lost when encoding the value. For instance, providing When providing a slice of items for an encoder, the encoder will
accept either When providing a value for an Avro union, the encoder will accept
func ExampleUnion() {
codec, err := goavro.NewCodec(`["null","string","int"]`)
if err != nil {
fmt.Println(err)
}
buf, err := codec.TextualFromNative(nil, goavro.Union("string", "some string"))
if err != nil {
fmt.Println(err)
}
fmt.Println(string(buf))
// Output: {"string":"some string"}
} LimitationsGoavro is a fully featured encoder and decoder of binary and textual JSON Avro data. It fully supports recursive data structures, unions, and namespacing. It does have a few limitations that have yet to be implemented. AliasesThe Avro specification allows an implementation to optionally map a writer's schema to a reader's schema using aliases. Although goavro can compile schemas with aliases, it does not yet implement this feature. Kafka StreamsKafka is the reason goavro was written. Similar to Avro Object Container Files being a layer of abstraction above Avro Data Serialization format, Kafka's use of Avro is a layer of abstraction that also sits above Avro Data Serialization format, but has its own schema. Like Avro Object Container Files, this has been implemented but removed until the API can be improved. Default Maximum Block Counts, and Block SizesWhen decoding arrays, maps, and OCF files, the Avro specification states that the binary includes block counts and block sizes that specify how many items are in the next block, and how many bytes are in the next block. To prevent possible denial-of-service attacks on clients that use this library caused by attempting to decode maliciously crafted data, decoded block counts and sizes are compared against public library variables MaxBlockCount and MaxBlockSize. When the decoded values exceed these values, the decoder returns an error. Because not every upstream client is the same, we've chosen some sane
defaults for these values, but left them as mutable variables, so that
clients are able to override if deemed necessary for their
purposes. Their initial default values are ( Schema EvolutionPlease see my reasons why schema evolution is broken for Avro 1.x. LicenseGoavro licenseCopyright 2017 LinkedIn Corp. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. Google Snappy licenseCopyright (c) 2011 The Snappy-Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Third Party DependenciesGoogle SnappyGoavro links with Google Snappy to provide Snappy compression and decompression support. |
2023-10-27
2022-08-15
2022-08-17
2022-09-23
2022-08-13
请发表评论