本文整理汇总了Java中org.kitesdk.morphline.api.Record类的典型用法代码示例。如果您正苦于以下问题:Java Record类的具体用法?Java Record怎么用?Java Record使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。
Record类属于org.kitesdk.morphline.api包,在下文中一共展示了Record类的20个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Java代码示例。
示例1: executePipeline
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
@Test
public void executePipeline(
final @Mocked MorphlineUtils.Pipeline pipeline,
final @Mocked Command morphline
) throws Exception {
final Record inputRecord = new Record();
final Record outputRecord = new Record();
outputRecord.put("field1", "value1");
new Expectations() {{
morphline.process(inputRecord); result = true;
pipeline.getCollector().getRecords(); result = Lists.newArrayList(outputRecord);
}};
List<Record> outputList = MorphlineUtils.executePipeline(pipeline, inputRecord);
assertEquals(Lists.newArrayList(outputRecord), outputList);
}
开发者ID:cloudera-labs,项目名称:envelope,代码行数:20,代码来源:TestMorphlineUtils.java
示例2: convertToRowValidValue
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
@Test
public void convertToRowValidValue(
final @Mocked RowUtils utils
) throws Exception {
Record record = new Record();
record.put("field1", "one");
StructType schema = DataTypes.createStructType(Lists.newArrayList(
DataTypes.createStructField("field1", DataTypes.StringType, false))
);
new Expectations() {{
RowUtils.toRowValue("one", DataTypes.StringType); result = "success";
}};
assertEquals("Invalid conversion", "success", MorphlineUtils.convertToRow(schema, record).get(0));
}
开发者ID:cloudera-labs,项目名称:envelope,代码行数:19,代码来源:TestMorphlineUtils.java
示例3: convertToRowValidNullValue
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
@Test
public void convertToRowValidNullValue(
final @Mocked RowUtils utils
) throws Exception {
Record record = new Record();
record.put("field1", null);
StructType schema = DataTypes.createStructType(Lists.newArrayList(
DataTypes.createStructField("field1", DataTypes.StringType, true))
);
assertEquals("Invalid conversion", null, MorphlineUtils.convertToRow(schema, record).get(0));
new Verifications() {{
RowUtils.toRowValue(any, (DataType) any); times = 0;
}};
}
开发者ID:cloudera-labs,项目名称:envelope,代码行数:19,代码来源:TestMorphlineUtils.java
示例4: convertToRowInvalidNullValue
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
@Test
public void convertToRowInvalidNullValue(
final @Mocked RowUtils utils
) throws Exception {
Record record = new Record();
record.put("field1", null);
StructType schema = DataTypes.createStructType(Lists.newArrayList(
DataTypes.createStructField("field1", DataTypes.StringType, false))
);
try {
MorphlineUtils.convertToRow(schema, record);
fail("Did not throw a RuntimeException");
} catch (Exception e) {
assertThat(e.getMessage(), JUnitMatchers.containsString("DataType cannot contain 'null'"));
}
new Verifications() {{
RowUtils.toRowValue(any, (DataType) any); times = 0;
}};
}
开发者ID:cloudera-labs,项目名称:envelope,代码行数:24,代码来源:TestMorphlineUtils.java
示例5: convertToRowInvalidTypeNotNullable
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
@Test
public void convertToRowInvalidTypeNotNullable(
final @Mocked RowUtils utils
) throws Exception {
Record record = new Record();
record.put("field1", "one");
StructType schema = DataTypes.createStructType(Lists.newArrayList(
DataTypes.createStructField("field1", DataTypes.StringType, false))
);
new Expectations() {{
RowUtils.toRowValue("one", DataTypes.StringType); result = new RuntimeException("Conversion exception");
}};
try {
MorphlineUtils.convertToRow(schema, record);
fail("Did not throw a RuntimeException");
} catch (Exception e) {
assertThat(e.getMessage(), JUnitMatchers.containsString("Error converting Field"));
}
}
开发者ID:cloudera-labs,项目名称:envelope,代码行数:24,代码来源:TestMorphlineUtils.java
示例6: convertToRowInvalidTypeNullable
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
@Test
public void convertToRowInvalidTypeNullable(
final @Mocked RowUtils utils
) throws Exception {
Record record = new Record();
record.put("field1", "one");
StructType schema = DataTypes.createStructType(Lists.newArrayList(
DataTypes.createStructField("field1", DataTypes.StringType, true))
);
new Expectations() {{
RowUtils.toRowValue("one", DataTypes.StringType); result = new RuntimeException("Conversion exception");
}};
try {
MorphlineUtils.convertToRow(schema, record);
fail("Did not throw a RuntimeException");
} catch (Exception e) {
assertThat(e.getMessage(), JUnitMatchers.containsString("Error converting Field"));
}
}
开发者ID:cloudera-labs,项目名称:envelope,代码行数:24,代码来源:TestMorphlineUtils.java
示例7: convertToRowMissingColumnNotNullable
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
@Test
public void convertToRowMissingColumnNotNullable(
final @Mocked RowUtils utils
) throws Exception {
Record record = new Record();
record.put("foo", "one");
StructType schema = DataTypes.createStructType(Lists.newArrayList(
DataTypes.createStructField("field1", DataTypes.StringType, false))
);
try {
MorphlineUtils.convertToRow(schema, record);
fail("Did not throw a RuntimeException");
} catch (Exception e) {
assertThat(e.getMessage(), JUnitMatchers.containsString("Error converting Record"));
}
new Verifications() {{
RowUtils.toRowValue(any, (DataType) any); times = 0;
}};
}
开发者ID:cloudera-labs,项目名称:envelope,代码行数:24,代码来源:TestMorphlineUtils.java
示例8: convertToRowMissingColumnNullable
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
@Test
public void convertToRowMissingColumnNullable(
final @Mocked RowUtils utils
) throws Exception {
Record record = new Record();
record.put("foo", "one");
StructType schema = DataTypes.createStructType(Lists.newArrayList(
DataTypes.createStructField("field1", DataTypes.StringType, true))
);
try {
MorphlineUtils.convertToRow(schema, record);
fail("Did not throw a RuntimeException");
} catch (Exception e) {
assertThat(e.getMessage(), JUnitMatchers.containsString("Error converting Record"));
}
new Verifications() {{
RowUtils.toRowValue(any, (DataType) any); times = 0;
}};
}
开发者ID:cloudera-labs,项目名称:envelope,代码行数:24,代码来源:TestMorphlineUtils.java
示例9: detectParser
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
private Parser detectParser(Record record) {
if (!hasAtLeastOneMimeType(record)) {
return null;
}
String mediaTypeStr = (String) record.getFirstValue(Fields.ATTACHMENT_MIME_TYPE); //ExtractingParams.STREAM_TYPE);
assert mediaTypeStr != null;
MediaType mediaType = parseMediaType(mediaTypeStr).getBaseType();
Parser parser = mediaTypeToParserMap.get(mediaType); // fast path
if (parser != null) {
return parser;
}
// wildcard matching
for (Map.Entry<MediaType, Parser> entry : mediaTypeToParserMap.entrySet()) {
if (isMediaTypeMatch(mediaType, entry.getKey())) {
return entry.getValue();
}
}
if (LOG.isDebugEnabled()) {
LOG.debug("No supported MIME type parser found for " + Fields.ATTACHMENT_MIME_TYPE + "=" + mediaTypeStr);
}
return null;
}
开发者ID:europeana,项目名称:search,代码行数:24,代码来源:SolrCellBuilder.java
示例10: doProcess
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
@Override
protected boolean doProcess(Record record) {
Collection<Map.Entry> entries = new ArrayList<Map.Entry>(record.getFields().asMap().entrySet());
for (Map.Entry<String, Collection<Object>> entry : entries) {
String key = entry.getKey();
if (schema.getFieldOrNull(key) == null) {
LOG.debug("Sanitizing unknown Solr field: {}", key);
Collection values = entry.getValue();
if (renameToPrefix != null) {
record.getFields().putAll(renameToPrefix + key, values);
}
values.clear(); // implicitly removes key from record
}
}
return super.doProcess(record);
}
开发者ID:europeana,项目名称:search,代码行数:17,代码来源:SanitizeUnknownSolrFieldsBuilder.java
示例11: doProcess
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
@Override
protected boolean doProcess(Record record) {
try {
List outputValues = record.get(outputFieldName);
for (Object value : record.get(inputFieldName)) {
reader.setValue(value.toString());
TokenStream tokenStream = analyzer.tokenStream("content", reader);
tokenStream.reset();
while (tokenStream.incrementToken()) {
if (token.length() > 0) { // incrementToken() updates the token!
String tokenStr = new String(token.buffer(), 0, token.length());
outputValues.add(tokenStr);
}
}
tokenStream.end();
tokenStream.close();
}
} catch (IOException e) {
throw new MorphlineRuntimeException(e);
}
// pass record to next command in chain:
return super.doProcess(record);
}
开发者ID:europeana,项目名称:search,代码行数:25,代码来源:TokenizeTextBuilder.java
示例12: testLoadSolrBasic
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
@Test
public void testLoadSolrBasic() throws Exception {
//System.setProperty("ENV_SOLR_HOME", testSolrHome + "/collection1");
morphline = createMorphline("test-morphlines/loadSolrBasic");
//System.clearProperty("ENV_SOLR_HOME");
Record record = new Record();
record.put(Fields.ID, "id0");
record.put("first_name", "Nadja"); // will be sanitized
startSession();
Notifications.notifyBeginTransaction(morphline);
assertTrue(morphline.process(record));
assertEquals(1, collector.getNumStartEvents());
Notifications.notifyCommitTransaction(morphline);
Record expected = new Record();
expected.put(Fields.ID, "id0");
assertEquals(Arrays.asList(expected), collector.getRecords());
assertEquals(1, queryResultSetSize("*:*"));
Notifications.notifyRollbackTransaction(morphline);
Notifications.notifyShutdown(morphline);
}
开发者ID:europeana,项目名称:search,代码行数:21,代码来源:SolrMorphlineTest.java
示例13: testTokenizeText
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
@Test
public void testTokenizeText() throws Exception {
morphline = createMorphline("test-morphlines" + File.separator + "tokenizeText");
for (int i = 0; i < 3; i++) {
Record record = new Record();
record.put(Fields.MESSAGE, "Hello World!");
record.put(Fields.MESSAGE, "\[email protected] #%()123");
Record expected = record.copy();
expected.getFields().putAll("tokens", Arrays.asList("hello", "world", "foo", "bar.com", "123"));
collector.reset();
startSession();
Notifications.notifyBeginTransaction(morphline);
assertTrue(morphline.process(record));
assertEquals(1, collector.getNumStartEvents());
Notifications.notifyCommitTransaction(morphline);
assertEquals(expected, collector.getFirstRecord());
}
}
开发者ID:europeana,项目名称:search,代码行数:19,代码来源:SolrMorphlineTest.java
示例14: assertTweetEquals
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
private void assertTweetEquals(GenericData.Record expected, Record actual, int i) {
Preconditions.checkNotNull(expected);
Preconditions.checkNotNull(actual);
// System.out.println("\n\nexpected: " + toString(expected));
// System.out.println("actual: " + actual);
String[] fieldNames = new String[] {
"id",
"in_reply_to_status_id",
"in_reply_to_user_id",
"retweet_count",
"text",
};
for (String fieldName : fieldNames) {
assertEquals(
i + " fieldName: " + fieldName,
expected.get(fieldName).toString(),
actual.getFirstValue(fieldName).toString());
}
}
开发者ID:europeana,项目名称:search,代码行数:20,代码来源:SolrMorphlineZkAvroTest.java
示例15: doProcess
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
@Override
protected boolean doProcess(Record record) {
ListIterator iter = record.get(fieldName).listIterator();
while (iter.hasNext()) {
String[] segments = iter.next().toString().split(separator);
iter.remove();
for (int i = 0; i < segments.length; i++) {
if (i < newFields.size()) {
record.put(newFields.get(i), trimIfNeeded(segments[i]));
} else {
if (!dropUndeclaredField) {
record.put(String.valueOf(i), trimIfNeeded(segments[i]));
}
}
}
}
return super.doProcess(record);
}
开发者ID:sequenceiq,项目名称:sequenceiq-samples,代码行数:19,代码来源:SplitBuilder.java
示例16: testAddValues
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
@Test
public void testAddValues() throws Exception {
morphline = createMorphline("test-morphlines/addValues");
Record record = new Record();
record.put("first_name", "Nadja");
Record expected = new Record();
expected.put("first_name", "Nadja");
expected.put("source_type", "text/log");
expected.put("source_type", "text/log2");
expected.put("source_host", "123");
expected.put("name", "Nadja");
expected.put("names", "@{first_name}");
expected.put("pids", 456);
expected.put("pids", "hello");
processAndVerifySuccess(record, expected);
}
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:17,代码来源:ExampleMorphlineTest.java
示例17: testGrokSyslogNgCisco
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
@Test
public void testGrokSyslogNgCisco() throws Exception {
morphline = createMorphline("test-morphlines/grokSyslogNgCisco");
Record record = new Record();
String msg = "<179>Jun 10 04:42:51 www.foo.com Jun 10 2013 04:42:51 : %myproduct-3-mysubfacility-251010: " +
"Health probe failed for server 1.2.3.4 on port 8083, connection refused by server";
record.put(Fields.MESSAGE, msg);
assertTrue(morphline.process(record));
Record expected = new Record();
expected.put(Fields.MESSAGE, msg);
expected.put("cisco_message_code", "%myproduct-3-mysubfacility-251010");
expected.put("cisco_product", "myproduct");
expected.put("cisco_level", "3");
expected.put("cisco_subfacility", "mysubfacility");
expected.put("cisco_message_id", "251010");
expected.put("syslog_message", "%myproduct-3-mysubfacility-251010: Health probe failed for server 1.2.3.4 " +
"on port 8083, connection refused by server");
assertEquals(expected, collector.getFirstRecord());
assertNotSame(record, collector.getFirstRecord());
}
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:21,代码来源:ExampleMorphlineTest.java
示例18: testGrokSyslogNgCiscoWithoutSubFacility
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
public void testGrokSyslogNgCiscoWithoutSubFacility() throws Exception {
morphline = createMorphline("test-morphlines/grokSyslogNgCisco");
Record record = new Record();
String msg = "<179>Jun 10 04:42:51 www.foo.com Jun 10 2013 04:42:51 : %myproduct-3-mysubfacility-251010: " +
"Health probe failed for server 1.2.3.4 on port 8083, connection refused by server";
record.put(Fields.MESSAGE, msg);
assertTrue(morphline.process(record));
Record expected = new Record();
expected.put(Fields.MESSAGE, msg);
expected.put("cisco_message_code", "%myproduct-3-251010");
expected.put("cisco_product", "myproduct");
expected.put("cisco_level", "3");
// expected.put("cisco_subfacility", "mysubfacility");
expected.put("cisco_message_id", "251010");
expected.put("syslog_message", "%myproduct-3-mysubfacility-251010: Health probe failed for server 1.2.3.4 " +
"on port 8083, connection refused by server");
assertEquals(expected, collector.getFirstRecord());
assertNotSame(record, collector.getFirstRecord());
}
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:20,代码来源:ExampleMorphlineTest.java
示例19: testGrokEmail
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
@Test
public void testGrokEmail() throws Exception {
morphline = createMorphline("test-morphlines/grokEmail");
Record record = new Record();
byte[] bytes = Files.toByteArray(new File(RESOURCES_DIR + "/test-documents/email.txt"));
record.put(Fields.ATTACHMENT_BODY, bytes);
assertTrue(morphline.process(record));
Record expected = new Record();
String msg = new String(bytes, "UTF-8"); //.replaceAll("(\r)?\n", "\n");
expected.put(Fields.MESSAGE, msg);
expected.put("message_id", "[email protected]");
expected.put("date", "Wed, 6 Feb 2012 06:06:05 -0800");
expected.put("from", "[email protected]");
expected.put("to", "[email protected]");
expected.put("subject", "WEDNESDAY WEATHER HEADLINES");
expected.put("from_names", "Foo Bar <[email protected]>@xxx");
expected.put("to_names", "'Weather News Distribution' <[email protected]>");
expected.put("text",
"Average 1 to 3- degrees above normal: Mid-Atlantic, Southern Plains.." +
"\nAverage 4 to 6-degrees above normal: Ohio Valley, Rockies, Central Plains");
assertEquals(expected, collector.getFirstRecord());
assertNotSame(record, collector.getFirstRecord());
}
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:24,代码来源:ExampleMorphlineTest.java
示例20: testExtractJsonPathsFlattened
import org.kitesdk.morphline.api.Record; //导入依赖的package包/类
@Test
public void testExtractJsonPathsFlattened() throws Exception {
morphline = createMorphline("test-morphlines/extractJsonPathsFlattened");
File file = new File(RESOURCES_DIR + "/test-documents/arrays.json");
InputStream in = new BufferedInputStream(new FileInputStream(file));
Record record = new Record();
record.put(Fields.ATTACHMENT_BODY, in);
startSession();
assertEquals(1, collector.getNumStartEvents());
assertTrue(morphline.process(record));
assertEquals(1, collector.getRecords().size());
List expected = Arrays.asList(1, 2, 3, 4, 5, 10, 20, 100, 200);
assertEquals(1, collector.getRecords().size());
assertEquals(expected, collector.getFirstRecord().get("/price"));
assertEquals(expected, collector.getFirstRecord().get("/price/[]"));
assertEquals(Arrays.asList(), collector.getFirstRecord().get("/unknownField"));
in.close();
}
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:22,代码来源:ExampleMorphlineTest.java
注:本文中的org.kitesdk.morphline.api.Record类示例整理自Github/MSDocs等源码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。 |
请发表评论