本文整理汇总了Java中org.kitesdk.morphline.base.Fields类的典型用法代码示例。如果您正苦于以下问题:Java Fields类的具体用法?Java Fields怎么用?Java Fields使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。
Fields类属于org.kitesdk.morphline.base包,在下文中一共展示了Fields类的20个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Java代码示例。
示例1: testReadClob
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
@Test
public void testReadClob() throws Exception {
Context context = new Context();
context.put(MorphlineHandlerImpl.MORPHLINE_FILE_PARAM,
RESOURCES_DIR + "/test-morphlines/readClob.conf");
Event input = EventBuilder.withBody("foo", Charsets.UTF_8);
input.getHeaders().put("name", "nadja");
Event actual = build(context).intercept(input);
Event expected = EventBuilder.withBody(null,
ImmutableMap.of("name", "nadja", Fields.MESSAGE, "foo"));
assertEqualsEvent(expected, actual);
List<Event> actualList = build(context).intercept(Collections.singletonList(input));
List<Event> expectedList = Collections.singletonList(expected);
assertEqualsEventList(expectedList, actualList);
}
开发者ID:moueimei,项目名称:flume-release-1.7.0,代码行数:17,代码来源:TestMorphlineInterceptor.java
示例2: testGrokIfNotMatchDropEventRetain
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
@Test
public void testGrokIfNotMatchDropEventRetain() throws Exception {
Context context = new Context();
context.put(MorphlineHandlerImpl.MORPHLINE_FILE_PARAM,
RESOURCES_DIR + "/test-morphlines/grokIfNotMatchDropRecord.conf");
String msg = "<164>Feb 4 10:46:14 syslog sshd[607]: Server listening on 0.0.0.0 port 22.";
Event input = EventBuilder.withBody(null, ImmutableMap.of(Fields.MESSAGE, msg));
Event actual = build(context).intercept(input);
Map<String, String> expected = new HashMap();
expected.put(Fields.MESSAGE, msg);
expected.put("syslog_pri", "164");
expected.put("syslog_timestamp", "Feb 4 10:46:14");
expected.put("syslog_hostname", "syslog");
expected.put("syslog_program", "sshd");
expected.put("syslog_pid", "607");
expected.put("syslog_message", "Server listening on 0.0.0.0 port 22.");
Event expectedEvent = EventBuilder.withBody(null, expected);
assertEqualsEvent(expectedEvent, actual);
}
开发者ID:moueimei,项目名称:flume-release-1.7.0,代码行数:22,代码来源:TestMorphlineInterceptor.java
示例3: testIfDetectMimeTypeRouteToSouthPole
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
@Test
/** morphline says route to southpole if it's an avro file, otherwise route to northpole */
public void testIfDetectMimeTypeRouteToSouthPole() throws Exception {
Context context = new Context();
context.put(MorphlineHandlerImpl.MORPHLINE_FILE_PARAM,
RESOURCES_DIR + "/test-morphlines/ifDetectMimeType.conf");
context.put(MorphlineHandlerImpl.MORPHLINE_VARIABLE_PARAM + ".MY.MIME_TYPE", "avro/binary");
Event input = EventBuilder.withBody(Files.toByteArray(
new File(RESOURCES_DIR + "/test-documents/sample-statuses-20120906-141433.avro")));
Event actual = build(context).intercept(input);
Map<String, String> expected = new HashMap();
expected.put(Fields.ATTACHMENT_MIME_TYPE, "avro/binary");
expected.put("flume.selector.header", "goToSouthPole");
Event expectedEvent = EventBuilder.withBody(input.getBody(), expected);
assertEqualsEvent(expectedEvent, actual);
}
开发者ID:moueimei,项目名称:flume-release-1.7.0,代码行数:19,代码来源:TestMorphlineInterceptor.java
示例4: testIfDetectMimeTypeRouteToNorthPole
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
@Test
/** morphline says route to southpole if it's an avro file, otherwise route to northpole */
public void testIfDetectMimeTypeRouteToNorthPole() throws Exception {
Context context = new Context();
context.put(MorphlineHandlerImpl.MORPHLINE_FILE_PARAM,
RESOURCES_DIR + "/test-morphlines/ifDetectMimeType.conf");
context.put(MorphlineHandlerImpl.MORPHLINE_VARIABLE_PARAM + ".MY.MIME_TYPE", "avro/binary");
Event input = EventBuilder.withBody(
Files.toByteArray(new File(RESOURCES_DIR + "/test-documents/testPDF.pdf")));
Event actual = build(context).intercept(input);
Map<String, String> expected = new HashMap();
expected.put(Fields.ATTACHMENT_MIME_TYPE, "application/pdf");
expected.put("flume.selector.header", "goToNorthPole");
Event expectedEvent = EventBuilder.withBody(input.getBody(), expected);
assertEqualsEvent(expectedEvent, actual);
}
开发者ID:moueimei,项目名称:flume-release-1.7.0,代码行数:19,代码来源:TestMorphlineInterceptor.java
示例5: GenerateSolrSequenceKey
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
public GenerateSolrSequenceKey(CommandBuilder builder, Config config, Command parent, Command child, MorphlineContext context) {
super(builder, config, parent, child, context);
this.baseIdFieldName = getConfigs().getString(config, "baseIdField", Fields.BASE_ID);
this.preserveExisting = getConfigs().getBoolean(config, "preserveExisting", true);
Config solrLocatorConfig = getConfigs().getConfig(config, "solrLocator");
SolrLocator locator = new SolrLocator(solrLocatorConfig, context);
LOG.debug("solrLocator: {}", locator);
IndexSchema schema = locator.getIndexSchema();
SchemaField uniqueKey = schema.getUniqueKeyField();
uniqueKeyName = uniqueKey == null ? null : uniqueKey.getName();
String tmpIdPrefix = getConfigs().getString(config, "idPrefix", null); // for load testing only
Random tmpRandomIdPrefx = null;
if ("random".equals(tmpIdPrefix)) { // for load testing only
tmpRandomIdPrefx = new Random(new SecureRandom().nextLong());
tmpIdPrefix = null;
}
idPrefix = tmpIdPrefix;
randomIdPrefix = tmpRandomIdPrefx;
validateArguments();
}
开发者ID:europeana,项目名称:search,代码行数:23,代码来源:GenerateSolrSequenceKeyBuilder.java
示例6: testLoadSolrBasic
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
@Test
public void testLoadSolrBasic() throws Exception {
//System.setProperty("ENV_SOLR_HOME", testSolrHome + "/collection1");
morphline = createMorphline("test-morphlines/loadSolrBasic");
//System.clearProperty("ENV_SOLR_HOME");
Record record = new Record();
record.put(Fields.ID, "id0");
record.put("first_name", "Nadja"); // will be sanitized
startSession();
Notifications.notifyBeginTransaction(morphline);
assertTrue(morphline.process(record));
assertEquals(1, collector.getNumStartEvents());
Notifications.notifyCommitTransaction(morphline);
Record expected = new Record();
expected.put(Fields.ID, "id0");
assertEquals(Arrays.asList(expected), collector.getRecords());
assertEquals(1, queryResultSetSize("*:*"));
Notifications.notifyRollbackTransaction(morphline);
Notifications.notifyShutdown(morphline);
}
开发者ID:europeana,项目名称:search,代码行数:21,代码来源:SolrMorphlineTest.java
示例7: testTokenizeText
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
@Test
public void testTokenizeText() throws Exception {
morphline = createMorphline("test-morphlines" + File.separator + "tokenizeText");
for (int i = 0; i < 3; i++) {
Record record = new Record();
record.put(Fields.MESSAGE, "Hello World!");
record.put(Fields.MESSAGE, "\[email protected] #%()123");
Record expected = record.copy();
expected.getFields().putAll("tokens", Arrays.asList("hello", "world", "foo", "bar.com", "123"));
collector.reset();
startSession();
Notifications.notifyBeginTransaction(morphline);
assertTrue(morphline.process(record));
assertEquals(1, collector.getNumStartEvents());
Notifications.notifyCommitTransaction(morphline);
assertEquals(expected, collector.getFirstRecord());
}
}
开发者ID:europeana,项目名称:search,代码行数:19,代码来源:SolrMorphlineTest.java
示例8: testGrokSyslogNgCisco
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
@Test
public void testGrokSyslogNgCisco() throws Exception {
morphline = createMorphline("test-morphlines/grokSyslogNgCisco");
Record record = new Record();
String msg = "<179>Jun 10 04:42:51 www.foo.com Jun 10 2013 04:42:51 : %myproduct-3-mysubfacility-251010: " +
"Health probe failed for server 1.2.3.4 on port 8083, connection refused by server";
record.put(Fields.MESSAGE, msg);
assertTrue(morphline.process(record));
Record expected = new Record();
expected.put(Fields.MESSAGE, msg);
expected.put("cisco_message_code", "%myproduct-3-mysubfacility-251010");
expected.put("cisco_product", "myproduct");
expected.put("cisco_level", "3");
expected.put("cisco_subfacility", "mysubfacility");
expected.put("cisco_message_id", "251010");
expected.put("syslog_message", "%myproduct-3-mysubfacility-251010: Health probe failed for server 1.2.3.4 " +
"on port 8083, connection refused by server");
assertEquals(expected, collector.getFirstRecord());
assertNotSame(record, collector.getFirstRecord());
}
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:21,代码来源:ExampleMorphlineTest.java
示例9: testGrokSyslogNgCiscoWithoutSubFacility
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
public void testGrokSyslogNgCiscoWithoutSubFacility() throws Exception {
morphline = createMorphline("test-morphlines/grokSyslogNgCisco");
Record record = new Record();
String msg = "<179>Jun 10 04:42:51 www.foo.com Jun 10 2013 04:42:51 : %myproduct-3-mysubfacility-251010: " +
"Health probe failed for server 1.2.3.4 on port 8083, connection refused by server";
record.put(Fields.MESSAGE, msg);
assertTrue(morphline.process(record));
Record expected = new Record();
expected.put(Fields.MESSAGE, msg);
expected.put("cisco_message_code", "%myproduct-3-251010");
expected.put("cisco_product", "myproduct");
expected.put("cisco_level", "3");
// expected.put("cisco_subfacility", "mysubfacility");
expected.put("cisco_message_id", "251010");
expected.put("syslog_message", "%myproduct-3-mysubfacility-251010: Health probe failed for server 1.2.3.4 " +
"on port 8083, connection refused by server");
assertEquals(expected, collector.getFirstRecord());
assertNotSame(record, collector.getFirstRecord());
}
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:20,代码来源:ExampleMorphlineTest.java
示例10: testGrokEmail
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
@Test
public void testGrokEmail() throws Exception {
morphline = createMorphline("test-morphlines/grokEmail");
Record record = new Record();
byte[] bytes = Files.toByteArray(new File(RESOURCES_DIR + "/test-documents/email.txt"));
record.put(Fields.ATTACHMENT_BODY, bytes);
assertTrue(morphline.process(record));
Record expected = new Record();
String msg = new String(bytes, "UTF-8"); //.replaceAll("(\r)?\n", "\n");
expected.put(Fields.MESSAGE, msg);
expected.put("message_id", "[email protected]");
expected.put("date", "Wed, 6 Feb 2012 06:06:05 -0800");
expected.put("from", "[email protected]");
expected.put("to", "[email protected]");
expected.put("subject", "WEDNESDAY WEATHER HEADLINES");
expected.put("from_names", "Foo Bar <[email protected]>@xxx");
expected.put("to_names", "'Weather News Distribution' <[email protected]>");
expected.put("text",
"Average 1 to 3- degrees above normal: Mid-Atlantic, Southern Plains.." +
"\nAverage 4 to 6-degrees above normal: Ohio Valley, Rockies, Central Plains");
assertEquals(expected, collector.getFirstRecord());
assertNotSame(record, collector.getFirstRecord());
}
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:24,代码来源:ExampleMorphlineTest.java
示例11: testExtractJsonPathsFlattened
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
@Test
public void testExtractJsonPathsFlattened() throws Exception {
morphline = createMorphline("test-morphlines/extractJsonPathsFlattened");
File file = new File(RESOURCES_DIR + "/test-documents/arrays.json");
InputStream in = new BufferedInputStream(new FileInputStream(file));
Record record = new Record();
record.put(Fields.ATTACHMENT_BODY, in);
startSession();
assertEquals(1, collector.getNumStartEvents());
assertTrue(morphline.process(record));
assertEquals(1, collector.getRecords().size());
List expected = Arrays.asList(1, 2, 3, 4, 5, 10, 20, 100, 200);
assertEquals(1, collector.getRecords().size());
assertEquals(expected, collector.getFirstRecord().get("/price"));
assertEquals(expected, collector.getFirstRecord().get("/price/[]"));
assertEquals(Arrays.asList(), collector.getFirstRecord().get("/unknownField"));
in.close();
}
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:22,代码来源:ExampleMorphlineTest.java
示例12: benchmarkJson
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
@Test
@Ignore
/** Crude quick n' dirty benchmark */
// Before running this disable debug logging
// via log4j.logger.org.kitesdk.morphline=INFO in log4j.properties
public void benchmarkJson() throws Exception {
String morphlineConfigFile = "test-morphlines/readJson";
long durationSecs = 20;
//File file = new File(RESOURCES_DIR + "/test-documents/stream.json");
File file = new File(RESOURCES_DIR + "/test-documents/sample-statuses-20120906-141433.json");
System.out.println("Now benchmarking " + morphlineConfigFile + " ...");
morphline = createMorphline(morphlineConfigFile);
byte[] bytes = Files.toByteArray(file);
long start = System.currentTimeMillis();
long duration = durationSecs * 1000;
int iters = 0;
while (System.currentTimeMillis() < start + duration) {
Record record = new Record();
record.put(Fields.ATTACHMENT_BODY, bytes);
collector.reset();
assertTrue(morphline.process(record));
iters++;
}
float secs = (System.currentTimeMillis() - start) / 1000.0f;
System.out.println("Results: iters=" + iters + ", took[secs]=" + secs + ", iters/secs=" + (iters/secs));
}
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:27,代码来源:ExampleMorphlineTest.java
示例13: process
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
@Override
public void process(Event event) {
numRecords.mark();
Timer.Context timerContext = mappingTimer.time();
try {
Record record = new Record();
for (Entry<String, String> entry : event.getHeaders().entrySet()) {
record.put(entry.getKey(), entry.getValue());
}
byte[] bytes = event.getBody();
if (bytes != null && bytes.length > 0) {
record.put(Fields.ATTACHMENT_BODY, bytes);
}
try {
Notifications.notifyStartSession(morphline);
if (!morphline.process(record)) {
numFailedRecords.mark();
LOG.warn("Morphline {} failed to process record: {}", morphlineFileAndId, record);
}
} catch (RuntimeException t) {
numExceptionRecords.mark();
morphlineContext.getExceptionHandler().handleException(t, record);
}
} finally {
timerContext.stop();
}
}
开发者ID:moueimei,项目名称:flume-release-1.7.0,代码行数:28,代码来源:MorphlineHandlerImpl.java
示例14: testGrokIfNotMatchDropEventDrop
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
@Test
/* leading XXXXX does not match regex, thus we expect the event to be dropped */
public void testGrokIfNotMatchDropEventDrop() throws Exception {
Context context = new Context();
context.put(MorphlineHandlerImpl.MORPHLINE_FILE_PARAM,
RESOURCES_DIR + "/test-morphlines/grokIfNotMatchDropRecord.conf");
String msg = "<XXXXXXXXXXXXX164>Feb 4 10:46:14 syslog sshd[607]: Server listening on 0.0.0.0" +
" port 22.";
Event input = EventBuilder.withBody(null, ImmutableMap.of(Fields.MESSAGE, msg));
Event actual = build(context).intercept(input);
assertNull(actual);
}
开发者ID:moueimei,项目名称:flume-release-1.7.0,代码行数:13,代码来源:TestMorphlineInterceptor.java
示例15: testDocumentTypesInternal
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
private void testDocumentTypesInternal(String... files) throws Exception {
int numDocs = 0;
long startTime = System.currentTimeMillis();
assertEquals(numDocs, queryResultSetSize("*:*"));
// assertQ(req("*:*"), "//*[@numFound='0']");
for (int i = 0; i < 1; i++) {
for (String file : files) {
File f = new File(file);
byte[] body = Files.toByteArray(f);
Event event = EventBuilder.withBody(body);
event.getHeaders().put(Fields.ATTACHMENT_NAME, f.getName());
load(event);
Integer count = expectedRecords.get(file);
if (count != null) {
numDocs += count;
} else {
numDocs++;
}
assertEquals(numDocs, queryResultSetSize("*:*"));
}
LOGGER.trace("iter: {}", i);
}
LOGGER.trace("all done with put at {}", System.currentTimeMillis() - startTime);
assertEquals(numDocs, queryResultSetSize("*:*"));
LOGGER.trace("sink: ", sink);
}
开发者ID:moueimei,项目名称:flume-release-1.7.0,代码行数:28,代码来源:TestMorphlineSolrSink.java
示例16: hasAtLeastOneMimeType
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
private boolean hasAtLeastOneMimeType(Record record) {
if (!record.getFields().containsKey(Fields.ATTACHMENT_MIME_TYPE)) {
LOG.debug("Command failed because of missing MIME type for record: {}", record);
return false;
}
return true;
}
开发者ID:europeana,项目名称:search,代码行数:8,代码来源:SolrCellBuilder.java
示例17: getRecord
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
protected Record getRecord(PathParts parts) {
FileStatus stats;
try {
stats = parts.getFileStatus();
} catch (IOException e) {
stats = null;
}
if (stats == null) {
LOG.warn("Ignoring file that somehow has become unavailable since the job was submitted: {}",
parts.getUploadURL());
return null;
}
Record headers = new Record();
//headers.put(getSchema().getUniqueKeyField().getName(), parts.getId()); // use HDFS file path as docId if no docId is specified
headers.put(Fields.BASE_ID, parts.getId()); // with sanitizeUniqueKey command, use HDFS file path as docId if no docId is specified
headers.put(Fields.ATTACHMENT_NAME, parts.getName()); // Tika can use the file name in guessing the right MIME type
// enable indexing and storing of file meta data in Solr
headers.put(HdfsFileFieldNames.FILE_UPLOAD_URL, parts.getUploadURL());
headers.put(HdfsFileFieldNames.FILE_DOWNLOAD_URL, parts.getDownloadURL());
headers.put(HdfsFileFieldNames.FILE_SCHEME, parts.getScheme());
headers.put(HdfsFileFieldNames.FILE_HOST, parts.getHost());
headers.put(HdfsFileFieldNames.FILE_PORT, String.valueOf(parts.getPort()));
headers.put(HdfsFileFieldNames.FILE_PATH, parts.getURIPath());
headers.put(HdfsFileFieldNames.FILE_NAME, parts.getName());
headers.put(HdfsFileFieldNames.FILE_LAST_MODIFIED, String.valueOf(stats.getModificationTime())); // FIXME also add in SpoolDirectorySource
headers.put(HdfsFileFieldNames.FILE_LENGTH, String.valueOf(stats.getLen())); // FIXME also add in SpoolDirectorySource
headers.put(HdfsFileFieldNames.FILE_OWNER, stats.getOwner());
headers.put(HdfsFileFieldNames.FILE_GROUP, stats.getGroup());
headers.put(HdfsFileFieldNames.FILE_PERMISSIONS_USER, stats.getPermission().getUserAction().SYMBOL);
headers.put(HdfsFileFieldNames.FILE_PERMISSIONS_GROUP, stats.getPermission().getGroupAction().SYMBOL);
headers.put(HdfsFileFieldNames.FILE_PERMISSIONS_OTHER, stats.getPermission().getOtherAction().SYMBOL);
headers.put(HdfsFileFieldNames.FILE_PERMISSIONS_STICKYBIT, String.valueOf(stats.getPermission().getStickyBit()));
// TODO: consider to add stats.getAccessTime(), stats.getReplication(), stats.isSymlink(), stats.getBlockSize()
return headers;
}
开发者ID:europeana,项目名称:search,代码行数:39,代码来源:MorphlineMapRunner.java
示例18: map
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
record.put(Fields.ATTACHMENT_BODY, new ByteArrayInputStream(value.toString().getBytes()));
if (!morphline.process(record)) {
LOGGER.info("Morphline failed to process record: {}", record);
}
record.removeAll(Fields.ATTACHMENT_BODY);
}
开发者ID:sequenceiq,项目名称:sequenceiq-samples,代码行数:8,代码来源:MapperCleaner.java
示例19: assertResult
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
protected Record assertResult(String morphlineSpecification, String inputFile) throws IOException {
morphline = createMorphline(morphlineSpecification);
InputStream in = new FileInputStream(new File(RESOURCES_DIR + inputFile));
Record record = new Record();
record.put(Fields.ATTACHMENT_BODY, in);
return record;
}
开发者ID:sequenceiq,项目名称:sequenceiq-samples,代码行数:8,代码来源:LatestSongCommandTest.java
示例20: testSimpleCSV
import org.kitesdk.morphline.base.Fields; //导入依赖的package包/类
@Test
public void testSimpleCSV() throws Exception {
morphline = createMorphline("test-morphlines/simpleCSV");
Notifications.notifyBeginTransaction(morphline);
InputStream in = new FileInputStream(new File(RESOURCES_DIR + "/test-documents/simpleCSV.txt"));
Record record = new Record();
record.put(Fields.ATTACHMENT_BODY, in);
record.put(Fields.ATTACHMENT_MIME_TYPE, "text/plain");
// Actually process the input file.
assertTrue(morphline.process(record));
assertEquals(collector.getRecords().size(), 2);
Record rec = collector.getRecords().get(0);
// Since id and timestamp vary with run, just see if they have anything in them
assertTrue(rec.get("id").toString().length() > 5);
assertTrue(rec.get("timestamp").toString().length() > 5);
assertEquals(rec.get("text").toString(), "[text for body]");
// Now look at second record
rec = collector.getRecords().get(1);
assertTrue(rec.get("id").toString().length() > 5);
assertTrue(rec.get("timestamp").toString().length() > 5);
assertEquals(rec.get("text").toString(), "[second record]");
in.close();
Notifications.notifyCommitTransaction(morphline);
}
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:33,代码来源:ExampleMorphlineTest.java
注:本文中的org.kitesdk.morphline.base.Fields类示例整理自Github/MSDocs等源码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。 |
请发表评论