本文整理汇总了Java中org.apache.hadoop.mapred.gridmix.GenerateData.DataStatistics类的典型用法代码示例。如果您正苦于以下问题:Java DataStatistics类的具体用法?Java DataStatistics怎么用?Java DataStatistics使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。
DataStatistics类属于org.apache.hadoop.mapred.gridmix.GenerateData包,在下文中一共展示了DataStatistics类的4个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Java代码示例。
示例1: finalize
import org.apache.hadoop.mapred.gridmix.GenerateData.DataStatistics; //导入依赖的package包/类
@SuppressWarnings("unchecked")
void finalize(JobFactory factory, String inputPath, long dataSize,
UserResolver userResolver, DataStatistics stats,
Configuration conf)
throws IOException {
numJobsInInputTrace = factory.numJobsInTrace;
endTime = System.currentTimeMillis();
if ("-".equals(inputPath)) {
inputTraceLocation = Summarizer.NA;
inputTraceSignature = Summarizer.NA;
} else {
Path inputTracePath = new Path(inputPath);
FileSystem fs = inputTracePath.getFileSystem(conf);
inputTraceLocation = fs.makeQualified(inputTracePath).toString();
inputTraceSignature = getTraceSignature(inputPath);
}
jobSubmissionPolicy = Gridmix.getJobSubmissionPolicy(conf).name();
resolver = userResolver.getClass().getName();
if (dataSize > 0) {
expectedDataSize = StringUtils.humanReadableInt(dataSize);
} else {
expectedDataSize = Summarizer.NA;
}
dataStats = stats;
totalRuntime = System.currentTimeMillis() - getStartTime();
}
开发者ID:naver,项目名称:hadoop,代码行数:27,代码来源:ExecutionSummarizer.java
示例2: stringifyDataStatistics
import org.apache.hadoop.mapred.gridmix.GenerateData.DataStatistics; //导入依赖的package包/类
static String stringifyDataStatistics(DataStatistics stats) {
if (stats != null) {
StringBuffer buffer = new StringBuffer();
String compressionStatus = stats.isDataCompressed()
? "Compressed"
: "Uncompressed";
buffer.append(compressionStatus).append(" input data size: ");
buffer.append(StringUtils.humanReadableInt(stats.getDataSize()));
buffer.append(", ");
buffer.append("Number of files: ").append(stats.getNumFiles());
return buffer.toString();
} else {
return Summarizer.NA;
}
}
开发者ID:naver,项目名称:hadoop,代码行数:17,代码来源:ExecutionSummarizer.java
示例3: finalize
import org.apache.hadoop.mapred.gridmix.GenerateData.DataStatistics; //导入依赖的package包/类
/**
* This finalizes the summarizer.
*/
@SuppressWarnings("unchecked")
void finalize(JobFactory factory, String path, long size,
UserResolver resolver, DataStatistics stats, Configuration conf)
throws IOException {
executionSummarizer.finalize(factory, path, size, resolver, stats, conf);
}
开发者ID:naver,项目名称:hadoop,代码行数:10,代码来源:Summarizer.java
示例4: publishCompressedDataStatistics
import org.apache.hadoop.mapred.gridmix.GenerateData.DataStatistics; //导入依赖的package包/类
/** Publishes compression related data statistics. Following statistics are
* published
* <ul>
* <li>Total compressed input data size</li>
* <li>Number of compressed input data files</li>
* <li>Compression Ratio</li>
* <li>Text data dictionary size</li>
* <li>Random text word size</li>
* </ul>
*/
static DataStatistics publishCompressedDataStatistics(Path inputDir,
Configuration conf, long uncompressedDataSize)
throws IOException {
FileSystem fs = inputDir.getFileSystem(conf);
CompressionCodecFactory compressionCodecs =
new CompressionCodecFactory(conf);
// iterate over compressed files and sum up the compressed file sizes
long compressedDataSize = 0;
int numCompressedFiles = 0;
// obtain input data file statuses
FileStatus[] outFileStatuses =
fs.listStatus(inputDir, new Utils.OutputFileUtils.OutputFilesFilter());
for (FileStatus status : outFileStatuses) {
// check if the input file is compressed
if (compressionCodecs != null) {
CompressionCodec codec = compressionCodecs.getCodec(status.getPath());
if (codec != null) {
++numCompressedFiles;
compressedDataSize += status.getLen();
}
}
}
LOG.info("Gridmix is configured to use compressed input data.");
// publish the input data size
LOG.info("Total size of compressed input data : "
+ StringUtils.humanReadableInt(compressedDataSize));
LOG.info("Total number of compressed input data files : "
+ numCompressedFiles);
if (numCompressedFiles == 0) {
throw new RuntimeException("No compressed file found in the input"
+ " directory : " + inputDir.toString() + ". To enable compression"
+ " emulation, run Gridmix either with "
+ " an input directory containing compressed input file(s) or"
+ " use the -generate option to (re)generate it. If compression"
+ " emulation is not desired, disable it by setting '"
+ COMPRESSION_EMULATION_ENABLE + "' to 'false'.");
}
// publish compression ratio only if its generated in this gridmix run
if (uncompressedDataSize > 0) {
// compute the compression ratio
double ratio = ((double)compressedDataSize) / uncompressedDataSize;
// publish the compression ratio
LOG.info("Input Data Compression Ratio : " + ratio);
}
return new DataStatistics(compressedDataSize, numCompressedFiles, true);
}
开发者ID:naver,项目名称:hadoop,代码行数:63,代码来源:CompressionEmulationUtil.java
注:本文中的org.apache.hadoop.mapred.gridmix.GenerateData.DataStatistics类示例整理自Github/MSDocs等源码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。 |
请发表评论