• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    迪恩网络公众号

Java DataStatistics类代码示例

原作者: [db:作者] 来自: [db:来源] 收藏 邀请

本文整理汇总了Java中org.apache.hadoop.mapred.gridmix.GenerateData.DataStatistics的典型用法代码示例。如果您正苦于以下问题:Java DataStatistics类的具体用法?Java DataStatistics怎么用?Java DataStatistics使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。



DataStatistics类属于org.apache.hadoop.mapred.gridmix.GenerateData包,在下文中一共展示了DataStatistics类的4个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Java代码示例。

示例1: finalize

import org.apache.hadoop.mapred.gridmix.GenerateData.DataStatistics; //导入依赖的package包/类
@SuppressWarnings("unchecked")
void finalize(JobFactory factory, String inputPath, long dataSize, 
              UserResolver userResolver, DataStatistics stats,
              Configuration conf) 
throws IOException {
  numJobsInInputTrace = factory.numJobsInTrace;
  endTime = System.currentTimeMillis();
   if ("-".equals(inputPath)) {
    inputTraceLocation = Summarizer.NA;
    inputTraceSignature = Summarizer.NA;
  } else {
    Path inputTracePath = new Path(inputPath);
    FileSystem fs = inputTracePath.getFileSystem(conf);
    inputTraceLocation = fs.makeQualified(inputTracePath).toString();
    inputTraceSignature = getTraceSignature(inputPath);
  }
  jobSubmissionPolicy = Gridmix.getJobSubmissionPolicy(conf).name();
  resolver = userResolver.getClass().getName();
  if (dataSize > 0) {
    expectedDataSize = StringUtils.humanReadableInt(dataSize);
  } else {
    expectedDataSize = Summarizer.NA;
  }
  dataStats = stats;
  totalRuntime = System.currentTimeMillis() - getStartTime();
}
 
开发者ID:naver,项目名称:hadoop,代码行数:27,代码来源:ExecutionSummarizer.java


示例2: stringifyDataStatistics

import org.apache.hadoop.mapred.gridmix.GenerateData.DataStatistics; //导入依赖的package包/类
static String stringifyDataStatistics(DataStatistics stats) {
  if (stats != null) {
    StringBuffer buffer = new StringBuffer();
    String compressionStatus = stats.isDataCompressed() 
                               ? "Compressed" 
                               : "Uncompressed";
    buffer.append(compressionStatus).append(" input data size: ");
    buffer.append(StringUtils.humanReadableInt(stats.getDataSize()));
    buffer.append(", ");
    buffer.append("Number of files: ").append(stats.getNumFiles());

    return buffer.toString();
  } else {
    return Summarizer.NA;
  }
}
 
开发者ID:naver,项目名称:hadoop,代码行数:17,代码来源:ExecutionSummarizer.java


示例3: finalize

import org.apache.hadoop.mapred.gridmix.GenerateData.DataStatistics; //导入依赖的package包/类
/**
 * This finalizes the summarizer.
 */
@SuppressWarnings("unchecked")
void finalize(JobFactory factory, String path, long size, 
              UserResolver resolver, DataStatistics stats, Configuration conf)
throws IOException {
  executionSummarizer.finalize(factory, path, size, resolver, stats, conf);
}
 
开发者ID:naver,项目名称:hadoop,代码行数:10,代码来源:Summarizer.java


示例4: publishCompressedDataStatistics

import org.apache.hadoop.mapred.gridmix.GenerateData.DataStatistics; //导入依赖的package包/类
/** Publishes compression related data statistics. Following statistics are
 * published
 * <ul>
 *   <li>Total compressed input data size</li>
 *   <li>Number of compressed input data files</li>
 *   <li>Compression Ratio</li>
 *   <li>Text data dictionary size</li>
 *   <li>Random text word size</li>
 * </ul>
 */
static DataStatistics publishCompressedDataStatistics(Path inputDir, 
                        Configuration conf, long uncompressedDataSize) 
throws IOException {
  FileSystem fs = inputDir.getFileSystem(conf);
  CompressionCodecFactory compressionCodecs = 
    new CompressionCodecFactory(conf);

  // iterate over compressed files and sum up the compressed file sizes
  long compressedDataSize = 0;
  int numCompressedFiles = 0;
  // obtain input data file statuses
  FileStatus[] outFileStatuses = 
    fs.listStatus(inputDir, new Utils.OutputFileUtils.OutputFilesFilter());
  for (FileStatus status : outFileStatuses) {
    // check if the input file is compressed
    if (compressionCodecs != null) {
      CompressionCodec codec = compressionCodecs.getCodec(status.getPath());
      if (codec != null) {
        ++numCompressedFiles;
        compressedDataSize += status.getLen();
      }
    }
  }

  LOG.info("Gridmix is configured to use compressed input data.");
  // publish the input data size
  LOG.info("Total size of compressed input data : " 
           + StringUtils.humanReadableInt(compressedDataSize));
  LOG.info("Total number of compressed input data files : " 
           + numCompressedFiles);

  if (numCompressedFiles == 0) {
    throw new RuntimeException("No compressed file found in the input" 
        + " directory : " + inputDir.toString() + ". To enable compression"
        + " emulation, run Gridmix either with "
        + " an input directory containing compressed input file(s) or" 
        + " use the -generate option to (re)generate it. If compression"
        + " emulation is not desired, disable it by setting '" 
        + COMPRESSION_EMULATION_ENABLE + "' to 'false'.");
  }
  
  // publish compression ratio only if its generated in this gridmix run
  if (uncompressedDataSize > 0) {
    // compute the compression ratio
    double ratio = ((double)compressedDataSize) / uncompressedDataSize;

    // publish the compression ratio
    LOG.info("Input Data Compression Ratio : " + ratio);
  }
  
  return new DataStatistics(compressedDataSize, numCompressedFiles, true);
}
 
开发者ID:naver,项目名称:hadoop,代码行数:63,代码来源:CompressionEmulationUtil.java



注:本文中的org.apache.hadoop.mapred.gridmix.GenerateData.DataStatistics类示例整理自Github/MSDocs等源码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。


鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
上一篇:
Java ProgressMessage类代码示例发布时间:2022-05-22
下一篇:
Java RecognizerDialogListener类代码示例发布时间:2022-05-22
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap