Java MROutputFiles类代码示例

OGeek|极客世界-中国程序员成长平台 › 门户 › 编程› Java›Java编程经验

原作者: [db:作者] 来自: [db:来源] 收藏邀请

本文整理汇总了Java中org.apache.hadoop.mapred.MROutputFiles类的典型用法代码示例。如果您正苦于以下问题：Java MROutputFiles类的具体用法？Java MROutputFiles怎么用？Java MROutputFiles使用的例子？那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。

MROutputFiles类属于org.apache.hadoop.mapred包，在下文中一共展示了MROutputFiles类的5个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于我们的系统推荐出更棒的Java代码示例。

示例1: testLargeMemoryLimits

import org.apache.hadoop.mapred.MROutputFiles; //导入依赖的package包/类
@Test
public void testLargeMemoryLimits() throws Exception {
  final JobConf conf = new JobConf();
  // Xmx in production
  conf.setLong(MRJobConfig.REDUCE_MEMORY_TOTAL_BYTES,
      8L * 1024 * 1024 * 1024);

  // M1 = Xmx fraction for map outputs
  conf.setFloat(MRJobConfig.SHUFFLE_INPUT_BUFFER_PERCENT, 1.0f);

  // M2 = max M1 fraction for a single maple output
  conf.setFloat(MRJobConfig.SHUFFLE_MEMORY_LIMIT_PERCENT, 0.95f);

  // M3 = M1 fraction at which in memory merge is triggered
  conf.setFloat(MRJobConfig.SHUFFLE_MERGE_PERCENT, 1.0f);

  // M4 = M1 fraction of map outputs remaining in memory for a reduce
  conf.setFloat(MRJobConfig.REDUCE_INPUT_BUFFER_PERCENT, 1.0f);

  final MergeManagerImpl<Text, Text> mgr = new MergeManagerImpl<Text, Text>(
      null, conf, mock(LocalFileSystem.class), null, null, null, null, null,
      null, null, null, null, null, new MROutputFiles());
  assertTrue("Large shuffle area unusable: " + mgr.memoryLimit,
      mgr.memoryLimit > Integer.MAX_VALUE);
  final long maxInMemReduce = mgr.getMaxInMemReduceLimit();
  assertTrue("Large in-memory reduce area unusable: " + maxInMemReduce,
      maxInMemReduce > Integer.MAX_VALUE);
}

开发者ID:aliyun-beta，项目名称:aliyun-oss-hadoop-fs，代码行数:29，代码来源:TestMergeManager.java

示例2: testLargeMemoryLimits

import org.apache.hadoop.mapred.MROutputFiles; //导入依赖的package包/类
@Test
public void testLargeMemoryLimits() throws Exception {
  final JobConf conf = new JobConf();
  // Xmx in production
  conf.setLong(MRJobConfig.REDUCE_MEMORY_TOTAL_BYTES,
      8L * 1024 * 1024 * 1024);

  // M1 = Xmx fraction for map outputs
  conf.setFloat(MRJobConfig.SHUFFLE_INPUT_BUFFER_PERCENT, 1.0f);

  // M2 = max M1 fraction for a single maple output
  conf.setFloat(MRJobConfig.SHUFFLE_MEMORY_LIMIT_PERCENT, 0.95f);

  // M3 = M1 fraction at which in memory merge is triggered
  conf.setFloat(MRJobConfig.SHUFFLE_MERGE_PERCENT, 1.0f);

  // M4 = M1 fraction of map outputs remaining in memory for a reduce
  conf.setFloat(MRJobConfig.REDUCE_INPUT_BUFFER_PERCENT, 1.0f);

  final MergeManagerImpl<Text, Text> mgr = new MergeManagerImpl<Text, Text>(
      null, conf, mock(LocalFileSystem.class), null, null, null, null, null,
      null, null, null, null, null, new MROutputFiles());
  assertTrue("Large shuffle area unusable: " + mgr.memoryLimit,
      mgr.memoryLimit > Integer.MAX_VALUE);
  final long maxInMemReduce = mgr.getMaxInMemReduceLimit();
  assertTrue("Large in-memory reduce area unusable: " + maxInMemReduce,
      maxInMemReduce > Integer.MAX_VALUE);
  assertEquals("maxSingleShuffleLimit to be capped at Integer.MAX_VALUE",
      Integer.MAX_VALUE, mgr.maxSingleShuffleLimit);
  verifyReservedMapOutputType(mgr, 10L, "MEMORY");
  verifyReservedMapOutputType(mgr, 1L + Integer.MAX_VALUE, "DISK");
}

开发者ID:hopshadoop，项目名称:hops，代码行数:33，代码来源:TestMergeManager.java

示例3: testOnDiskMerger

import org.apache.hadoop.mapred.MROutputFiles; //导入依赖的package包/类
@SuppressWarnings({ "unchecked", "deprecation" })
@Test(timeout=10000)
public void testOnDiskMerger() throws IOException, URISyntaxException,
  InterruptedException {
  JobConf jobConf = new JobConf();
  final int SORT_FACTOR = 5;
  jobConf.setInt(MRJobConfig.IO_SORT_FACTOR, SORT_FACTOR);

  MapOutputFile mapOutputFile = new MROutputFiles();
  FileSystem fs = FileSystem.getLocal(jobConf);
  MergeManagerImpl<IntWritable, IntWritable> manager =
    new MergeManagerImpl<IntWritable, IntWritable>(null, jobConf, fs, null
      , null, null, null, null, null, null, null, null, null, mapOutputFile);

  MergeThread<MapOutput<IntWritable, IntWritable>, IntWritable, IntWritable>
    onDiskMerger = (MergeThread<MapOutput<IntWritable, IntWritable>,
      IntWritable, IntWritable>) Whitebox.getInternalState(manager,
        "onDiskMerger");
  int mergeFactor = (Integer) Whitebox.getInternalState(onDiskMerger,
    "mergeFactor");

  // make sure the io.sort.factor is set properly
  assertEquals(mergeFactor, SORT_FACTOR);

  // Stop the onDiskMerger thread so that we can intercept the list of files
  // waiting to be merged.
  onDiskMerger.suspend();

  //Send the list of fake files waiting to be merged
  Random rand = new Random();
  for(int i = 0; i < 2*SORT_FACTOR; ++i) {
    Path path = new Path("somePath");
    CompressAwarePath cap = new CompressAwarePath(path, 1l, rand.nextInt());
    manager.closeOnDiskFile(cap);
  }

  //Check that the files pending to be merged are in sorted order.
  LinkedList<List<CompressAwarePath>> pendingToBeMerged =
    (LinkedList<List<CompressAwarePath>>) Whitebox.getInternalState(
      onDiskMerger, "pendingToBeMerged");
  assertTrue("No inputs were added to list pending to merge",
    pendingToBeMerged.size() > 0);
  for(int i = 0; i < pendingToBeMerged.size(); ++i) {
    List<CompressAwarePath> inputs = pendingToBeMerged.get(i);
    for(int j = 1; j < inputs.size(); ++j) {
      assertTrue("Not enough / too many inputs were going to be merged",
        inputs.size() > 0 && inputs.size() <= SORT_FACTOR);
      assertTrue("Inputs to be merged were not sorted according to size: ",
        inputs.get(j).getCompressedSize()
        >= inputs.get(j-1).getCompressedSize());
    }
  }

}

开发者ID:naver，项目名称:hadoop，代码行数:55，代码来源:TestMergeManager.java

示例4: testInMemoryMerger

import org.apache.hadoop.mapred.MROutputFiles; //导入依赖的package包/类
@Test
public void testInMemoryMerger() throws IOException {
  JobID jobId = new JobID("a", 0);
  TaskAttemptID reduceId = new TaskAttemptID(
      new TaskID(jobId, TaskType.REDUCE, 0), 0);
  TaskAttemptID mapId1 = new TaskAttemptID(
      new TaskID(jobId, TaskType.MAP, 1), 0);
  TaskAttemptID mapId2 = new TaskAttemptID(
      new TaskID(jobId, TaskType.MAP, 2), 0);
  
  LocalDirAllocator lda = new LocalDirAllocator(MRConfig.LOCAL_DIR);
  
  MergeManagerImpl<Text, Text> mergeManager = new MergeManagerImpl<Text, Text>(
      reduceId, jobConf, fs, lda, Reporter.NULL, null, null, null, null, null,
      null, null, new Progress(), new MROutputFiles());
  
  // write map outputs
  Map<String, String> map1 = new TreeMap<String, String>();
  map1.put("apple", "disgusting");
  map1.put("carrot", "delicious");
  Map<String, String> map2 = new TreeMap<String, String>();
  map1.put("banana", "pretty good");
  byte[] mapOutputBytes1 = writeMapOutput(conf, map1);
  byte[] mapOutputBytes2 = writeMapOutput(conf, map2);
  InMemoryMapOutput<Text, Text> mapOutput1 = new InMemoryMapOutput<Text, Text>(
      conf, mapId1, mergeManager, mapOutputBytes1.length, null, true);
  InMemoryMapOutput<Text, Text> mapOutput2 = new InMemoryMapOutput<Text, Text>(
      conf, mapId2, mergeManager, mapOutputBytes2.length, null, true);
  System.arraycopy(mapOutputBytes1, 0, mapOutput1.getMemory(), 0,
      mapOutputBytes1.length);
  System.arraycopy(mapOutputBytes2, 0, mapOutput2.getMemory(), 0,
      mapOutputBytes2.length);
  
  // create merger and run merge
  MergeThread<InMemoryMapOutput<Text, Text>, Text, Text> inMemoryMerger =
      mergeManager.createInMemoryMerger();
  List<InMemoryMapOutput<Text, Text>> mapOutputs =
      new ArrayList<InMemoryMapOutput<Text, Text>>();
  mapOutputs.add(mapOutput1);
  mapOutputs.add(mapOutput2);
  
  inMemoryMerger.merge(mapOutputs);
  
  Assert.assertEquals(1, mergeManager.onDiskMapOutputs.size());
  Path outPath = mergeManager.onDiskMapOutputs.iterator().next();
  
  List<String> keys = new ArrayList<String>();
  List<String> values = new ArrayList<String>();
  readOnDiskMapOutput(conf, fs, outPath, keys, values);
  Assert.assertEquals(keys, Arrays.asList("apple", "banana", "carrot"));
  Assert.assertEquals(values, Arrays.asList("disgusting", "pretty good", "delicious"));
}

开发者ID:ict-carch，项目名称:hadoop-plus，代码行数:53，代码来源:TestMerger.java

示例5: testInMemoryMerger

import org.apache.hadoop.mapred.MROutputFiles; //导入依赖的package包/类
@Test
public void testInMemoryMerger() throws Throwable {
  JobID jobId = new JobID("a", 0);
  TaskAttemptID reduceId = new TaskAttemptID(
      new TaskID(jobId, TaskType.REDUCE, 0), 0);
  TaskAttemptID mapId1 = new TaskAttemptID(
      new TaskID(jobId, TaskType.MAP, 1), 0);
  TaskAttemptID mapId2 = new TaskAttemptID(
      new TaskID(jobId, TaskType.MAP, 2), 0);
  
  LocalDirAllocator lda = new LocalDirAllocator(MRConfig.LOCAL_DIR);
  
  MergeManagerImpl<Text, Text> mergeManager = new MergeManagerImpl<Text, Text>(
      reduceId, jobConf, fs, lda, Reporter.NULL, null, null, null, null, null,
      null, null, new Progress(), new MROutputFiles());
  
  // write map outputs
  Map<String, String> map1 = new TreeMap<String, String>();
  map1.put("apple", "disgusting");
  map1.put("carrot", "delicious");
  Map<String, String> map2 = new TreeMap<String, String>();
  map1.put("banana", "pretty good");
  byte[] mapOutputBytes1 = writeMapOutput(conf, map1);
  byte[] mapOutputBytes2 = writeMapOutput(conf, map2);
  InMemoryMapOutput<Text, Text> mapOutput1 = new InMemoryMapOutput<Text, Text>(
      conf, mapId1, mergeManager, mapOutputBytes1.length, null, true);
  InMemoryMapOutput<Text, Text> mapOutput2 = new InMemoryMapOutput<Text, Text>(
      conf, mapId2, mergeManager, mapOutputBytes2.length, null, true);
  System.arraycopy(mapOutputBytes1, 0, mapOutput1.getMemory(), 0,
      mapOutputBytes1.length);
  System.arraycopy(mapOutputBytes2, 0, mapOutput2.getMemory(), 0,
      mapOutputBytes2.length);
  
  // create merger and run merge
  MergeThread<InMemoryMapOutput<Text, Text>, Text, Text> inMemoryMerger =
      mergeManager.createInMemoryMerger();
  List<InMemoryMapOutput<Text, Text>> mapOutputs =
      new ArrayList<InMemoryMapOutput<Text, Text>>();
  mapOutputs.add(mapOutput1);
  mapOutputs.add(mapOutput2);
  
  inMemoryMerger.merge(mapOutputs);
  
  Assert.assertEquals(1, mergeManager.onDiskMapOutputs.size());
  Path outPath = mergeManager.onDiskMapOutputs.iterator().next();
  
  List<String> keys = new ArrayList<String>();
  List<String> values = new ArrayList<String>();
  readOnDiskMapOutput(conf, fs, outPath, keys, values);
  Assert.assertEquals(keys, Arrays.asList("apple", "banana", "carrot"));
  Assert.assertEquals(values, Arrays.asList("disgusting", "pretty good", "delicious"));

  mergeManager.close();
  Assert.assertEquals(0, mergeManager.inMemoryMapOutputs.size());
  Assert.assertEquals(0, mergeManager.inMemoryMergedMapOutputs.size());
  Assert.assertEquals(0, mergeManager.onDiskMapOutputs.size());
}

开发者ID:chendave，项目名称:hadoop-TCP，代码行数:58，代码来源:TestMerger.java

注：本文中的org.apache.hadoop.mapred.MROutputFiles类示例整理自Github/MSDocs等源码及文档管理平台，相关代码片段筛选自各路编程大神贡献的开源项目，源码版权归原作者所有，传播和使用请参考对应项目的License；未经允许，请勿转载。

鲜花

握手

雷人

路过

鸡蛋

该文章已有0人参与评论

请发表评论

全部评论

专题导读

More+

10-27 六六分期app的软件客服如何联系？(六六分期

11-06 可心卡盟:win10系统火狐flash插件崩溃怎么

11-06 亲亲特价:怎么删除回收站图标

11-06 济南大学虚拟社区:鲁大师节能降温的具体办

11-06 xlueops.exe:无线网络安装向导

11-06 女斗合众国:win7系统cf与主机连接不稳定怎

11-06 0xc000022-[cf烟雾头]cf怎么调烟雾头

11-06 qizideyouhuo:应用程序无法正常启动0xc0000

11-06 ipz-185:win7系统vcf文件怎么打开

11-06 傻哥蹦迪:win10系统s4怎么打开usb调试

11-06 八神浩树gtaste:回收站清空了怎么恢复

11-06 妖尾之黑色守护:win10系统电脑没有1440x900

11-06 校园至尊魔王小说:win7系统浏览网页时字体

11-06 女斗合众国:win10系统访问共享文件夹提示请

11-06 tokyo hot n0654:恢复win7系统默认字体一招

11-06 雨酷仙境:设置win7系统转移临时文件夹腾出

11-06 阿穆纳伊之杖:win7系统开始菜单在右边还原

11-06 tunespotting:win10系统火狐flash插件总是

11-06 甘尔葛分析师：计谋网站seo关键词暴涨有什

11-06 蔡贵霖: 计谋网站seo关键词暴涨有什么秘密

11-06 博益网首页:ao3网页版进入不了解决方法

11-06 漏斗子专栏: 网站数据分析小白易懂精华篇

11-06 见证双虹怎么做:win7系统开启telnet命令的

11-06 颾狐蝶蜋:系统资源不足无法完成请求的服务

11-06 国光中学校歌:提交网站到alexa查询详细步骤

11-06 西安有情天:静态网页和动态网页的区别

11-06 红木雅尚斋:外部链接构造对网站的好处

11-06 前官礼遇：防止域名劫持–增强域安全性的10

11-06 密传二转答案: 中文分词算法有哪些

11-06 金泉家园邮编:百度快照劫持的表现及应对方

Java GeneratorBase类代码示例发布时间：2022-05-22

Java DexBackedClassDef类代码示例发布时间：2022-05-22

剪的笔顺,诠释剪的笔画,认识剪的部首

1 六六分期app的软件客服如何联系？(六六分期

六六分期app的软件客服如何联系？不知道吗？加qq群【895510560】即可！标题：六六分期

阅读：18165|2023-10-27

2 可心卡盟:win10系统火狐flash插件崩溃怎么

今天小编告诉大家如何处理win10系统火狐flash插件总是崩溃的问题，可能很多用户都不知

阅读：9639|2022-11-06

3 亲亲特价:怎么删除回收站图标

今天小编告诉大家如何对win10系统删除桌面回收站图标进行设置，可能很多用户都不知道

阅读：8163|2022-11-06

4 济南大学虚拟社区:鲁大师节能降温的具体办

今天小编告诉大家如何对win10系统电脑设置节能降温的设置方法，想必大家都遇到过需要

阅读：8541|2022-11-06

5 xlueops.exe:无线网络安装向导

我们在使用xp系统的过程中,经常需要对xp系统无线网络安装向导设置进行设置，可能很多

阅读：8443|2022-11-06

6 女斗合众国:win7系统cf与主机连接不稳定怎

今天小编告诉大家如何处理win7系统玩cf老是与主机连接不稳定的问题，可能很多用户都不

阅读：9363|2022-11-06

7 0xc000022-[cf烟雾头]cf怎么调烟雾头

电脑对日常生活的重要性小编就不多说了，可是一旦碰到win7系统设置cf烟雾头的问题，很

阅读：8413|2022-11-06

8 qizideyouhuo:应用程序无法正常启动0xc0000

我们在日常使用电脑的时候，有的小伙伴们可能在打开应用的时候会遇见提示应用程序无法

阅读：7850|2022-11-06

9 ipz-185:win7系统vcf文件怎么打开

今天小编告诉大家如何对win7系统打开vcf文件进行设置，可能很多用户都不知道怎么对win

阅读：8397|2022-11-06

10 傻哥蹦迪:win10系统s4怎么打开usb调试

今天小编告诉大家如何对win10系统s4开启USB调试模式进行设置，可能很多用户都不知道怎

阅读：7389|2022-11-06

客服电话

电子邮件

Java MROutputFiles类代码示例

示例1: testLargeMemoryLimits

示例2: testLargeMemoryLimits

示例3: testOnDiskMerger

示例4: testInMemoryMerger

示例5: testInMemoryMerger

请发表评论

全部评论

上一篇：

下一篇：

CVE-2022-33677

apache/apisix-ingress-controller: APISIX

librespeed/speedtest: Self-hosted Speedt

avehtari/BDA_m_demos: Bayesian Data Anal

四维彩超怎么看性别？四维看男孩女孩诀窍

剪的笔顺,诠释剪的笔画,认识剪的部首

六六分期app的软件客服如何联系？(六六分期

florent37/ViewAnimator: A fluent Android

florent37/Shrine-MaterialDesign2: implem

CVE-2020-36276

SimpleSoftwareIO/simple-sms: Send and re

关于我们

产品与服务

解决方案

139-2527-9053