• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    迪恩网络公众号

Java SchedulerAppReport类代码示例

原作者: [db:作者] 来自: [db:来源] 收藏 邀请

本文整理汇总了Java中org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport的典型用法代码示例。如果您正苦于以下问题:Java SchedulerAppReport类的具体用法?Java SchedulerAppReport怎么用?Java SchedulerAppReport使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。



SchedulerAppReport类属于org.apache.hadoop.yarn.server.resourcemanager.scheduler包,在下文中一共展示了SchedulerAppReport类的20个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Java代码示例。

示例1: checkTaskContainersHost

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
private void checkTaskContainersHost(ApplicationAttemptId attemptId,
    ContainerId containerId, ResourceManager rm, String host) {
  YarnScheduler scheduler = rm.getRMContext().getScheduler();
  SchedulerAppReport appReport = scheduler.getSchedulerAppInfo(attemptId);

  Assert.assertTrue(appReport.getLiveContainers().size() > 0);
  for (RMContainer c : appReport.getLiveContainers()) {
    if (c.getContainerId().equals(containerId)) {
      Assert.assertEquals(host, c.getAllocatedNode().getHost());
    }
  }
}
 
开发者ID:naver,项目名称:hadoop,代码行数:13,代码来源:TestContainerAllocation.java


示例2: getApplicationResourceUsageReport

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
@Override
public ApplicationResourceUsageReport getApplicationResourceUsageReport() {
  this.readLock.lock();
  
  try {
    int numUsedContainers = 0;
    int numReservedContainers = 0;
    Resource currentConsumption = Resources.createResource(0, 0);
    Resource reservedResources = Resources.createResource(0, 0);
    
    SchedulerAppReport schedApp = 
        scheduler.getSchedulerAppInfo(this.getAppAttemptId());
    Collection<RMContainer> liveContainers;
    Collection<RMContainer> reservedContainers;
    if (schedApp != null) {
      liveContainers = schedApp.getLiveContainers();
      reservedContainers = schedApp.getReservedContainers();
      if (liveContainers != null) {
        numUsedContainers = liveContainers.size();
        for (RMContainer lc : liveContainers) {
          Resources.addTo(currentConsumption, lc.getContainer().getResource());
        }
      }
      if (reservedContainers != null) {
        numReservedContainers = reservedContainers.size();
        for (RMContainer rc : reservedContainers) {
          Resources.addTo(reservedResources, rc.getContainer().getResource());
        }
      }
    }

    return BuilderUtils.newApplicationResourceUsageReport(
        numUsedContainers, numReservedContainers,
        currentConsumption, reservedResources,
        Resources.add(currentConsumption, reservedResources));
  } finally {
    this.readLock.unlock();
  }
}
 
开发者ID:ict-carch,项目名称:hadoop-plus,代码行数:40,代码来源:RMAppAttemptImpl.java


示例3: getSchedulerAppInfo

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
@Override
public SchedulerAppReport getSchedulerAppInfo(
    ApplicationAttemptId appAttemptId) {
  if (!applications.containsKey(appAttemptId)) {
    LOG.error("Request for appInfo of unknown attempt" + appAttemptId);
    return null;
  }
  return new SchedulerAppReport(applications.get(appAttemptId));
}
 
开发者ID:ict-carch,项目名称:hadoop-plus,代码行数:10,代码来源:FairScheduler.java


示例4: getSchedulerAppInfo

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
@Override
public SchedulerAppReport getSchedulerAppInfo(
    ApplicationAttemptId appAttemptId) {
  FSSchedulerApp attempt = getSchedulerApp(appAttemptId);
  if (attempt == null) {
    if (LOG.isDebugEnabled()) {
      LOG.debug("Request for appInfo of unknown attempt " + appAttemptId);
    }
    return null;
  }
  return new SchedulerAppReport(attempt);
}
 
开发者ID:Seagate,项目名称:hadoop-on-lustre2,代码行数:13,代码来源:FairScheduler.java


示例5: getContainers

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
@Override
public GetContainersResponse getContainers(GetContainersRequest request)
    throws YarnException, IOException {
  ApplicationAttemptId appAttemptId = request.getApplicationAttemptId();
  ApplicationId appId = appAttemptId.getApplicationId();
  UserGroupInformation callerUGI;
  try {
    callerUGI = UserGroupInformation.getCurrentUser();
  } catch (IOException ie) {
    LOG.info("Error getting UGI ", ie);
    throw RPCUtil.getRemoteException(ie);
  }
  RMApp application = this.rmContext.getRMApps().get(appId);
  if (application == null) {
    // If the RM doesn't have the application, throw
    // ApplicationNotFoundException and let client to handle.
    throw new ApplicationNotFoundException("Application with id '" + appId
        + "' doesn't exist in RM.");
  }
  boolean allowAccess = checkAccess(callerUGI, application.getUser(),
      ApplicationAccessType.VIEW_APP, application);
  GetContainersResponse response = null;
  if (allowAccess) {
    RMAppAttempt appAttempt = application.getAppAttempts().get(appAttemptId);
    if (appAttempt == null) {
      throw new ApplicationAttemptNotFoundException(
          "ApplicationAttempt with id '" + appAttemptId +
          "' doesn't exist in RM.");
    }
    Collection<RMContainer> rmContainers = Collections.emptyList();
    SchedulerAppReport schedulerAppReport =
        this.rmContext.getScheduler().getSchedulerAppInfo(appAttemptId);
    if (schedulerAppReport != null) {
      rmContainers = schedulerAppReport.getLiveContainers();
    }
    List<ContainerReport> listContainers = new ArrayList<ContainerReport>();
    for (RMContainer rmContainer : rmContainers) {
      listContainers.add(rmContainer.createContainerReport());
    }
    response = GetContainersResponse.newInstance(listContainers);
  } else {
    throw new YarnException("User " + callerUGI.getShortUserName()
        + " does not have privilage to see this aplication " + appId);
  }
  return response;
}
 
开发者ID:naver,项目名称:hadoop,代码行数:47,代码来源:ClientRMService.java


示例6: testNodeLocalAssignment

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
@Test(timeout=2000)
public void testNodeLocalAssignment() throws Exception {
  AsyncDispatcher dispatcher = new InlineDispatcher();
  Configuration conf = new Configuration();
  RMContainerTokenSecretManager containerTokenSecretManager =
      new RMContainerTokenSecretManager(conf);
  containerTokenSecretManager.rollMasterKey();
  NMTokenSecretManagerInRM nmTokenSecretManager =
      new NMTokenSecretManagerInRM(conf);
  nmTokenSecretManager.rollMasterKey();
  RMApplicationHistoryWriter writer = mock(RMApplicationHistoryWriter.class);
  
  FifoScheduler scheduler = new FifoScheduler();
  RMContext rmContext = new RMContextImpl(dispatcher, null, null, null, null,
      null, containerTokenSecretManager, nmTokenSecretManager, null, writer,
      scheduler);
  ((RMContextImpl) rmContext).setSystemMetricsPublisher(
      mock(SystemMetricsPublisher.class));

  scheduler.setRMContext(rmContext);
  scheduler.init(conf);
  scheduler.start();
  scheduler.reinitialize(new Configuration(), rmContext);

  RMNode node0 = MockNodes.newNodeInfo(1,
      Resources.createResource(1024 * 64), 1, "127.0.0.1");
  NodeAddedSchedulerEvent nodeEvent1 = new NodeAddedSchedulerEvent(node0);
  scheduler.handle(nodeEvent1);

  int _appId = 1;
  int _appAttemptId = 1;
  ApplicationAttemptId appAttemptId = createAppAttemptId(_appId,
      _appAttemptId);

  createMockRMApp(appAttemptId, rmContext);

  AppAddedSchedulerEvent appEvent =
      new AppAddedSchedulerEvent(appAttemptId.getApplicationId(), "queue1",
          "user1");
  scheduler.handle(appEvent);
  AppAttemptAddedSchedulerEvent attemptEvent =
      new AppAttemptAddedSchedulerEvent(appAttemptId, false);
  scheduler.handle(attemptEvent);

  int memory = 64;
  int nConts = 3;
  int priority = 20;

  List<ResourceRequest> ask = new ArrayList<ResourceRequest>();
  ResourceRequest nodeLocal = createResourceRequest(memory,
      node0.getHostName(), priority, nConts);
  ResourceRequest rackLocal = createResourceRequest(memory,
      node0.getRackName(), priority, nConts);
  ResourceRequest any = createResourceRequest(memory, ResourceRequest.ANY, priority,
      nConts);
  ask.add(nodeLocal);
  ask.add(rackLocal);
  ask.add(any);
  scheduler.allocate(appAttemptId, ask, new ArrayList<ContainerId>(), null, null);

  NodeUpdateSchedulerEvent node0Update = new NodeUpdateSchedulerEvent(node0);

  // Before the node update event, there are 3 local requests outstanding
  Assert.assertEquals(3, nodeLocal.getNumContainers());

  scheduler.handle(node0Update);

  // After the node update event, check that there are no more local requests
  // outstanding
  Assert.assertEquals(0, nodeLocal.getNumContainers());
  //Also check that the containers were scheduled
  SchedulerAppReport info = scheduler.getSchedulerAppInfo(appAttemptId);
  Assert.assertEquals(3, info.getLiveContainers().size());
  scheduler.stop();
}
 
开发者ID:naver,项目名称:hadoop,代码行数:76,代码来源:TestFifoScheduler.java


示例7: getRMApp

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
private RMAppImpl getRMApp(RMContext rmContext, YarnScheduler yarnScheduler,
    ApplicationId applicationId3, YarnConfiguration config, String queueName,
    final long memorySeconds, final long vcoreSeconds, final long gcoreSeconds) {
  ApplicationSubmissionContext asContext = mock(ApplicationSubmissionContext.class);
  when(asContext.getMaxAppAttempts()).thenReturn(1);

  RMAppImpl app =
      spy(new RMAppImpl(applicationId3, rmContext, config, null, null,
          queueName, asContext, yarnScheduler, null,
          System.currentTimeMillis(), "YARN", null,
          BuilderUtils.newResourceRequest(
              RMAppAttemptImpl.AM_CONTAINER_PRIORITY, ResourceRequest.ANY,
              Resource.newInstance(1024, 1, 1), 1)){
                @Override
                public ApplicationReport createAndGetApplicationReport(
                    String clientUserName, boolean allowAccess) {
                  ApplicationReport report = super.createAndGetApplicationReport(
                      clientUserName, allowAccess);
                  ApplicationResourceUsageReport usageReport = 
                      report.getApplicationResourceUsageReport();
                  usageReport.setMemorySeconds(memorySeconds);
                  usageReport.setVcoreSeconds(vcoreSeconds);
                  usageReport.setGcoreSeconds(gcoreSeconds);
                  report.setApplicationResourceUsageReport(usageReport);
                  return report;
                }
            });

  ApplicationAttemptId attemptId = ApplicationAttemptId.newInstance(
      ApplicationId.newInstance(123456, 1), 1);
  RMAppAttemptImpl rmAppAttemptImpl = spy(new RMAppAttemptImpl(attemptId,
      rmContext, yarnScheduler, null, asContext, config, false, null));
  Container container = Container.newInstance(
      ContainerId.newContainerId(attemptId, 1), null, "", null, null, null);
  RMContainerImpl containerimpl = spy(new RMContainerImpl(container,
      attemptId, null, "", rmContext));
  Map<ApplicationAttemptId, RMAppAttempt> attempts = 
    new HashMap<ApplicationAttemptId, RMAppAttempt>();
  attempts.put(attemptId, rmAppAttemptImpl);
  when(app.getCurrentAppAttempt()).thenReturn(rmAppAttemptImpl);
  when(app.getAppAttempts()).thenReturn(attempts);
  when(rmAppAttemptImpl.getMasterContainer()).thenReturn(container);
  ResourceScheduler rs = mock(ResourceScheduler.class);
  when(rmContext.getScheduler()).thenReturn(rs);
  when(rmContext.getScheduler().getRMContainer(any(ContainerId.class)))
      .thenReturn(containerimpl);
  SchedulerAppReport sAppReport = mock(SchedulerAppReport.class);
  when(
      rmContext.getScheduler().getSchedulerAppInfo(
          any(ApplicationAttemptId.class))).thenReturn(sAppReport);
  List<RMContainer> rmContainers = new ArrayList<RMContainer>();
  rmContainers.add(containerimpl);
  when(
      rmContext.getScheduler().getSchedulerAppInfo(attemptId)
          .getLiveContainers()).thenReturn(rmContainers);
  ContainerStatus cs = mock(ContainerStatus.class);
  when(containerimpl.getFinishedStatus()).thenReturn(cs);
  when(containerimpl.getDiagnosticsInfo()).thenReturn("N/A");
  when(containerimpl.getContainerExitStatus()).thenReturn(0);
  when(containerimpl.getContainerState()).thenReturn(ContainerState.COMPLETE);
  return app;
}
 
开发者ID:naver,项目名称:hadoop,代码行数:63,代码来源:TestClientRMService.java


示例8: updateQueueWithNodeUpdate

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
private void updateQueueWithNodeUpdate(
        NodeUpdateSchedulerEventWrapper eventWrapper) {
  RMNodeWrapper node = (RMNodeWrapper) eventWrapper.getRMNode();
  List<UpdatedContainerInfo> containerList = node.getContainerUpdates();
  for (UpdatedContainerInfo info : containerList) {
    for (ContainerStatus status : info.getCompletedContainers()) {
      ContainerId containerId = status.getContainerId();
      SchedulerAppReport app = scheduler.getSchedulerAppInfo(
              containerId.getApplicationAttemptId());

      if (app == null) {
        // this happens for the AM container
        // The app have already removed when the NM sends the release
        // information.
        continue;
      }

      String queue =
          appQueueMap.get(containerId.getApplicationAttemptId()
            .getApplicationId());
      int releasedMemory = 0, releasedVCores = 0;
      if (status.getExitStatus() == ContainerExitStatus.SUCCESS) {
        for (RMContainer rmc : app.getLiveContainers()) {
          if (rmc.getContainerId() == containerId) {
            releasedMemory += rmc.getContainer().getResource().getMemory();
            releasedVCores += rmc.getContainer()
                    .getResource().getVirtualCores();
            break;
          }
        }
      } else if (status.getExitStatus() == ContainerExitStatus.ABORTED) {
        if (preemptionContainerMap.containsKey(containerId)) {
          Resource preResource = preemptionContainerMap.get(containerId);
          releasedMemory += preResource.getMemory();
          releasedVCores += preResource.getVirtualCores();
          preemptionContainerMap.remove(containerId);
        }
      }
      // update queue counters
      updateQueueMetrics(queue, releasedMemory, releasedVCores);
    }
  }
}
 
开发者ID:naver,项目名称:hadoop,代码行数:44,代码来源:ResourceSchedulerWrapper.java


示例9: getSchedulerAppInfo

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
@Override
public SchedulerAppReport getSchedulerAppInfo(
        ApplicationAttemptId attemptId) {
  return scheduler.getSchedulerAppInfo(attemptId);
}
 
开发者ID:naver,项目名称:hadoop,代码行数:6,代码来源:ResourceSchedulerWrapper.java


示例10: testNodeLocalAssignment

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
@Test(timeout=2000)
public void testNodeLocalAssignment() throws Exception {
  AsyncDispatcher dispatcher = new InlineDispatcher();
  Configuration conf = new Configuration();
  RMContainerTokenSecretManager containerTokenSecretManager =
      new RMContainerTokenSecretManager(conf);
  containerTokenSecretManager.rollMasterKey();
  NMTokenSecretManagerInRM nmTokenSecretManager =
      new NMTokenSecretManagerInRM(conf);
  nmTokenSecretManager.rollMasterKey();
  RMApplicationHistoryWriter writer = mock(RMApplicationHistoryWriter.class);
  
  FifoScheduler scheduler = new FifoScheduler();
  RMContext rmContext = new RMContextImpl(dispatcher, null, null, null, null,
      null, containerTokenSecretManager, nmTokenSecretManager, null, scheduler);
  rmContext.setSystemMetricsPublisher(mock(SystemMetricsPublisher.class));
  rmContext.setRMApplicationHistoryWriter(
      mock(RMApplicationHistoryWriter.class));
  ((RMContextImpl) rmContext).setYarnConfiguration(new YarnConfiguration());

  scheduler.setRMContext(rmContext);
  scheduler.init(conf);
  scheduler.start();
  scheduler.reinitialize(new Configuration(), rmContext);

  RMNode node0 = MockNodes.newNodeInfo(1,
      Resources.createResource(1024 * 64), 1, "127.0.0.1");
  NodeAddedSchedulerEvent nodeEvent1 = new NodeAddedSchedulerEvent(node0);
  scheduler.handle(nodeEvent1);

  int _appId = 1;
  int _appAttemptId = 1;
  ApplicationAttemptId appAttemptId = createAppAttemptId(_appId,
      _appAttemptId);

  createMockRMApp(appAttemptId, rmContext);

  AppAddedSchedulerEvent appEvent =
      new AppAddedSchedulerEvent(appAttemptId.getApplicationId(), "queue1",
          "user1");
  scheduler.handle(appEvent);
  AppAttemptAddedSchedulerEvent attemptEvent =
      new AppAttemptAddedSchedulerEvent(appAttemptId, false);
  scheduler.handle(attemptEvent);

  int memory = 64;
  int nConts = 3;
  int priority = 20;

  List<ResourceRequest> ask = new ArrayList<ResourceRequest>();
  ResourceRequest nodeLocal = createResourceRequest(memory,
      node0.getHostName(), priority, nConts);
  ResourceRequest rackLocal = createResourceRequest(memory,
      node0.getRackName(), priority, nConts);
  ResourceRequest any = createResourceRequest(memory, ResourceRequest.ANY, priority,
      nConts);
  ask.add(nodeLocal);
  ask.add(rackLocal);
  ask.add(any);
  scheduler.allocate(appAttemptId, ask, new ArrayList<ContainerId>(), null, null, null, null);

  NodeUpdateSchedulerEvent node0Update = new NodeUpdateSchedulerEvent(node0);

  // Before the node update event, there are 3 local requests outstanding
  Assert.assertEquals(3, nodeLocal.getNumContainers());

  scheduler.handle(node0Update);

  // After the node update event, check that there are no more local requests
  // outstanding
  Assert.assertEquals(0, nodeLocal.getNumContainers());
  //Also check that the containers were scheduled
  SchedulerAppReport info = scheduler.getSchedulerAppInfo(appAttemptId);
  Assert.assertEquals(3, info.getLiveContainers().size());
  scheduler.stop();
}
 
开发者ID:aliyun-beta,项目名称:aliyun-oss-hadoop-fs,代码行数:77,代码来源:TestFifoScheduler.java


示例11: getRMApp

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
private RMAppImpl getRMApp(RMContext rmContext, YarnScheduler yarnScheduler,
    ApplicationId applicationId3, YarnConfiguration config, String queueName,
    final long memorySeconds, final long vcoreSeconds,
    String appNodeLabelExpression, String amNodeLabelExpression) {
  ApplicationSubmissionContext asContext = mock(ApplicationSubmissionContext.class);
  when(asContext.getMaxAppAttempts()).thenReturn(1);
  when(asContext.getNodeLabelExpression()).thenReturn(appNodeLabelExpression);
  RMAppImpl app =
      spy(new RMAppImpl(applicationId3, rmContext, config, null, null,
          queueName, asContext, yarnScheduler, null,
          System.currentTimeMillis(), "YARN", null,
          BuilderUtils.newResourceRequest(
              RMAppAttemptImpl.AM_CONTAINER_PRIORITY, ResourceRequest.ANY,
              Resource.newInstance(1024, 1), 1)){
                @Override
                public ApplicationReport createAndGetApplicationReport(
                    String clientUserName, boolean allowAccess) {
                  ApplicationReport report = super.createAndGetApplicationReport(
                      clientUserName, allowAccess);
                  ApplicationResourceUsageReport usageReport = 
                      report.getApplicationResourceUsageReport();
                  usageReport.setMemorySeconds(memorySeconds);
                  usageReport.setVcoreSeconds(vcoreSeconds);
                  report.setApplicationResourceUsageReport(usageReport);
                  return report;
                }
            });
  app.getAMResourceRequest().setNodeLabelExpression(amNodeLabelExpression);
  ApplicationAttemptId attemptId = ApplicationAttemptId.newInstance(
      ApplicationId.newInstance(123456, 1), 1);
  RMAppAttemptImpl rmAppAttemptImpl = spy(new RMAppAttemptImpl(attemptId,
      rmContext, yarnScheduler, null, asContext, config, false, null));
  Container container = Container.newInstance(
      ContainerId.newContainerId(attemptId, 1), null, "", null, null, null);
  RMContainerImpl containerimpl = spy(new RMContainerImpl(container,
      attemptId, null, "", rmContext));
  Map<ApplicationAttemptId, RMAppAttempt> attempts = 
    new HashMap<ApplicationAttemptId, RMAppAttempt>();
  attempts.put(attemptId, rmAppAttemptImpl);
  when(app.getCurrentAppAttempt()).thenReturn(rmAppAttemptImpl);
  when(app.getAppAttempts()).thenReturn(attempts);
  when(rmAppAttemptImpl.getMasterContainer()).thenReturn(container);
  ResourceScheduler rs = mock(ResourceScheduler.class);
  when(rmContext.getScheduler()).thenReturn(rs);
  when(rmContext.getScheduler().getRMContainer(any(ContainerId.class)))
      .thenReturn(containerimpl);
  SchedulerAppReport sAppReport = mock(SchedulerAppReport.class);
  when(
      rmContext.getScheduler().getSchedulerAppInfo(
          any(ApplicationAttemptId.class))).thenReturn(sAppReport);
  List<RMContainer> rmContainers = new ArrayList<RMContainer>();
  rmContainers.add(containerimpl);
  when(
      rmContext.getScheduler().getSchedulerAppInfo(attemptId)
          .getLiveContainers()).thenReturn(rmContainers);
  ContainerStatus cs = mock(ContainerStatus.class);
  when(containerimpl.getFinishedStatus()).thenReturn(cs);
  when(containerimpl.getDiagnosticsInfo()).thenReturn("N/A");
  when(containerimpl.getContainerExitStatus()).thenReturn(0);
  when(containerimpl.getContainerState()).thenReturn(ContainerState.COMPLETE);
  return app;
}
 
开发者ID:aliyun-beta,项目名称:aliyun-oss-hadoop-fs,代码行数:63,代码来源:TestClientRMService.java


示例12: updateQueueWithNodeUpdate

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
private void updateQueueWithNodeUpdate(
        NodeUpdateSchedulerEventWrapper eventWrapper) {
  RMNodeWrapper node = (RMNodeWrapper) eventWrapper.getRMNode();
  List<UpdatedContainerInfo> containerList = node.getContainerUpdates();
  for (UpdatedContainerInfo info : containerList) {
    for (ContainerStatus status : info.getCompletedContainers()) {
      ContainerId containerId = status.getContainerId();
      SchedulerAppReport app = super.getSchedulerAppInfo(
              containerId.getApplicationAttemptId());

      if (app == null) {
        // this happens for the AM container
        // The app have already removed when the NM sends the release
        // information.
        continue;
      }

      String queue = appQueueMap.get(containerId.getApplicationAttemptId());
      int releasedMemory = 0, releasedVCores = 0;
      if (status.getExitStatus() == ContainerExitStatus.SUCCESS) {
        for (RMContainer rmc : app.getLiveContainers()) {
          if (rmc.getContainerId() == containerId) {
            releasedMemory += rmc.getContainer().getResource().getMemory();
            releasedVCores += rmc.getContainer()
                    .getResource().getVirtualCores();
            break;
          }
        }
      } else if (status.getExitStatus() == ContainerExitStatus.ABORTED) {
        if (preemptionContainerMap.containsKey(containerId)) {
          Resource preResource = preemptionContainerMap.get(containerId);
          releasedMemory += preResource.getMemory();
          releasedVCores += preResource.getVirtualCores();
          preemptionContainerMap.remove(containerId);
        }
      }
      // update queue counters
      updateQueueMetrics(queue, releasedMemory, releasedVCores);
    }
  }
}
 
开发者ID:aliyun-beta,项目名称:aliyun-oss-hadoop-fs,代码行数:42,代码来源:SLSCapacityScheduler.java


示例13: getRMApp

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
private RMAppImpl getRMApp(RMContext rmContext, YarnScheduler yarnScheduler,
    ApplicationId applicationId3, YarnConfiguration config, String queueName,
    final long memorySeconds, final long vcoreSeconds) {
  ApplicationSubmissionContext asContext = mock(ApplicationSubmissionContext.class);
  when(asContext.getMaxAppAttempts()).thenReturn(1);

  RMAppImpl app =
      spy(new RMAppImpl(applicationId3, rmContext, config, null, null,
          queueName, asContext, yarnScheduler, null,
          System.currentTimeMillis(), "YARN", null,
          BuilderUtils.newResourceRequest(
              RMAppAttemptImpl.AM_CONTAINER_PRIORITY, ResourceRequest.ANY,
              Resource.newInstance(1024, 1), 1)){
                @Override
                public ApplicationReport createAndGetApplicationReport(
                    String clientUserName, boolean allowAccess) {
                  ApplicationReport report = super.createAndGetApplicationReport(
                      clientUserName, allowAccess);
                  ApplicationResourceUsageReport usageReport = 
                      report.getApplicationResourceUsageReport();
                  usageReport.setMemorySeconds(memorySeconds);
                  usageReport.setVcoreSeconds(vcoreSeconds);
                  report.setApplicationResourceUsageReport(usageReport);
                  return report;
                }
            });

  ApplicationAttemptId attemptId = ApplicationAttemptId.newInstance(
      ApplicationId.newInstance(123456, 1), 1);
  RMAppAttemptImpl rmAppAttemptImpl = spy(new RMAppAttemptImpl(attemptId,
      rmContext, yarnScheduler, null, asContext, config, false, null));
  Container container = Container.newInstance(
      ContainerId.newContainerId(attemptId, 1), null, "", null, null, null);
  RMContainerImpl containerimpl = spy(new RMContainerImpl(container,
      attemptId, null, "", rmContext));
  Map<ApplicationAttemptId, RMAppAttempt> attempts = 
    new HashMap<ApplicationAttemptId, RMAppAttempt>();
  attempts.put(attemptId, rmAppAttemptImpl);
  when(app.getCurrentAppAttempt()).thenReturn(rmAppAttemptImpl);
  when(app.getAppAttempts()).thenReturn(attempts);
  when(rmAppAttemptImpl.getMasterContainer()).thenReturn(container);
  ResourceScheduler rs = mock(ResourceScheduler.class);
  when(rmContext.getScheduler()).thenReturn(rs);
  when(rmContext.getScheduler().getRMContainer(any(ContainerId.class)))
      .thenReturn(containerimpl);
  SchedulerAppReport sAppReport = mock(SchedulerAppReport.class);
  when(
      rmContext.getScheduler().getSchedulerAppInfo(
          any(ApplicationAttemptId.class))).thenReturn(sAppReport);
  List<RMContainer> rmContainers = new ArrayList<RMContainer>();
  rmContainers.add(containerimpl);
  when(
      rmContext.getScheduler().getSchedulerAppInfo(attemptId)
          .getLiveContainers()).thenReturn(rmContainers);
  ContainerStatus cs = mock(ContainerStatus.class);
  when(containerimpl.getFinishedStatus()).thenReturn(cs);
  when(containerimpl.getDiagnosticsInfo()).thenReturn("N/A");
  when(containerimpl.getContainerExitStatus()).thenReturn(0);
  when(containerimpl.getContainerState()).thenReturn(ContainerState.COMPLETE);
  return app;
}
 
开发者ID:yncxcw,项目名称:big-c,代码行数:62,代码来源:TestClientRMService.java


示例14: getContainers

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
@Override
public GetContainersResponse getContainers(GetContainersRequest request)
    throws YarnException, IOException {
  ApplicationAttemptId appAttemptId = request.getApplicationAttemptId();
  ApplicationId appId = appAttemptId.getApplicationId();
  UserGroupInformation callerUGI;
  try {
    callerUGI = UserGroupInformation.getCurrentUser();
  } catch (IOException ie) {
    LOG.info("Error getting UGI ", ie);
    throw RPCUtil.getRemoteException(ie);
  }
  RMApp application = this.rmContext.getRMApps().get(appId);
  if (application == null) {
    // If the RM doesn't have the application, throw
    // ApplicationNotFoundException and let client to handle.
    throw new ApplicationNotFoundException("Application with id '" + appId
        + "' doesn't exist in RM.");
  }
  boolean allowAccess = checkAccess(callerUGI, application.getUser(),
      ApplicationAccessType.VIEW_APP, application);
  GetContainersResponse response = null;
  if (allowAccess) {
    RMAppAttempt appAttempt = application.getAppAttempts().get(appAttemptId);
    if (appAttempt == null) {
      throw new ApplicationAttemptNotFoundException("ApplicationAttempt "
          + appAttemptId + " Not Found in RM");
    }
    Collection<RMContainer> rmContainers = Collections.emptyList();
    SchedulerAppReport schedulerAppReport =
        this.rmContext.getScheduler().getSchedulerAppInfo(appAttemptId);
    if (schedulerAppReport != null) {
      rmContainers = schedulerAppReport.getLiveContainers();
    }
    List<ContainerReport> listContainers = new ArrayList<ContainerReport>();
    for (RMContainer rmContainer : rmContainers) {
      listContainers.add(rmContainer.createContainerReport());
    }
    response = GetContainersResponse.newInstance(listContainers);
  } else {
    throw new YarnException("User " + callerUGI.getShortUserName()
        + " does not have privilage to see this aplication " + appId);
  }
  return response;
}
 
开发者ID:Nextzero,项目名称:hadoop-2.6.0-cdh5.4.3,代码行数:46,代码来源:ClientRMService.java


示例15: testNodeLocalAssignment

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
@Test(timeout=2000)
public void testNodeLocalAssignment() throws Exception {
  AsyncDispatcher dispatcher = new InlineDispatcher();
  Configuration conf = new Configuration();
  RMContainerTokenSecretManager containerTokenSecretManager =
      new RMContainerTokenSecretManager(conf);
  containerTokenSecretManager.rollMasterKey();
  NMTokenSecretManagerInRM nmTokenSecretManager =
      new NMTokenSecretManagerInRM(conf);
  nmTokenSecretManager.rollMasterKey();
  RMApplicationHistoryWriter writer = mock(RMApplicationHistoryWriter.class);
  RMContext rmContext = new RMContextImpl(dispatcher, null, null, null, null,
      null, containerTokenSecretManager, nmTokenSecretManager, null, writer);
  ((RMContextImpl) rmContext).setSystemMetricsPublisher(
      mock(SystemMetricsPublisher.class));

  FifoScheduler scheduler = new FifoScheduler();
  scheduler.setRMContext(rmContext);
  scheduler.init(conf);
  scheduler.start();
  scheduler.reinitialize(new Configuration(), rmContext);

  RMNode node0 = MockNodes.newNodeInfo(1,
      Resources.createResource(1024 * 64), 1, "127.0.0.1");
  NodeAddedSchedulerEvent nodeEvent1 = new NodeAddedSchedulerEvent(node0);
  scheduler.handle(nodeEvent1);

  int _appId = 1;
  int _appAttemptId = 1;
  ApplicationAttemptId appAttemptId = createAppAttemptId(_appId,
      _appAttemptId);
  AppAddedSchedulerEvent appEvent =
      new AppAddedSchedulerEvent(appAttemptId.getApplicationId(), "queue1",
        "user1");
  scheduler.handle(appEvent);
  AppAttemptAddedSchedulerEvent attemptEvent =
      new AppAttemptAddedSchedulerEvent(appAttemptId, false);
  scheduler.handle(attemptEvent);

  int memory = 64;
  int nConts = 3;
  int priority = 20;

  List<ResourceRequest> ask = new ArrayList<ResourceRequest>();
  ResourceRequest nodeLocal = createResourceRequest(memory,
      node0.getHostName(), priority, nConts);
  ResourceRequest rackLocal = createResourceRequest(memory,
      node0.getRackName(), priority, nConts);
  ResourceRequest any = createResourceRequest(memory, ResourceRequest.ANY, priority,
      nConts);
  ask.add(nodeLocal);
  ask.add(rackLocal);
  ask.add(any);
  scheduler.allocate(appAttemptId, ask, new ArrayList<ContainerId>(), null, null);

  NodeUpdateSchedulerEvent node0Update = new NodeUpdateSchedulerEvent(node0);

  // Before the node update event, there are 3 local requests outstanding
  Assert.assertEquals(3, nodeLocal.getNumContainers());

  scheduler.handle(node0Update);

  // After the node update event, check that there are no more local requests
  // outstanding
  Assert.assertEquals(0, nodeLocal.getNumContainers());
  //Also check that the containers were scheduled
  SchedulerAppReport info = scheduler.getSchedulerAppInfo(appAttemptId);
  Assert.assertEquals(3, info.getLiveContainers().size());
  scheduler.stop();
}
 
开发者ID:Nextzero,项目名称:hadoop-2.6.0-cdh5.4.3,代码行数:71,代码来源:TestFifoScheduler.java


示例16: getSchedulerAppInfo

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
@Override
public SchedulerAppReport getSchedulerAppInfo(
    ApplicationAttemptId applicationAttemptId) {
  FiCaSchedulerApp app = getApplication(applicationAttemptId);
  return app == null ? null : new SchedulerAppReport(app);
}
 
开发者ID:ict-carch,项目名称:hadoop-plus,代码行数:7,代码来源:FifoScheduler.java


示例17: testNodeLocalAssignment

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
@Test(timeout=2000)
public void testNodeLocalAssignment() throws Exception {
  AsyncDispatcher dispatcher = new InlineDispatcher();
  Configuration conf = new Configuration();
  RMContainerTokenSecretManager containerTokenSecretManager =
      new RMContainerTokenSecretManager(conf);
  containerTokenSecretManager.rollMasterKey();
  NMTokenSecretManagerInRM nmTokenSecretManager =
      new NMTokenSecretManagerInRM(conf);
  nmTokenSecretManager.rollMasterKey();
  RMContext rmContext = new RMContextImpl(dispatcher, null, null, null, null,
      null, containerTokenSecretManager, nmTokenSecretManager, null);

  FifoScheduler scheduler = new FifoScheduler();
  scheduler.reinitialize(new Configuration(), rmContext);

  RMNode node0 = MockNodes.newNodeInfo(1,
      Resources.createResource(1024 * 64), 1, "127.0.0.1");
  NodeAddedSchedulerEvent nodeEvent1 = new NodeAddedSchedulerEvent(node0);
  scheduler.handle(nodeEvent1);

  int _appId = 1;
  int _appAttemptId = 1;
  ApplicationAttemptId appAttemptId = createAppAttemptId(_appId,
      _appAttemptId);
  AppAddedSchedulerEvent appEvent1 = new AppAddedSchedulerEvent(appAttemptId,
      "queue1", "user1");
  scheduler.handle(appEvent1);

  int memory = 64;
  int nConts = 3;
  int priority = 20;

  List<ResourceRequest> ask = new ArrayList<ResourceRequest>();
  ResourceRequest nodeLocal = createResourceRequest(memory,
      node0.getHostName(), priority, nConts);
  ResourceRequest rackLocal = createResourceRequest(memory,
      node0.getRackName(), priority, nConts);
  ResourceRequest any = createResourceRequest(memory, ResourceRequest.ANY, priority,
      nConts);
  ask.add(nodeLocal);
  ask.add(rackLocal);
  ask.add(any);
  scheduler.allocate(appAttemptId, ask, new ArrayList<ContainerId>(), null, null);

  NodeUpdateSchedulerEvent node0Update = new NodeUpdateSchedulerEvent(node0);

  // Before the node update event, there are 3 local requests outstanding
  Assert.assertEquals(3, nodeLocal.getNumContainers());

  scheduler.handle(node0Update);

  // After the node update event, check that there are no more local requests
  // outstanding
  Assert.assertEquals(0, nodeLocal.getNumContainers());
  //Also check that the containers were scheduled
  SchedulerAppReport info = scheduler.getSchedulerAppInfo(appAttemptId);
  Assert.assertEquals(3, info.getLiveContainers().size());
}
 
开发者ID:ict-carch,项目名称:hadoop-plus,代码行数:60,代码来源:TestFifoScheduler.java


示例18: getContainers

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
@Override
public GetContainersResponse getContainers(GetContainersRequest request)
    throws YarnException, IOException {
  ApplicationAttemptId appAttemptId = request.getApplicationAttemptId();
  ApplicationId appId = appAttemptId.getApplicationId();
  UserGroupInformation callerUGI;
  try {
    callerUGI = UserGroupInformation.getCurrentUser();
  } catch (IOException ie) {
    LOG.info("Error getting UGI ", ie);
    throw RPCUtil.getRemoteException(ie);
  }
  RMApp application = this.rmContext.getRMApps().get(appId);
  if (application == null) {
    // If the RM doesn't have the application, throw
    // ApplicationNotFoundException and let client to handle.
    throw new ApplicationNotFoundException("Application with id '" + appId
        + "' doesn't exist in RM.");
  }
  boolean allowAccess = checkAccess(callerUGI, application.getUser(),
      ApplicationAccessType.VIEW_APP, application);
  GetContainersResponse response = null;
  if (allowAccess) {
    RMAppAttempt appAttempt = application.getAppAttempts().get(appAttemptId);
    if (appAttempt == null) {
      throw new ApplicationAttemptNotFoundException(
          "ApplicationAttempt with id '" + appAttemptId +
          "' doesn't exist in RM.");
    }
    Collection<RMContainer> rmContainers = Collections.emptyList();
    SchedulerAppReport schedulerAppReport =
        this.rmContext.getScheduler().getSchedulerAppInfo(appAttemptId);
    if (schedulerAppReport != null) {
      rmContainers = schedulerAppReport.getLiveContainers();
    }
    List<ContainerReport> listContainers = new ArrayList<ContainerReport>();
    for (RMContainer rmContainer : rmContainers) {
      listContainers.add(rmContainer.createContainerReport());
    }
    response = GetContainersResponse.newInstance(listContainers);
  } else {
    throw new YarnException("User " + callerUGI.getShortUserName()
        + " does not have privilage to see this application " + appId);
  }
  return response;
}
 
开发者ID:hopshadoop,项目名称:hops,代码行数:47,代码来源:ClientRMService.java


示例19: getRMApp

import org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerAppReport; //导入依赖的package包/类
private RMAppImpl getRMApp(RMContext rmContext, YarnScheduler yarnScheduler,
    ApplicationId applicationId3, YarnConfiguration config, String queueName,
    final long memorySeconds, final long vcoreSeconds, final long gpuSeconds,
    String appNodeLabelExpression, String amNodeLabelExpression) throws IOException {
  ApplicationSubmissionContext asContext = mock(ApplicationSubmissionContext.class);
  when(asContext.getMaxAppAttempts()).thenReturn(1);
  when(asContext.getNodeLabelExpression()).thenReturn(appNodeLabelExpression);
  RMAppImpl app =
      spy(new RMAppImpl(applicationId3, rmContext, config, null, null,
          queueName, asContext, yarnScheduler, null,
          System.currentTimeMillis(), "YARN", null,
          BuilderUtils.newResourceRequest(
              RMAppAttemptImpl.AM_CONTAINER_PRIORITY, ResourceRequest.ANY,
              Resource.newInstance(1024, 1, 1), 1), null, null, null, null){
                @Override
                public ApplicationReport createAndGetApplicationReport(
                    String clientUserName, boolean allowAccess) {
                  ApplicationReport report = super.createAndGetApplicationReport(
                      clientUserName, allowAccess);
                  ApplicationResourceUsageReport usageReport = 
                      report.getApplicationResourceUsageReport();
                  usageReport.setMemorySeconds(memorySeconds);
                  usageReport.setVcoreSeconds(vcoreSeconds);
             

鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
上一篇:
Java IdlingRegistry类代码示例发布时间:2022-05-22
下一篇:
Java PreemptionMessageProtoOrBuilder类代码示例发布时间:2022-05-22
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap