• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    迪恩网络公众号

Java ResourceReleaseEvent类代码示例

原作者: [db:作者] 来自: [db:来源] 收藏 邀请

本文整理汇总了Java中org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ResourceReleaseEvent的典型用法代码示例。如果您正苦于以下问题:Java ResourceReleaseEvent类的具体用法?Java ResourceReleaseEvent怎么用?Java ResourceReleaseEvent使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。



ResourceReleaseEvent类属于org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event包,在下文中一共展示了ResourceReleaseEvent类的14个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Java代码示例。

示例1: transition

import org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ResourceReleaseEvent; //导入依赖的package包/类
@Override
public void transition(LocalizedResource rsrc, ResourceEvent event) {
  // Note: assumes that localizing container must succeed or fail
  ResourceReleaseEvent relEvent = (ResourceReleaseEvent) event;
  rsrc.release(relEvent.getContainer());
}
 
开发者ID:naver,项目名称:hadoop,代码行数:7,代码来源:LocalizedResource.java


示例2: handleCleanupContainerResources

import org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ResourceReleaseEvent; //导入依赖的package包/类
@SuppressWarnings("unchecked")
private void handleCleanupContainerResources(
    ContainerLocalizationCleanupEvent rsrcCleanup) {
  Container c = rsrcCleanup.getContainer();
  Map<LocalResourceVisibility, Collection<LocalResourceRequest>> rsrcs =
    rsrcCleanup.getResources();
  for (Map.Entry<LocalResourceVisibility, Collection<LocalResourceRequest>> e :
       rsrcs.entrySet()) {
    LocalResourcesTracker tracker = getLocalResourcesTracker(e.getKey(), c.getUser(), 
        c.getContainerId().getApplicationAttemptId()
        .getApplicationId());
    for (LocalResourceRequest req : e.getValue()) {
      tracker.handle(new ResourceReleaseEvent(req,
          c.getContainerId()));
    }
  }
  String locId = ConverterUtils.toString(c.getContainerId());
  localizerTracker.cleanupPrivLocalizers(locId);
  
  // Delete the container directories
  String userName = c.getUser();
  String containerIDStr = c.toString();
  String appIDStr = ConverterUtils.toString(
      c.getContainerId().getApplicationAttemptId().getApplicationId());
  
  // Try deleting from good local dirs and full local dirs because a dir might
  // have gone bad while the app was running(disk full). In addition
  // a dir might have become good while the app was running.
  // Check if the container dir exists and if it does, try to delete it

  for (String localDir : dirsHandler.getLocalDirsForCleanup()) {
    // Delete the user-owned container-dir
    Path usersdir = new Path(localDir, ContainerLocalizer.USERCACHE);
    Path userdir = new Path(usersdir, userName);
    Path allAppsdir = new Path(userdir, ContainerLocalizer.APPCACHE);
    Path appDir = new Path(allAppsdir, appIDStr);
    Path containerDir = new Path(appDir, containerIDStr);
    submitDirForDeletion(userName, containerDir);

    // Delete the nmPrivate container-dir

    Path sysDir = new Path(localDir, NM_PRIVATE_DIR);
    Path appSysDir = new Path(sysDir, appIDStr);
    Path containerSysDir = new Path(appSysDir, containerIDStr);
    submitDirForDeletion(null, containerSysDir);
  }

  dispatcher.getEventHandler().handle(
      new ContainerEvent(c.getContainerId(),
          ContainerEventType.CONTAINER_RESOURCES_CLEANEDUP));
}
 
开发者ID:naver,项目名称:hadoop,代码行数:52,代码来源:ResourceLocalizationService.java


示例3: handle

import org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ResourceReleaseEvent; //导入依赖的package包/类
@Override
public synchronized void handle(ResourceEvent event) {
  LocalResourceRequest req = event.getLocalResourceRequest();
  LocalizedResource rsrc = localrsrc.get(req);
  switch (event.getType()) {
  case LOCALIZED:
    if (useLocalCacheDirectoryManager) {
      inProgressLocalResourcesMap.remove(req);
    }
    break;
  case REQUEST:
    if (rsrc != null && (!isResourcePresent(rsrc))) {
      LOG.info("Resource " + rsrc.getLocalPath()
          + " is missing, localizing it again");
      removeResource(req);
      rsrc = null;
    }
    if (null == rsrc) {
      rsrc = new LocalizedResource(req, dispatcher);
      localrsrc.put(req, rsrc);
    }
    break;
  case RELEASE:
    if (null == rsrc) {
      // The container sent a release event on a resource which 
      // 1) Failed
      // 2) Removed for some reason (ex. disk is no longer accessible)
      ResourceReleaseEvent relEvent = (ResourceReleaseEvent) event;
      LOG.info("Container " + relEvent.getContainer()
          + " sent RELEASE event on a resource request " + req
          + " not present in cache.");
      return;
    }
    break;
  case LOCALIZATION_FAILED:
    /*
     * If resource localization fails then Localized resource will be
     * removed from local cache.
     */
    removeResource(req);
    break;
  case RECOVERED:
    if (rsrc != null) {
      LOG.warn("Ignoring attempt to recover existing resource " + rsrc);
      return;
    }
    rsrc = recoverResource(req, (ResourceRecoveredEvent) event);
    localrsrc.put(req, rsrc);
    break;
  }

  rsrc.handle(event);

  if (event.getType() == ResourceEventType.LOCALIZED) {
    if (rsrc.getLocalPath() != null) {
      try {
        stateStore.finishResourceLocalization(user, appId,
            buildLocalizedResourceProto(rsrc));
      } catch (IOException ioe) {
        LOG.error("Error storing resource state for " + rsrc, ioe);
      }
    } else {
      LOG.warn("Resource " + rsrc + " localized without a location");
    }
  }
}
 
开发者ID:naver,项目名称:hadoop,代码行数:67,代码来源:LocalResourcesTrackerImpl.java


示例4: testConsistency

import org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ResourceReleaseEvent; //导入依赖的package包/类
@Test(timeout=10000)
@SuppressWarnings("unchecked")
public void testConsistency() {
  String user = "testuser";
  DrainDispatcher dispatcher = null;
  try {
    Configuration conf = new Configuration();
    dispatcher = createDispatcher(conf);
    EventHandler<LocalizerEvent> localizerEventHandler = mock(EventHandler.class);
    EventHandler<LocalizerEvent> containerEventHandler = mock(EventHandler.class);
    dispatcher.register(LocalizerEventType.class, localizerEventHandler);
    dispatcher.register(ContainerEventType.class, containerEventHandler);

    ContainerId cId1 = BuilderUtils.newContainerId(1, 1, 1, 1);
    LocalizerContext lc1 = new LocalizerContext(user, cId1, null);
    LocalResourceRequest req1 = createLocalResourceRequest(user, 1, 1,
        LocalResourceVisibility.PUBLIC);
    LocalizedResource lr1 = createLocalizedResource(req1, dispatcher);
    ConcurrentMap<LocalResourceRequest, LocalizedResource> localrsrc = new ConcurrentHashMap<LocalResourceRequest, LocalizedResource>();
    localrsrc.put(req1, lr1);
    LocalResourcesTracker tracker = new LocalResourcesTrackerImpl(user,
        null, dispatcher, localrsrc, false, conf,
        new NMNullStateStoreService(), null);

    ResourceEvent req11Event = new ResourceRequestEvent(req1,
        LocalResourceVisibility.PUBLIC, lc1);

    ResourceEvent rel11Event = new ResourceReleaseEvent(req1, cId1);

    // Localize R1 for C1
    tracker.handle(req11Event);

    dispatcher.await();

    // Verify refCount for R1 is 1
    Assert.assertEquals(1, lr1.getRefCount());

    dispatcher.await();
    verifyTrackedResourceCount(tracker, 1);

    // Localize resource1
    ResourceLocalizedEvent rle = new ResourceLocalizedEvent(req1, new Path(
        "file:///tmp/r1"), 1);
    lr1.handle(rle);
    Assert.assertTrue(lr1.getState().equals(ResourceState.LOCALIZED));
    Assert.assertTrue(createdummylocalizefile(new Path("file:///tmp/r1")));
    LocalizedResource rsrcbefore = tracker.iterator().next();
    File resFile = new File(lr1.getLocalPath().toUri().getRawPath()
        .toString());
    Assert.assertTrue(resFile.exists());
    Assert.assertTrue(resFile.delete());

    // Localize R1 for C1
    tracker.handle(req11Event);

    dispatcher.await();
    lr1.handle(rle);
    Assert.assertTrue(lr1.getState().equals(ResourceState.LOCALIZED));
    LocalizedResource rsrcafter = tracker.iterator().next();
    if (rsrcbefore == rsrcafter) {
      Assert.fail("Localized resource should not be equal");
    }
    // Release resource1
    tracker.handle(rel11Event);
  } finally {
    if (dispatcher != null) {
      dispatcher.stop();
    }
  }
}
 
开发者ID:naver,项目名称:hadoop,代码行数:71,代码来源:TestLocalResourcesTrackerImpl.java


示例5: testStateStoreSuccessfulLocalization

import org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ResourceReleaseEvent; //导入依赖的package包/类
@Test
@SuppressWarnings("unchecked")
public void testStateStoreSuccessfulLocalization() throws Exception {
  final String user = "someuser";
  final ApplicationId appId = ApplicationId.newInstance(1, 1);
  // This is a random path. NO File creation will take place at this place.
  final Path localDir = new Path("/tmp");
  Configuration conf = new YarnConfiguration();
  DrainDispatcher dispatcher = null;
  dispatcher = createDispatcher(conf);
  EventHandler<LocalizerEvent> localizerEventHandler =
      mock(EventHandler.class);
  EventHandler<LocalizerEvent> containerEventHandler =
      mock(EventHandler.class);
  dispatcher.register(LocalizerEventType.class, localizerEventHandler);
  dispatcher.register(ContainerEventType.class, containerEventHandler);
  DeletionService mockDelService = mock(DeletionService.class);
  NMStateStoreService stateStore = mock(NMStateStoreService.class);

  try {
    LocalResourcesTracker tracker = new LocalResourcesTrackerImpl(user,
        appId, dispatcher, false, conf, stateStore);
    // Container 1 needs lr1 resource
    ContainerId cId1 = BuilderUtils.newContainerId(1, 1, 1, 1);
    LocalResourceRequest lr1 = createLocalResourceRequest(user, 1, 1,
        LocalResourceVisibility.APPLICATION);
    LocalizerContext lc1 = new LocalizerContext(user, cId1, null);

    // Container 1 requests lr1 to be localized
    ResourceEvent reqEvent1 = new ResourceRequestEvent(lr1,
        LocalResourceVisibility.APPLICATION, lc1);
    tracker.handle(reqEvent1);
    dispatcher.await();

    // Simulate the process of localization of lr1
    Path hierarchicalPath1 = tracker.getPathForLocalization(lr1, localDir,
        null);

    ArgumentCaptor<LocalResourceProto> localResourceCaptor =
        ArgumentCaptor.forClass(LocalResourceProto.class);
    ArgumentCaptor<Path> pathCaptor = ArgumentCaptor.forClass(Path.class);
    verify(stateStore).startResourceLocalization(eq(user), eq(appId),
        localResourceCaptor.capture(), pathCaptor.capture());
    LocalResourceProto lrProto = localResourceCaptor.getValue();
    Path localizedPath1 = pathCaptor.getValue();
    Assert.assertEquals(lr1,
        new LocalResourceRequest(new LocalResourcePBImpl(lrProto)));
    Assert.assertEquals(hierarchicalPath1, localizedPath1.getParent());

    // Simulate lr1 getting localized
    ResourceLocalizedEvent rle1 =
        new ResourceLocalizedEvent(lr1, pathCaptor.getValue(), 120);
    tracker.handle(rle1);
    dispatcher.await();

    ArgumentCaptor<LocalizedResourceProto> localizedProtoCaptor =
        ArgumentCaptor.forClass(LocalizedResourceProto.class);
    verify(stateStore).finishResourceLocalization(eq(user), eq(appId),
        localizedProtoCaptor.capture());
    LocalizedResourceProto localizedProto = localizedProtoCaptor.getValue();
    Assert.assertEquals(lr1, new LocalResourceRequest(
        new LocalResourcePBImpl(localizedProto.getResource())));
    Assert.assertEquals(localizedPath1.toString(),
        localizedProto.getLocalPath());
    LocalizedResource localizedRsrc1 = tracker.getLocalizedResource(lr1);
    Assert.assertNotNull(localizedRsrc1);

    // simulate release and retention processing
    tracker.handle(new ResourceReleaseEvent(lr1, cId1));
    dispatcher.await();
    boolean removeResult = tracker.remove(localizedRsrc1, mockDelService);

    Assert.assertTrue(removeResult);
    verify(stateStore).removeLocalizedResource(eq(user), eq(appId),
        eq(localizedPath1));
  } finally {
    if (dispatcher != null) {
      dispatcher.stop();
    }
  }
}
 
开发者ID:naver,项目名称:hadoop,代码行数:82,代码来源:TestLocalResourcesTrackerImpl.java


示例6: testReleaseWhileDownloading

import org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ResourceReleaseEvent; //导入依赖的package包/类
@Test
@SuppressWarnings("unchecked")
public void testReleaseWhileDownloading() throws Exception {
  String user = "testuser";
  DrainDispatcher dispatcher = null;
  try {
    Configuration conf = new Configuration();
    dispatcher = createDispatcher(conf);
    EventHandler<LocalizerEvent> localizerEventHandler =
        mock(EventHandler.class);
    EventHandler<LocalizerEvent> containerEventHandler =
        mock(EventHandler.class);
    dispatcher.register(LocalizerEventType.class, localizerEventHandler);
    dispatcher.register(ContainerEventType.class, containerEventHandler);

    ContainerId cId = BuilderUtils.newContainerId(1, 1, 1, 1);
    LocalizerContext lc = new LocalizerContext(user, cId, null);

    LocalResourceRequest req =
        createLocalResourceRequest(user, 1, 1, LocalResourceVisibility.PUBLIC);
    LocalizedResource lr = createLocalizedResource(req, dispatcher);
    ConcurrentMap<LocalResourceRequest, LocalizedResource> localrsrc =
        new ConcurrentHashMap<LocalResourceRequest, LocalizedResource>();
    localrsrc.put(req, lr);
    LocalResourcesTracker tracker =
        new LocalResourcesTrackerImpl(user, null, dispatcher, localrsrc,
            false, conf, new NMNullStateStoreService(), null);

    // request the resource
    ResourceEvent reqEvent =
        new ResourceRequestEvent(req, LocalResourceVisibility.PUBLIC, lc);
    tracker.handle(reqEvent);

    // release the resource
    ResourceEvent relEvent = new ResourceReleaseEvent(req, cId);
    tracker.handle(relEvent);

    // download completing after release
    ResourceLocalizedEvent rle =
        new ResourceLocalizedEvent(req, new Path("file:///tmp/r1"), 1);
    tracker.handle(rle);

    dispatcher.await();
  } finally {
    if (dispatcher != null) {
      dispatcher.stop();
    }
  }
}
 
开发者ID:aliyun-beta,项目名称:aliyun-oss-hadoop-fs,代码行数:50,代码来源:TestLocalResourcesTrackerImpl.java


示例7: testConsistency

import org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ResourceReleaseEvent; //导入依赖的package包/类
@Test(timeout=10000)
@SuppressWarnings("unchecked")
public void testConsistency() {
  String user = "testuser";
  DrainDispatcher dispatcher = null;
  try {
    Configuration conf = new Configuration();
    dispatcher = createDispatcher(conf);
    EventHandler<LocalizerEvent> localizerEventHandler = mock(EventHandler.class);
    EventHandler<LocalizerEvent> containerEventHandler = mock(EventHandler.class);
    dispatcher.register(LocalizerEventType.class, localizerEventHandler);
    dispatcher.register(ContainerEventType.class, containerEventHandler);

    ContainerId cId1 = BuilderUtils.newContainerId(1, 1, 1, 1);
    LocalizerContext lc1 = new LocalizerContext(user, cId1, null);
    LocalResourceRequest req1 = createLocalResourceRequest(user, 1, 1,
        LocalResourceVisibility.PUBLIC);
    LocalizedResource lr1 = createLocalizedResource(req1, dispatcher);
    ConcurrentMap<LocalResourceRequest, LocalizedResource> localrsrc = new ConcurrentHashMap<LocalResourceRequest, LocalizedResource>();
    localrsrc.put(req1, lr1);
    LocalResourcesTracker tracker = new LocalResourcesTrackerImpl(user,
        null, dispatcher, localrsrc, false, conf,
        new NMNullStateStoreService());

    ResourceEvent req11Event = new ResourceRequestEvent(req1,
        LocalResourceVisibility.PUBLIC, lc1);

    ResourceEvent rel11Event = new ResourceReleaseEvent(req1, cId1);

    // Localize R1 for C1
    tracker.handle(req11Event);

    dispatcher.await();

    // Verify refCount for R1 is 1
    Assert.assertEquals(1, lr1.getRefCount());

    dispatcher.await();
    verifyTrackedResourceCount(tracker, 1);

    // Localize resource1
    ResourceLocalizedEvent rle = new ResourceLocalizedEvent(req1, new Path(
        "file:///tmp/r1"), 1);
    lr1.handle(rle);
    Assert.assertTrue(lr1.getState().equals(ResourceState.LOCALIZED));
    Assert.assertTrue(createdummylocalizefile(new Path("file:///tmp/r1")));
    LocalizedResource rsrcbefore = tracker.iterator().next();
    File resFile = new File(lr1.getLocalPath().toUri().getRawPath()
        .toString());
    Assert.assertTrue(resFile.exists());
    Assert.assertTrue(resFile.delete());

    // Localize R1 for C1
    tracker.handle(req11Event);

    dispatcher.await();
    lr1.handle(rle);
    Assert.assertTrue(lr1.getState().equals(ResourceState.LOCALIZED));
    LocalizedResource rsrcafter = tracker.iterator().next();
    if (rsrcbefore == rsrcafter) {
      Assert.fail("Localized resource should not be equal");
    }
    // Release resource1
    tracker.handle(rel11Event);
  } finally {
    if (dispatcher != null) {
      dispatcher.stop();
    }
  }
}
 
开发者ID:yncxcw,项目名称:big-c,代码行数:71,代码来源:TestLocalResourcesTrackerImpl.java


示例8: testStateStoreSuccessfulLocalization

import org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ResourceReleaseEvent; //导入依赖的package包/类
@Test
@SuppressWarnings("unchecked")
public void testStateStoreSuccessfulLocalization() throws Exception {
  final String user = "someuser";
  final ApplicationId appId = ApplicationId.newInstance(1, 1);
  // This is a random path. NO File creation will take place at this place.
  final Path localDir = new Path("/tmp");
  Configuration conf = new YarnConfiguration();
  DrainDispatcher dispatcher = null;
  dispatcher = createDispatcher(conf);
  EventHandler<LocalizerEvent> localizerEventHandler =
      mock(EventHandler.class);
  EventHandler<LocalizerEvent> containerEventHandler =
      mock(EventHandler.class);
  dispatcher.register(LocalizerEventType.class, localizerEventHandler);
  dispatcher.register(ContainerEventType.class, containerEventHandler);
  DeletionService mockDelService = mock(DeletionService.class);
  NMStateStoreService stateStore = mock(NMStateStoreService.class);

  try {
    LocalResourcesTracker tracker = new LocalResourcesTrackerImpl(user,
        appId, dispatcher, false, conf, stateStore);
    // Container 1 needs lr1 resource
    ContainerId cId1 = BuilderUtils.newContainerId(1, 1, 1, 1);
    LocalResourceRequest lr1 = createLocalResourceRequest(user, 1, 1,
        LocalResourceVisibility.APPLICATION);
    LocalizerContext lc1 = new LocalizerContext(user, cId1, null);

    // Container 1 requests lr1 to be localized
    ResourceEvent reqEvent1 = new ResourceRequestEvent(lr1,
        LocalResourceVisibility.APPLICATION, lc1);
    tracker.handle(reqEvent1);
    dispatcher.await();

    // Simulate the process of localization of lr1
    Path hierarchicalPath1 = tracker.getPathForLocalization(lr1, localDir);

    ArgumentCaptor<LocalResourceProto> localResourceCaptor =
        ArgumentCaptor.forClass(LocalResourceProto.class);
    ArgumentCaptor<Path> pathCaptor = ArgumentCaptor.forClass(Path.class);
    verify(stateStore).startResourceLocalization(eq(user), eq(appId),
        localResourceCaptor.capture(), pathCaptor.capture());
    LocalResourceProto lrProto = localResourceCaptor.getValue();
    Path localizedPath1 = pathCaptor.getValue();
    Assert.assertEquals(lr1,
        new LocalResourceRequest(new LocalResourcePBImpl(lrProto)));
    Assert.assertEquals(hierarchicalPath1, localizedPath1.getParent());

    // Simulate lr1 getting localized
    ResourceLocalizedEvent rle1 =
        new ResourceLocalizedEvent(lr1, pathCaptor.getValue(), 120);
    tracker.handle(rle1);
    dispatcher.await();

    ArgumentCaptor<LocalizedResourceProto> localizedProtoCaptor =
        ArgumentCaptor.forClass(LocalizedResourceProto.class);
    verify(stateStore).finishResourceLocalization(eq(user), eq(appId),
        localizedProtoCaptor.capture());
    LocalizedResourceProto localizedProto = localizedProtoCaptor.getValue();
    Assert.assertEquals(lr1, new LocalResourceRequest(
        new LocalResourcePBImpl(localizedProto.getResource())));
    Assert.assertEquals(localizedPath1.toString(),
        localizedProto.getLocalPath());
    LocalizedResource localizedRsrc1 = tracker.getLocalizedResource(lr1);
    Assert.assertNotNull(localizedRsrc1);

    // simulate release and retention processing
    tracker.handle(new ResourceReleaseEvent(lr1, cId1));
    dispatcher.await();
    boolean removeResult = tracker.remove(localizedRsrc1, mockDelService);

    Assert.assertTrue(removeResult);
    verify(stateStore).removeLocalizedResource(eq(user), eq(appId),
        eq(localizedPath1));
  } finally {
    if (dispatcher != null) {
      dispatcher.stop();
    }
  }
}
 
开发者ID:yncxcw,项目名称:big-c,代码行数:81,代码来源:TestLocalResourcesTrackerImpl.java


示例9: handleCleanupContainerResources

import org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ResourceReleaseEvent; //导入依赖的package包/类
@SuppressWarnings("unchecked")
private void handleCleanupContainerResources(
    ContainerLocalizationCleanupEvent rsrcCleanup) {
  Container c = rsrcCleanup.getContainer();
  Map<LocalResourceVisibility, Collection<LocalResourceRequest>> rsrcs =
    rsrcCleanup.getResources();
  for (Map.Entry<LocalResourceVisibility, Collection<LocalResourceRequest>> e :
       rsrcs.entrySet()) {
    LocalResourcesTracker tracker = getLocalResourcesTracker(e.getKey(), c.getUser(), 
        c.getContainerId().getApplicationAttemptId()
        .getApplicationId());
    for (LocalResourceRequest req : e.getValue()) {
      tracker.handle(new ResourceReleaseEvent(req,
          c.getContainerId()));
    }
  }
  String locId = ConverterUtils.toString(c.getContainerId());
  localizerTracker.cleanupPrivLocalizers(locId);
  
  // Delete the container directories
  String userName = c.getUser();
  String containerIDStr = c.toString();
  String appIDStr = ConverterUtils.toString(
      c.getContainerId().getApplicationAttemptId().getApplicationId());
  for (String localDir : dirsHandler.getLocalDirs()) {

    // Delete the user-owned container-dir
    Path usersdir = new Path(localDir, ContainerLocalizer.USERCACHE);
    Path userdir = new Path(usersdir, userName);
    Path allAppsdir = new Path(userdir, ContainerLocalizer.APPCACHE);
    Path appDir = new Path(allAppsdir, appIDStr);
    Path containerDir = new Path(appDir, containerIDStr);
    delService.delete(userName, containerDir, new Path[] {});

    // Delete the nmPrivate container-dir
    
    Path sysDir = new Path(localDir, NM_PRIVATE_DIR);
    Path appSysDir = new Path(sysDir, appIDStr);
    Path containerSysDir = new Path(appSysDir, containerIDStr);
    delService.delete(null, containerSysDir,  new Path[] {});
  }

  dispatcher.getEventHandler().handle(
      new ContainerEvent(c.getContainerId(),
          ContainerEventType.CONTAINER_RESOURCES_CLEANEDUP));
}
 
开发者ID:ict-carch,项目名称:hadoop-plus,代码行数:47,代码来源:ResourceLocalizationService.java


示例10: handle

import org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ResourceReleaseEvent; //导入依赖的package包/类
@Override
public synchronized void handle(ResourceEvent event) {
  LocalResourceRequest req = event.getLocalResourceRequest();
  LocalizedResource rsrc = localrsrc.get(req);
  switch (event.getType()) {
  case LOCALIZED:
    if (useLocalCacheDirectoryManager) {
      inProgressLocalResourcesMap.remove(req);
    }
    break;
  case REQUEST:
    if (rsrc != null && (!isResourcePresent(rsrc))) {
      LOG.info("Resource " + rsrc.getLocalPath()
          + " is missing, localizing it again");
      localrsrc.remove(req);
      decrementFileCountForLocalCacheDirectory(req, rsrc);
      rsrc = null;
    }
    if (null == rsrc) {
      rsrc = new LocalizedResource(req, dispatcher);
      localrsrc.put(req, rsrc);
    }
    break;
  case RELEASE:
    if (null == rsrc) {
      // The container sent a release event on a resource which 
      // 1) Failed
      // 2) Removed for some reason (ex. disk is no longer accessible)
      ResourceReleaseEvent relEvent = (ResourceReleaseEvent) event;
      LOG.info("Container " + relEvent.getContainer()
          + " sent RELEASE event on a resource request " + req
          + " not present in cache.");
      return;
    }
    break;
  case LOCALIZATION_FAILED:
    decrementFileCountForLocalCacheDirectory(req, null);
    /*
     * If resource localization fails then Localized resource will be
     * removed from local cache.
     */
    localrsrc.remove(req);
    break;
  }
  rsrc.handle(event);
}
 
开发者ID:ict-carch,项目名称:hadoop-plus,代码行数:47,代码来源:LocalResourcesTrackerImpl.java


示例11: testConsistency

import org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ResourceReleaseEvent; //导入依赖的package包/类
@Test(timeout=10000)
@SuppressWarnings("unchecked")
public void testConsistency() {
  String user = "testuser";
  DrainDispatcher dispatcher = null;
  try {
    Configuration conf = new Configuration();
    dispatcher = createDispatcher(conf);
    EventHandler<LocalizerEvent> localizerEventHandler = mock(EventHandler.class);
    EventHandler<LocalizerEvent> containerEventHandler = mock(EventHandler.class);
    dispatcher.register(LocalizerEventType.class, localizerEventHandler);
    dispatcher.register(ContainerEventType.class, containerEventHandler);

    ContainerId cId1 = BuilderUtils.newContainerId(1, 1, 1, 1);
    LocalizerContext lc1 = new LocalizerContext(user, cId1, null);
    LocalResourceRequest req1 = createLocalResourceRequest(user, 1, 1,
        LocalResourceVisibility.PUBLIC);
    LocalizedResource lr1 = createLocalizedResource(req1, dispatcher);
    ConcurrentMap<LocalResourceRequest, LocalizedResource> localrsrc = new ConcurrentHashMap<LocalResourceRequest, LocalizedResource>();
    localrsrc.put(req1, lr1);
    LocalResourcesTracker tracker = new LocalResourcesTrackerImpl(user,
        dispatcher, localrsrc, false, conf);

    ResourceEvent req11Event = new ResourceRequestEvent(req1,
        LocalResourceVisibility.PUBLIC, lc1);

    ResourceEvent rel11Event = new ResourceReleaseEvent(req1, cId1);

    // Localize R1 for C1
    tracker.handle(req11Event);

    dispatcher.await();

    // Verify refCount for R1 is 1
    Assert.assertEquals(1, lr1.getRefCount());

    dispatcher.await();
    verifyTrackedResourceCount(tracker, 1);

    // Localize resource1
    ResourceLocalizedEvent rle = new ResourceLocalizedEvent(req1, new Path(
        "file:///tmp/r1"), 1);
    lr1.handle(rle);
    Assert.assertTrue(lr1.getState().equals(ResourceState.LOCALIZED));
    Assert.assertTrue(createdummylocalizefile(new Path("file:///tmp/r1")));
    LocalizedResource rsrcbefore = tracker.iterator().next();
    File resFile = new File(lr1.getLocalPath().toUri().getRawPath()
        .toString());
    Assert.assertTrue(resFile.exists());
    Assert.assertTrue(resFile.delete());

    // Localize R1 for C1
    tracker.handle(req11Event);

    dispatcher.await();
    lr1.handle(rle);
    Assert.assertTrue(lr1.getState().equals(ResourceState.LOCALIZED));
    LocalizedResource rsrcafter = tracker.iterator().next();
    if (rsrcbefore == rsrcafter) {
      Assert.fail("Localized resource should not be equal");
    }
    // Release resource1
    tracker.handle(rel11Event);
  } finally {
    if (dispatcher != null) {
      dispatcher.stop();
    }
  }
}
 
开发者ID:ict-carch,项目名称:hadoop-plus,代码行数:70,代码来源:TestLocalResourcesTrackerImpl.java


示例12: handleCleanupContainerResources

import org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ResourceReleaseEvent; //导入依赖的package包/类
@SuppressWarnings("unchecked")
private void handleCleanupContainerResources(
    ContainerLocalizationCleanupEvent rsrcCleanup) {
  Container c = rsrcCleanup.getContainer();
  Map<LocalResourceVisibility, Collection<LocalResourceRequest>> rsrcs =
    rsrcCleanup.getResources();
  for (Map.Entry<LocalResourceVisibility, Collection<LocalResourceRequest>> e :
       rsrcs.entrySet()) {
    LocalResourcesTracker tracker = getLocalResourcesTracker(e.getKey(), c.getUser(), 
        c.getContainerId().getApplicationAttemptId()
        .getApplicationId());
    for (LocalResourceRequest req : e.getValue()) {
      tracker.handle(new ResourceReleaseEvent(req,
          c.getContainerId()));
    }
  }
  String locId = c.getContainerId().toString();
  localizerTracker.cleanupPrivLocalizers(locId);
  
  // Delete the container directories
  String userName = c.getUser();
  String userFolder = c.getUserFolder();
  String containerIDStr = c.toString();
  String appIDStr =
      c.getContainerId().getApplicationAttemptId().getApplicationId()
          .toString();
  
  // Try deleting from good local dirs and full local dirs because a dir might
  // have gone bad while the app was running(disk full). In addition
  // a dir might have become good while the app was running.
  // Check if the container dir exists and if it does, try to delete it

  for (String localDir : dirsHandler.getLocalDirsForCleanup()) {
    // Delete the user-owned container-dir
    Path usersdir = new Path(localDir, ContainerLocalizer.USERCACHE);
    Path userdir = new Path(usersdir, userFolder);
    Path allAppsdir = new Path(userdir, ContainerLocalizer.APPCACHE);
    Path appDir = new Path(allAppsdir, appIDStr);
    Path containerDir = new Path(appDir, containerIDStr);
    submitDirForDeletion(userName, containerDir);

    // Delete the nmPrivate container-dir

    Path sysDir = new Path(localDir, NM_PRIVATE_DIR);
    Path appSysDir = new Path(sysDir, appIDStr);
    Path containerSysDir = new Path(appSysDir, containerIDStr);
    submitDirForDeletion(null, containerSysDir);
  }

  dispatcher.getEventHandler().handle(
      new ContainerEvent(c.getContainerId(),
          ContainerEventType.CONTAINER_RESOURCES_CLEANEDUP));
}
 
开发者ID:hopshadoop,项目名称:hops,代码行数:54,代码来源:ResourceLocalizationService.java


示例13: testConsistency

import org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ResourceReleaseEvent; //导入依赖的package包/类
@Test(timeout=10000)
@SuppressWarnings("unchecked")
public void testConsistency() {
  String user = "testuser";
  String userFolder = "testuserFolder";
  DrainDispatcher dispatcher = null;
  try {
    Configuration conf = new Configuration();
    dispatcher = createDispatcher(conf);
    EventHandler<LocalizerEvent> localizerEventHandler = mock(EventHandler.class);
    EventHandler<LocalizerEvent> containerEventHandler = mock(EventHandler.class);
    dispatcher.register(LocalizerEventType.class, localizerEventHandler);
    dispatcher.register(ContainerEventType.class, containerEventHandler);

    ContainerId cId1 = BuilderUtils.newContainerId(1, 1, 1, 1);
    LocalizerContext lc1 = new LocalizerContext(user, cId1, null, userFolder);
    LocalResourceRequest req1 = createLocalResourceRequest(user, 1, 1,
        LocalResourceVisibility.PUBLIC);
    LocalizedResource lr1 = createLocalizedResource(req1, dispatcher);
    ConcurrentMap<LocalResourceRequest, LocalizedResource> localrsrc = new ConcurrentHashMap<LocalResourceRequest, LocalizedResource>();
    localrsrc.put(req1, lr1);
    LocalResourcesTracker tracker = new LocalResourcesTrackerImpl(user,
        null, dispatcher, localrsrc, false, conf,
        new NMNullStateStoreService(), null);

    ResourceEvent req11Event = new ResourceRequestEvent(req1,
        LocalResourceVisibility.PUBLIC, lc1);

    ResourceEvent rel11Event = new ResourceReleaseEvent(req1, cId1);

    // Localize R1 for C1
    tracker.handle(req11Event);

    dispatcher.await();

    // Verify refCount for R1 is 1
    Assert.assertEquals(1, lr1.getRefCount());

    dispatcher.await();
    verifyTrackedResourceCount(tracker, 1);

    // Localize resource1
    ResourceLocalizedEvent rle = new ResourceLocalizedEvent(req1, new Path(
        "file:///tmp/r1"), 1);
    lr1.handle(rle);
    Assert.assertTrue(lr1.getState().equals(ResourceState.LOCALIZED));
    Assert.assertTrue(createdummylocalizefile(new Path("file:///tmp/r1")));
    LocalizedResource rsrcbefore = tracker.iterator().next();
    File resFile = new File(lr1.getLocalPath().toUri().getRawPath()
        .toString());
    Assert.assertTrue(resFile.exists());
    Assert.assertTrue(resFile.delete());

    // Localize R1 for C1
    tracker.handle(req11Event);

    dispatcher.await();
    lr1.handle(rle);
    Assert.assertTrue(lr1.getState().equals(ResourceState.LOCALIZED));
    LocalizedResource rsrcafter = tracker.iterator().next();
    if (rsrcbefore == rsrcafter) {
      Assert.fail("Localized resource should not be equal");
    }
    // Release resource1
    tracker.handle(rel11Event);
  } finally {
    if (dispatcher != null) {
      dispatcher.stop();
    }
  }
}
 
开发者ID:hopshadoop,项目名称:hops,代码行数:72,代码来源:TestLocalResourcesTrackerImpl.java


示例14: testReleaseWhileDownloading

import org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ResourceReleaseEvent; //导入依赖的package包/类
@Test
@SuppressWarnings("unchecked")
public void testReleaseWhileDownloading() throws Exception {
  String user = "testuser";
  String userFolder = "testuserFolder";
  DrainDispatcher dispatcher = null;
  try {
    Configuration conf = new Configuration();
    dispatcher = createDispatcher(conf);
    EventHandler<LocalizerEvent> localizerEventHandler =
        mock(EventHandler.class);
    EventHandler<LocalizerEvent> containerEventHandler =
        mock(EventHandler.class);
    dispatcher.register(LocalizerEventType.class, localizerEventHandler);
    dispatcher.register(ContainerEventType.class, containerEventHandler);

    ContainerId cId = BuilderUtils.newContainerId(1, 1, 1, 1);
    LocalizerContext lc = new LocalizerContext(user, cId, null, userFolder);

    LocalResourceRequest req =
        createLocalResourceRequest(user, 1, 1, LocalResourceVisibility.PUBLIC);
    LocalizedResource lr = createLocalizedResource(req, dispatcher);
    ConcurrentMap<LocalResourceRequest, LocalizedResource> localrsrc =
        new ConcurrentHashMap<LocalResourceRequest, LocalizedResource>();
    localrsrc.put(req, lr);
    LocalResourcesTracker tracker =
        new LocalResourcesTrackerImpl(user, null, dispatcher, localrsrc,
            false, conf, new NMNullStateStoreService(), null);

    // request the resource
    ResourceEvent reqEvent =
        new ResourceRequestEvent(req, LocalResourceVisibility.PUBLIC, lc);
    tracker.handle(reqEvent);

    // release the resource
    ResourceEvent relEvent = new ResourceReleaseEvent(req, cId);
    tracker.handle(relEvent);

    // download completing after release
    ResourceLocalizedEvent rle =
        new ResourceLocalizedEvent(req, new Path("file:///tmp/r1"), 1);
    tracker.handle(rle);

    dispatcher.await();
  } finally {
    if (dispatcher != null) {
      dispatcher.stop();
    }
  }
}
 
开发者ID:hopshadoop,项目名称:hops,代码行数:51,代码来源:TestLocalResourcesTrackerImpl.java



注:本文中的org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ResourceReleaseEvent类示例整理自Github/MSDocs等源码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。


鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
上一篇:
Java DiffReportEntry类代码示例发布时间:2022-05-22
下一篇:
Java Truffle类代码示例发布时间:2022-05-22
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap