本文整理汇总了Java中org.apache.helix.model.builder.CustomModeISBuilder类的典型用法代码示例。如果您正苦于以下问题:Java CustomModeISBuilder类的具体用法?Java CustomModeISBuilder怎么用?Java CustomModeISBuilder使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。
CustomModeISBuilder类属于org.apache.helix.model.builder包,在下文中一共展示了CustomModeISBuilder类的14个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Java代码示例。
示例1: buildCustomIdealStateFor
import org.apache.helix.model.builder.CustomModeISBuilder; //导入依赖的package包/类
public static IdealState buildCustomIdealStateFor(String topicName,
int numTopicPartitions,
PriorityQueue<InstanceTopicPartitionHolder> instanceToNumServingTopicPartitionMap) {
final CustomModeISBuilder customModeIdealStateBuilder = new CustomModeISBuilder(topicName);
customModeIdealStateBuilder
.setStateModel(OnlineOfflineStateModel.name)
.setNumPartitions(numTopicPartitions).setNumReplica(1)
.setMaxPartitionsPerNode(numTopicPartitions);
for (int i = 0; i < numTopicPartitions; ++i) {
synchronized (instanceToNumServingTopicPartitionMap) {
InstanceTopicPartitionHolder liveInstance = instanceToNumServingTopicPartitionMap.poll();
customModeIdealStateBuilder.assignInstanceAndState(Integer.toString(i),
liveInstance.getInstanceName(), "ONLINE");
liveInstance.addTopicPartition(new TopicPartition(topicName, i));
instanceToNumServingTopicPartitionMap.add(liveInstance);
}
}
return customModeIdealStateBuilder.build();
}
开发者ID:uber,项目名称:uReplicator,代码行数:23,代码来源:HelixUtils.java
示例2: buildCustomIdealStateFor
import org.apache.helix.model.builder.CustomModeISBuilder; //导入依赖的package包/类
public static IdealState buildCustomIdealStateFor(String topicName,
int numTopicPartitions,
PriorityQueue<InstanceTopicPartitionHolder> instanceToNumServingTopicPartitionMap) {
final CustomModeISBuilder customModeIdealStateBuilder = new CustomModeISBuilder(topicName);
customModeIdealStateBuilder
.setStateModel(OnlineOfflineStateModel.name)
.setNumPartitions(numTopicPartitions).setNumReplica(1)
.setMaxPartitionsPerNode(numTopicPartitions);
for (int i = 0; i < numTopicPartitions; ++i) {
InstanceTopicPartitionHolder liveInstance = instanceToNumServingTopicPartitionMap.poll();
if (liveInstance != null) {
customModeIdealStateBuilder.assignInstanceAndState(Integer.toString(i),
liveInstance.getInstanceName(), "ONLINE");
liveInstance.addTopicPartition(new TopicPartition(topicName, i));
instanceToNumServingTopicPartitionMap.add(liveInstance);
}
}
return customModeIdealStateBuilder.build();
}
开发者ID:uber,项目名称:uReplicator,代码行数:23,代码来源:IdealStateBuilder.java
示例3: getIdealStatesFromAssignment
import org.apache.helix.model.builder.CustomModeISBuilder; //导入依赖的package包/类
public static Map<String, IdealState> getIdealStatesFromAssignment(
Set<InstanceTopicPartitionHolder> newAssignment) {
Map<String, CustomModeISBuilder> idealStatesBuilderMap =
new HashMap<String, CustomModeISBuilder>();
for (InstanceTopicPartitionHolder instance : newAssignment) {
for (TopicPartition tpi : instance.getServingTopicPartitionSet()) {
String topicName = tpi.getTopic();
String partition = Integer.toString(tpi.getPartition());
if (!idealStatesBuilderMap.containsKey(topicName)) {
final CustomModeISBuilder customModeIdealStateBuilder =
new CustomModeISBuilder(topicName);
customModeIdealStateBuilder
.setStateModel(OnlineOfflineStateModel.name)
.setNumReplica(1);
idealStatesBuilderMap.put(topicName, customModeIdealStateBuilder);
}
idealStatesBuilderMap.get(topicName).assignInstanceAndState(partition,
instance.getInstanceName(),
"ONLINE");
}
}
Map<String, IdealState> idealStatesMap = new HashMap<String, IdealState>();
for (String topic : idealStatesBuilderMap.keySet()) {
IdealState idealState = idealStatesBuilderMap.get(topic).build();
idealState.setMaxPartitionsPerInstance(idealState.getPartitionSet().size());
idealState.setNumPartitions(idealState.getPartitionSet().size());
idealStatesMap.put(topic, idealState);
}
return idealStatesMap;
}
开发者ID:uber,项目名称:uReplicator,代码行数:32,代码来源:HelixUtils.java
示例4: expandCustomRebalanceModeIdealStateFor
import org.apache.helix.model.builder.CustomModeISBuilder; //导入依赖的package包/类
public static IdealState expandCustomRebalanceModeIdealStateFor(IdealState oldIdealState,
String topicName, int newNumTopicPartitions,
PriorityQueue<InstanceTopicPartitionHolder> instanceToNumServingTopicPartitionMap) {
final CustomModeISBuilder customModeIdealStateBuilder = new CustomModeISBuilder(topicName);
customModeIdealStateBuilder
.setStateModel(OnlineOfflineStateModel.name)
.setNumPartitions(newNumTopicPartitions).setNumReplica(1)
.setMaxPartitionsPerNode(newNumTopicPartitions);
int numOldPartitions = oldIdealState.getNumPartitions();
for (int i = 0; i < numOldPartitions; ++i) {
String partitionName = Integer.toString(i);
try {
String instanceName =
oldIdealState.getInstanceStateMap(partitionName).keySet().iterator().next();
customModeIdealStateBuilder.assignInstanceAndState(partitionName, instanceName, "ONLINE");
} catch (Exception e) {
// No worker added into the cluster.
}
}
for (int i = numOldPartitions; i < newNumTopicPartitions; ++i) {
InstanceTopicPartitionHolder liveInstance = instanceToNumServingTopicPartitionMap.poll();
customModeIdealStateBuilder.assignInstanceAndState(Integer.toString(i),
liveInstance.getInstanceName(), "ONLINE");
liveInstance.addTopicPartition(new TopicPartition(topicName, i));
instanceToNumServingTopicPartitionMap.add(liveInstance);
}
return customModeIdealStateBuilder.build();
}
开发者ID:uber,项目名称:uReplicator,代码行数:31,代码来源:IdealStateBuilder.java
示例5: cleanUpResource
import org.apache.helix.model.builder.CustomModeISBuilder; //导入依赖的package包/类
private boolean cleanUpResource(String resource, String resourceDir, FileSetInfo fileSetInfo)
throws Exception {
if (isResourceOfflined(resource)) {
LOG.info("Dropping resource " + resource);
helixAdmin.dropResource(clusterName, resource);
zkManager.deleteViewInfo(resource);
// Perform a recursive delete of the version directory.
LOG.info("Cleaning up " + resourceDir);
hdfsClient.delete(resourceDir, true);
return true;
} else {
LOG.info("Offlining " + resourceDir);
boolean enableZkCompression = configuration.getBoolean(
Constants.ENABLE_ZK_COMPRESSION, Constants.ENABLE_ZK_COMPRESSION_DEFAULT);
int bucketSize = TerrapinUtil.getBucketSize(
fileSetInfo.servingInfo.numPartitions, enableZkCompression);
if (bucketSize > 0) {
LOG.info("Disabling resource " + resource);
helixAdmin.enableResource(clusterName, resource, false);
} else {
CustomModeISBuilder offlineStateBuilder = new CustomModeISBuilder(resource);
offlineStateBuilder.setStateModel("OnlineOffline");
offlineStateBuilder.setNumReplica(configuration.getInt(Constants.NUM_SERVING_REPLICAS, 3));
offlineStateBuilder.setNumPartitions(fileSetInfo.servingInfo.numPartitions);
IdealState offlinedState = offlineStateBuilder.build();
offlinedState.setRebalanceMode(IdealState.RebalanceMode.CUSTOMIZED);
if (enableZkCompression) {
TerrapinUtil.compressIdealState(offlinedState);
}
helixAdmin.setResourceIdealState(clusterName, resource, offlinedState);
}
return false;
}
}
开发者ID:pinterest-attic,项目名称:terrapin,代码行数:36,代码来源:HdfsManager.java
示例6: testCalculateDeviationForResource
import org.apache.helix.model.builder.CustomModeISBuilder; //导入依赖的package包/类
@Test
public void testCalculateDeviationForResource() {
String resource = "resource";
CustomModeISBuilder isBuilder = new CustomModeISBuilder(resource);
isBuilder.assignInstanceAndState("0", "host1", "ONLINE");
isBuilder.assignInstanceAndState("0", "host2", "ONLINE");
isBuilder.assignInstanceAndState("1", "host2", "ONLINE");
isBuilder.assignInstanceAndState("1", "host3", "ONLINE");
isBuilder.setNumPartitions(2);
isBuilder.setNumReplica(3);
isBuilder.setRebalancerMode(IdealState.RebalanceMode.CUSTOMIZED);
isBuilder.setStateModel("OnlineOffline");
IdealState idealState = isBuilder.build();
// Check with deviation.
Map<String, List<InstanceConfig>> onlinePartitionMap1 = (Map)ImmutableMap.of(
"0", ImmutableList.of(hostConfig1, hostConfig2),
"1", ImmutableList.of(hostConfig3, hostConfig4));
TestRoutingTableProvider routingTableProvider =
new TestRoutingTableProvider(ImmutableMap.of(resource, onlinePartitionMap1));
assertEquals(0.25, HdfsManager.calculateDeviationForResource(
resource, idealState, routingTableProvider), 0.01);
// Check perfect match.
Map<String, List<InstanceConfig>> onlinePartitionMap2 = (Map)ImmutableMap.of(
"0", ImmutableList.of(hostConfig1, hostConfig2),
"1", ImmutableList.of(hostConfig2, hostConfig3));
routingTableProvider.setOnlinePartitionMap(ImmutableMap.of(resource, onlinePartitionMap2));
assertEquals(0.0, HdfsManager.calculateDeviationForResource(
resource, idealState, routingTableProvider), 0.01);
}
开发者ID:pinterest-attic,项目名称:terrapin,代码行数:32,代码来源:HdfsManagerTest.java
示例7: buildEmptyIdealStateFor
import org.apache.helix.model.builder.CustomModeISBuilder; //导入依赖的package包/类
/**
*
* Building an empty idealState for a given table.
* Used when creating a new table.
*
* @param tableName
* @param helixAdmin
* @param helixClusterName
* @return
*/
public static IdealState buildEmptyIdealStateFor(String tableName, int numCopies, HelixAdmin helixAdmin,
String helixClusterName) {
final CustomModeISBuilder customModeIdealStateBuilder = new CustomModeISBuilder(tableName);
final int replicas = numCopies;
customModeIdealStateBuilder
.setStateModel(PinotHelixSegmentOnlineOfflineStateModelGenerator.PINOT_SEGMENT_ONLINE_OFFLINE_STATE_MODEL)
.setNumPartitions(0).setNumReplica(replicas).setMaxPartitionsPerNode(1);
final IdealState idealState = customModeIdealStateBuilder.build();
idealState.setInstanceGroupTag(tableName);
return idealState;
}
开发者ID:Hanmourang,项目名称:Pinot,代码行数:22,代码来源:PinotTableIdealStateBuilder.java
示例8: buildEmptyIdealStateForBrokerResource
import org.apache.helix.model.builder.CustomModeISBuilder; //导入依赖的package包/类
/**
*
* Building an empty idealState for a given table.
* Used when creating a new table.
*
* @param helixAdmin
* @param helixClusterName
* @return
*/
public static IdealState buildEmptyIdealStateForBrokerResource(HelixAdmin helixAdmin, String helixClusterName) {
final CustomModeISBuilder customModeIdealStateBuilder =
new CustomModeISBuilder(CommonConstants.Helix.BROKER_RESOURCE_INSTANCE);
customModeIdealStateBuilder
.setStateModel(
PinotHelixBrokerResourceOnlineOfflineStateModelGenerator.PINOT_BROKER_RESOURCE_ONLINE_OFFLINE_STATE_MODEL)
.setMaxPartitionsPerNode(Integer.MAX_VALUE).setNumReplica(Integer.MAX_VALUE)
.setNumPartitions(Integer.MAX_VALUE);
final IdealState idealState = customModeIdealStateBuilder.build();
return idealState;
}
开发者ID:Hanmourang,项目名称:Pinot,代码行数:21,代码来源:PinotTableIdealStateBuilder.java
示例9: buildInitialKafkaHighLevelConsumerRealtimeIdealStateFor
import org.apache.helix.model.builder.CustomModeISBuilder; //导入依赖的package包/类
public static IdealState buildInitialKafkaHighLevelConsumerRealtimeIdealStateFor(String realtimeTableName,
HelixAdmin helixAdmin, String helixClusterName, ZkHelixPropertyStore<ZNRecord> zkHelixPropertyStore) {
final CustomModeISBuilder customModeIdealStateBuilder = new CustomModeISBuilder(realtimeTableName);
customModeIdealStateBuilder
.setStateModel(PinotHelixSegmentOnlineOfflineStateModelGenerator.PINOT_SEGMENT_ONLINE_OFFLINE_STATE_MODEL)
.setNumPartitions(0).setNumReplica(1).setMaxPartitionsPerNode(1);
final IdealState idealState = customModeIdealStateBuilder.build();
idealState.setInstanceGroupTag(realtimeTableName);
return idealState;
}
开发者ID:Hanmourang,项目名称:Pinot,代码行数:12,代码来源:PinotTableIdealStateBuilder.java
示例10: buildWorkflowIdealState
import org.apache.helix.model.builder.CustomModeISBuilder; //导入依赖的package包/类
private IdealState buildWorkflowIdealState(String workflow) {
CustomModeISBuilder IsBuilder = new CustomModeISBuilder(workflow);
IsBuilder.setRebalancerMode(IdealState.RebalanceMode.TASK).setNumReplica(1)
.setNumPartitions(1).setStateModel(TaskConstants.STATE_MODEL_NAME).disableExternalView();
IdealState is = IsBuilder.build();
is.getRecord().setListField(workflow, new ArrayList<String>());
is.getRecord().setMapField(workflow, new HashMap<String, String>());
is.setRebalancerClassName(WorkflowRebalancer.class.getName());
return is;
}
开发者ID:apache,项目名称:helix,代码行数:13,代码来源:TaskDriver.java
示例11: buildEmptyIdealStateFor
import org.apache.helix.model.builder.CustomModeISBuilder; //导入依赖的package包/类
/**
*
* Building an empty idealState for a given table.
* Used when creating a new table.
*
* @param tableName resource name
* @param numCopies is the number of replicas
* @return
*/
public static IdealState buildEmptyIdealStateFor(String tableName, int numCopies) {
final CustomModeISBuilder customModeIdealStateBuilder = new CustomModeISBuilder(tableName);
final int replicas = numCopies;
customModeIdealStateBuilder
.setStateModel(PinotHelixSegmentOnlineOfflineStateModelGenerator.PINOT_SEGMENT_ONLINE_OFFLINE_STATE_MODEL)
.setNumPartitions(0).setNumReplica(replicas).setMaxPartitionsPerNode(1);
final IdealState idealState = customModeIdealStateBuilder.build();
idealState.setInstanceGroupTag(tableName);
return idealState;
}
开发者ID:linkedin,项目名称:pinot,代码行数:20,代码来源:PinotTableIdealStateBuilder.java
示例12: buildEmptyKafkaConsumerRealtimeIdealStateFor
import org.apache.helix.model.builder.CustomModeISBuilder; //导入依赖的package包/类
public static IdealState buildEmptyKafkaConsumerRealtimeIdealStateFor(String realtimeTableName, int replicaCount) {
final CustomModeISBuilder customModeIdealStateBuilder = new CustomModeISBuilder(realtimeTableName);
customModeIdealStateBuilder
.setStateModel(PinotHelixSegmentOnlineOfflineStateModelGenerator.PINOT_SEGMENT_ONLINE_OFFLINE_STATE_MODEL)
.setNumPartitions(0).setNumReplica(replicaCount).setMaxPartitionsPerNode(1);
final IdealState idealState = customModeIdealStateBuilder.build();
idealState.setInstanceGroupTag(realtimeTableName);
return idealState;
}
开发者ID:linkedin,项目名称:pinot,代码行数:11,代码来源:PinotTableIdealStateBuilder.java
示例13: testLoadFileSet
import org.apache.helix.model.builder.CustomModeISBuilder; //导入依赖的package包/类
@Test
@PrepareForTest({ControllerUtil.class})
public void testLoadFileSet() throws Exception {
PowerMockito.mockStatic(ControllerUtil.class);
TerrapinLoadRequest request =
new TerrapinLoadRequest("fileset", "/terrapin/data/fileset/12345564534", 200);
Options requestOptions = new Options();
requestOptions.setNumVersionsToKeep(2);
request.setOptions(requestOptions);
FileSetInfo fileSetInfo = new FileSetInfo(
request.getFileSet(), "/terrapin/data/fileset/12345563434",
request.getExpectedNumPartitions(),
Lists.newArrayList(mock(FileSetInfo.ServingInfo.class)), new Options()
);
String resourceName = TerrapinUtil.hdfsDirToHelixResource(request.getHdfsDirectory());
CustomModeISBuilder idealStateBuilder = new CustomModeISBuilder(resourceName);
idealStateBuilder.assignInstanceAndState(resourceName + "$0", "host0", "ONLINE");
idealStateBuilder.assignInstanceAndState(resourceName + "$1", "host1", "ONLINE");
idealStateBuilder.setStateModel("OnlineOffline");
idealStateBuilder.setNumReplica(2);
idealStateBuilder.setNumPartitions(request.getExpectedNumPartitions());
IdealState is = idealStateBuilder.build();
is.setBucketSize(2);
is.setRebalanceMode(IdealState.RebalanceMode.CUSTOMIZED);
ViewInfo viewInfo1 = mock(ViewInfo.class);
ViewInfo viewInfo2 = mock(ViewInfo.class);
when(configuration.getInt(eq(Constants.NUM_SERVING_REPLICAS), eq(3))).thenReturn(3);
when(zkManager.getFileSetInfo(eq(request.getFileSet()))).thenReturn(fileSetInfo);
when(zkManager.getViewInfo(eq(resourceName))).thenReturn(viewInfo1).thenReturn(viewInfo2);
doNothing().when(zkManager).setFileSetInfo(eq(request.getFileSet()), any(FileSetInfo.class));
when(viewInfo1.getNumOnlinePartitions()).thenReturn(request.getExpectedNumPartitions() / 2);
when(viewInfo2.getNumOnlinePartitions()).thenReturn(request.getExpectedNumPartitions());
when(helixAdmin.getResourcesInCluster(CLUSTER)).thenReturn(new ArrayList<String>());
when(ControllerUtil.buildIdealStateForHdfsDir(any(DFSClient.class),
anyString(), anyString(), any(PartitionerType.class), anyInt(), anyBoolean())).
thenReturn(is);
doNothing().when(helixAdmin).addResource(eq(CLUSTER), eq(resourceName), eq(is));
doNothing().when(helixAdmin).addResource(eq(CLUSTER), eq(resourceName),
eq(is.getNumPartitions()), eq("OnlineOffline"), eq("CUSTOMIZED"), eq(is.getBucketSize()));
doNothing().when(helixAdmin).setResourceIdealState(eq(CLUSTER), eq(resourceName), eq(is));
serviceImpl.loadFileSet(request).apply();
ArgumentCaptor<FileSetInfo> fileSetInfoCaptor = ArgumentCaptor.forClass(FileSetInfo.class);
verify(zkManager).setFileSetInfo(eq(request.getFileSet()), fileSetInfoCaptor.capture());
FileSetInfo capturedInfo = fileSetInfoCaptor.getValue();
assertEquals(request.getFileSet(), capturedInfo.fileSetName);
assertEquals(request.getExpectedNumPartitions(), capturedInfo.servingInfo.numPartitions);
assertEquals(Lists.newArrayList(fileSetInfo.servingInfo),
(ArrayList)capturedInfo.oldServingInfoList);
}
开发者ID:pinterest-attic,项目名称:terrapin,代码行数:53,代码来源:TerrapinControllerServiceImplTest.java
示例14: scheduleSingleJob
import org.apache.helix.model.builder.CustomModeISBuilder; //导入依赖的package包/类
/**
* Posts new job to cluster
*/
private void scheduleSingleJob(String jobResource, JobConfig jobConfig) {
HelixAdmin admin = _manager.getClusterManagmentTool();
IdealState jobIS = admin.getResourceIdealState(_manager.getClusterName(), jobResource);
if (jobIS != null) {
LOG.info("Job " + jobResource + " idealstate already exists!");
return;
}
// Set up job resource based on partitions from target resource
TaskUtil.createUserContent(_manager.getHelixPropertyStore(), jobResource,
new ZNRecord(TaskUtil.USER_CONTENT_NODE));
int numIndependentTasks = jobConfig.getTaskConfigMap().size();
int numPartitions = numIndependentTasks;
if (numPartitions == 0) {
IdealState targetIs =
admin.getResourceIdealState(_manager.getClusterName(), jobConfig.getTargetResource());
if (targetIs == null) {
LOG.warn("Target resource does not exist for job " + jobResource);
// do not need to fail here, the job will be marked as failure immediately when job starts running.
} else {
numPartitions = targetIs.getPartitionSet().size();
}
}
admin.addResource(_manager.getClusterName(), jobResource, numPartitions,
TaskConstants.STATE_MODEL_NAME);
HelixDataAccessor accessor = _manager.getHelixDataAccessor();
// Set the job configuration
PropertyKey.Builder keyBuilder = accessor.keyBuilder();
HelixProperty resourceConfig = new HelixProperty(jobResource);
resourceConfig.getRecord().getSimpleFields().putAll(jobConfig.getResourceConfigMap());
Map<String, TaskConfig> taskConfigMap = jobConfig.getTaskConfigMap();
if (taskConfigMap != null) {
for (TaskConfig taskConfig : taskConfigMap.values()) {
resourceConfig.getRecord().setMapField(taskConfig.getId(), taskConfig.getConfigMap());
}
}
accessor.setProperty(keyBuilder.resourceConfig(jobResource), resourceConfig);
// Push out new ideal state based on number of target partitions
IdealStateBuilder builder = new CustomModeISBuilder(jobResource);
builder.setRebalancerMode(IdealState.RebalanceMode.TASK);
builder.setNumReplica(1);
builder.setNumPartitions(numPartitions);
builder.setStateModel(TaskConstants.STATE_MODEL_NAME);
if (jobConfig.getInstanceGroupTag() != null) {
builder.setNodeGroup(jobConfig.getInstanceGroupTag());
}
if (jobConfig.isDisableExternalView()) {
builder.disableExternalView();
}
jobIS = builder.build();
for (int i = 0; i < numPartitions; i++) {
jobIS.getRecord().setListField(jobResource + "_" + i, new ArrayList<String>());
jobIS.getRecord().setMapField(jobResource + "_" + i, new HashMap<String, String>());
}
jobIS.setRebalancerClassName(JobRebalancer.class.getName());
admin.setResourceIdealState(_manager.getClusterName(), jobResource, jobIS);
}
开发者ID:apache,项目名称:helix,代码行数:70,代码来源:WorkflowRebalancer.java
注:本文中的org.apache.helix.model.builder.CustomModeISBuilder类示例整理自Github/MSDocs等源码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。 |
请发表评论