本文整理汇总了Java中com.facebook.presto.spi.ConnectorSplitSource类的典型用法代码示例。如果您正苦于以下问题：Java ConnectorSplitSource类的具体用法？Java ConnectorSplitSource怎么用？Java ConnectorSplitSource使用的例子？那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。
ConnectorSplitSource类属于com.facebook.presto.spi包，在下文中一共展示了ConnectorSplitSource类的20个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于我们的系统推荐出更棒的Java代码示例。
示例1: getSplits
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Override
public ConnectorSplitSource getSplits(ConnectorTransactionHandle handle, ConnectorSession session, ConnectorTableLayoutHandle layout)
{
log.info("INFORMATION: AmpoolSplitManager getSplits() called.");
AmpoolTableLayoutHandle layoutHandle = (AmpoolTableLayoutHandle) layout;
AmpoolTableHandle tableHandle = layoutHandle.getTable();
AmpoolTable table = new AmpoolTable(ampoolClient, tableHandle.getTableName());
// this can happen if table is removed during a query
checkState(table.getColumnsMetadata() != null, "Table %s.%s no longer exists", tableHandle.getSchemaName(), tableHandle.getTableName());
List<ConnectorSplit> splits = new ArrayList<>();
// TODO Pass here bucket id
splits.add(new AmpoolSplit(connectorId, tableHandle.getSchemaName(), tableHandle.getTableName(),"" ,HostAddress.fromParts("localhost",0)));
Collections.shuffle(splits);
return new FixedSplitSource(splits);
}
开发者ID:ampool，项目名称:monarch，代码行数:19，代码来源:AmpoolSplitManager.java
示例2: getSplits
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Override
public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHandle, ConnectorSession session, ConnectorTableLayoutHandle layout)
{
KuduTableLayoutHandle layoutHandle = checkType(layout, KuduTableLayoutHandle.class, "layout");
KuduTableHandle tableHandle = layoutHandle.getTable();
KuduClient kuduClient = kuduClientManager.getClient();
List<KuduScanToken> tokens = kuduClientManager.newScanTokenBuilder(kuduClient, tableHandle.getSchemaTableName().getTableName()).build();
TupleDomain<KuduColumnHandle> effectivePredicate = layoutHandle.getConstraint()
.transform(handle -> checkType(handle, KuduColumnHandle.class, "columnHandle"));
ImmutableList.Builder<ConnectorSplit> builder = ImmutableList.builder();
for (int i = 0; i < tokens.size(); i++) {
// nodeManager.getWorkerNodes()
List<HostAddress> hostAddresses = nodeManager.getWorkerNodes().stream()
.map(node -> node.getHostAndPort()).collect(Collectors.toList());
ConnectorSplit split = new KuduSplit(hostAddresses, tableHandle.getSchemaTableName(), i, effectivePredicate);
builder.add(split);
}
kuduClientManager.close(kuduClient);
return new FixedSplitSource(builder.build());
}
开发者ID:trackingio，项目名称:presto-kudu，代码行数:26，代码来源:KuduSplitManager.java
示例3: getSplits
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Override
public ConnectorSplitSource getSplits(ConnectorSession session, ConnectorTableLayoutHandle layout)
{
ExampleTableLayoutHandle layoutHandle = checkType(layout, ExampleTableLayoutHandle.class, "layout");
ExampleTableHandle tableHandle = layoutHandle.getTable();
ExampleTable table = exampleClient.getTable(tableHandle.getSchemaName(), tableHandle.getTableName());
// this can happen if table is removed during a query
checkState(table != null, "Table %s.%s no longer exists", tableHandle.getSchemaName(), tableHandle.getTableName());
List<ConnectorSplit> splits = new ArrayList<>();
for (URI uri : table.getSources()) {
splits.add(new ExampleSplit(connectorId, tableHandle.getSchemaName(), tableHandle.getTableName(), uri));
}
Collections.shuffle(splits);
return new FixedSplitSource(connectorId, splits);
}
开发者ID:y-lan，项目名称:presto，代码行数:18，代码来源:ExampleSplitManager.java
示例4: testSanity
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Test
public void testSanity()
throws InterruptedException
{
List<ConnectorTableLayoutResult> layouts = metadata.getTableLayouts(SESSION, tableHandle, Constraint.alwaysTrue(), Optional.empty());
assertEquals(layouts.size(), 1);
ConnectorTableLayoutResult layout = getOnlyElement(layouts);
assertInstanceOf(layout.getTableLayout().getHandle(), RaptorTableLayoutHandle.class);
ConnectorSplitSource splitSource = getSplits(raptorSplitManager, layout);
int splitCount = 0;
while (!splitSource.isFinished()) {
splitCount += getFutureValue(splitSource.getNextBatch(1000)).size();
}
assertEquals(splitCount, 4);
}
开发者ID:y-lan，项目名称:presto，代码行数:17，代码来源:TestRaptorSplitManager.java
示例5: testAssignRandomNodeWhenBackupAvailable
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Test
public void testAssignRandomNodeWhenBackupAvailable()
throws InterruptedException, URISyntaxException
{
InMemoryNodeManager nodeManager = new InMemoryNodeManager();
RaptorConnectorId connectorId = new RaptorConnectorId("raptor");
NodeSupplier nodeSupplier = new RaptorNodeSupplier(nodeManager, connectorId);
PrestoNode node = new PrestoNode(UUID.randomUUID().toString(), new URI("http://127.0.0.1/"), NodeVersion.UNKNOWN);
nodeManager.addNode(connectorId.toString(), node);
RaptorSplitManager raptorSplitManagerWithBackup = new RaptorSplitManager(connectorId, nodeSupplier, shardManager, true);
deleteShardNodes();
ConnectorTableLayoutResult layout = getOnlyElement(metadata.getTableLayouts(SESSION, tableHandle, Constraint.alwaysTrue(), Optional.empty()));
ConnectorSplitSource partitionSplit = getSplits(raptorSplitManagerWithBackup, layout);
List<ConnectorSplit> batch = getFutureValue(partitionSplit.getNextBatch(1), PrestoException.class);
assertEquals(getOnlyElement(getOnlyElement(batch).getAddresses()), node.getHostAndPort());
}
开发者ID:y-lan，项目名称:presto，代码行数:19，代码来源:TestRaptorSplitManager.java
示例6: getSplits
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Override
public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHandle, ConnectorSession session, ConnectorTableLayoutHandle layoutHandle)
{
BlackHoleTableLayoutHandle layout = checkType(
layoutHandle,
BlackHoleTableLayoutHandle.class,
"BlackHoleTableLayoutHandle");
ImmutableList.Builder<BlackHoleSplit> builder = ImmutableList.<BlackHoleSplit>builder();
for (int i = 0; i < layout.getSplitCount(); i++) {
builder.add(
new BlackHoleSplit(
layout.getPagesPerSplit(),
layout.getRowsPerPage(),
layout.getFieldsLength()));
}
return new FixedSplitSource("blackhole", builder.build());
}
开发者ID:y-lan，项目名称:presto，代码行数:20，代码来源:BlackHoleSplitManager.java
示例7: getSplits
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Override
public ConnectorSplitSource getSplits(ConnectorTransactionHandle transaction, ConnectorSession session, ConnectorTableLayoutHandle layout)
{
JmxTableLayoutHandle jmxLayout = checkType(layout, JmxTableLayoutHandle.class, "layout");
JmxTableHandle tableHandle = jmxLayout.getTable();
TupleDomain<ColumnHandle> predicate = jmxLayout.getConstraint();
//TODO is there a better way to get the node column?
JmxColumnHandle nodeColumnHandle = tableHandle.getColumns().get(0);
List<ConnectorSplit> splits = nodeManager.getNodes(ACTIVE)
.stream()
.filter(node -> {
NullableValue value = NullableValue.of(VARCHAR, utf8Slice(node.getNodeIdentifier()));
return predicate.overlaps(fromFixedValues(ImmutableMap.of(nodeColumnHandle, value)));
})
.map(node -> new JmxSplit(tableHandle, ImmutableList.of(node.getHostAndPort())))
.collect(toList());
return new FixedSplitSource(connectorId, splits);
}
开发者ID:y-lan，项目名称:presto，代码行数:22，代码来源:JmxSplitManager.java
示例8: testPredicatePushdown
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Test
public void testPredicatePushdown()
throws Exception
{
for (Node node : nodes) {
String nodeIdentifier = node.getNodeIdentifier();
TupleDomain<ColumnHandle> nodeTupleDomain = TupleDomain.fromFixedValues(ImmutableMap.of(columnHandle, NullableValue.of(VARCHAR, utf8Slice(nodeIdentifier))));
ConnectorTableLayoutHandle layout = new JmxTableLayoutHandle(tableHandle, nodeTupleDomain);
ConnectorSplitSource splitSource = splitManager.getSplits(JmxTransactionHandle.INSTANCE, SESSION, layout);
List<ConnectorSplit> allSplits = getAllSplits(splitSource);
assertEquals(allSplits.size(), 1);
assertEquals(allSplits.get(0).getAddresses().size(), 1);
assertEquals(allSplits.get(0).getAddresses().get(0).getHostText(), nodeIdentifier);
}
}
开发者ID:y-lan，项目名称:presto，代码行数:18，代码来源:TestJmxSplitManager.java
示例9: testNoPredicate
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Test
public void testNoPredicate()
throws Exception
{
ConnectorTableLayoutHandle layout = new JmxTableLayoutHandle(tableHandle, TupleDomain.all());
ConnectorSplitSource splitSource = splitManager.getSplits(JmxTransactionHandle.INSTANCE, SESSION, layout);
List<ConnectorSplit> allSplits = getAllSplits(splitSource);
assertEquals(allSplits.size(), nodes.size());
Set<String> actualNodes = nodes.stream().map(Node::getNodeIdentifier).collect(toSet());
Set<String> expectedNodes = new HashSet<>();
for (ConnectorSplit split : allSplits) {
List<HostAddress> addresses = split.getAddresses();
assertEquals(addresses.size(), 1);
expectedNodes.add(addresses.get(0).getHostText());
}
assertEquals(actualNodes, expectedNodes);
}
开发者ID:y-lan，项目名称:presto，代码行数:19，代码来源:TestJmxSplitManager.java
示例10: testRecordSetProvider
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Test
public void testRecordSetProvider()
throws Exception
{
for (SchemaTableName schemaTableName : metadata.listTables(SESSION, "jmx")) {
JmxTableHandle tableHandle = metadata.getTableHandle(SESSION, schemaTableName);
List<ColumnHandle> columnHandles = ImmutableList.copyOf(metadata.getColumnHandles(SESSION, tableHandle).values());
ConnectorTableLayoutHandle layout = new JmxTableLayoutHandle(tableHandle, TupleDomain.all());
ConnectorSplitSource splitSource = splitManager.getSplits(JmxTransactionHandle.INSTANCE, SESSION, layout);
List<ConnectorSplit> allSplits = getAllSplits(splitSource);
assertEquals(allSplits.size(), nodes.size());
ConnectorSplit split = allSplits.get(0);
RecordSet recordSet = recordSetProvider.getRecordSet(JmxTransactionHandle.INSTANCE, SESSION, split, columnHandles);
try (RecordCursor cursor = recordSet.cursor()) {
while (cursor.advanceNextPosition()) {
for (int i = 0; i < recordSet.getColumnTypes().size(); i++) {
cursor.isNull(i);
}
}
}
}
}
开发者ID:y-lan，项目名称:presto，代码行数:25，代码来源:TestJmxSplitManager.java
示例11: getSplits
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Override
public ConnectorSplitSource getSplits(ConnectorTransactionHandle transaction, ConnectorSession session, ConnectorTableLayoutHandle layout)
{
InformationSchemaTableLayoutHandle handle = checkType(layout, InformationSchemaTableLayoutHandle.class, "layout");
Map<ColumnHandle, NullableValue> bindings = extractFixedValues(handle.getConstraint()).orElse(ImmutableMap.of());
List<HostAddress> localAddress = ImmutableList.of(nodeManager.getCurrentNode().getHostAndPort());
Map<String, NullableValue> filters = bindings.entrySet().stream().collect(toMap(
entry -> checkType(entry.getKey(), InformationSchemaColumnHandle.class, "column").getColumnName(),
Entry::getValue));
ConnectorSplit split = new InformationSchemaSplit(handle.getTable(), filters, localAddress);
return new FixedSplitSource(null, ImmutableList.of(split));
}
开发者ID:y-lan，项目名称:presto，代码行数:17，代码来源:InformationSchemaSplitManager.java
示例12: getSplits
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Override
public ConnectorSplitSource getSplits(ConnectorTransactionHandle transaction, ConnectorSession session, ConnectorTableLayoutHandle layout)
{
TpchTableHandle tableHandle = checkType(layout, TpchTableLayoutHandle.class, "layout").getTable();
Set<Node> nodes = nodeManager.getActiveDatasourceNodes(connectorId);
checkState(!nodes.isEmpty(), "No TPCH nodes available");
int totalParts = nodes.size() * splitsPerNode;
int partNumber = 0;
// Split the data using split and skew by the number of nodes available.
ImmutableList.Builder<ConnectorSplit> splits = ImmutableList.builder();
for (Node node : nodes) {
for (int i = 0; i < splitsPerNode; i++) {
splits.add(new TpchSplit(tableHandle, partNumber, totalParts, ImmutableList.of(node.getHostAndPort())));
partNumber++;
}
}
return new FixedSplitSource(connectorId, splits.build());
}
开发者ID:y-lan，项目名称:presto，代码行数:22，代码来源:TpchSplitManager.java
示例13: getSplits
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Override
public ConnectorSplitSource getSplits(ConnectorTransactionHandle transaction, ConnectorSession session, ConnectorTableLayoutHandle layout)
{
CassandraTableLayoutHandle layoutHandle = checkType(layout, CassandraTableLayoutHandle.class, "layout");
CassandraTableHandle cassandraTableHandle = layoutHandle.getTable();
List<CassandraPartition> partitions = layoutHandle.getPartitions().get();
requireNonNull(partitions, "partitions is null");
if (partitions.isEmpty()) {
return new FixedSplitSource(connectorId, ImmutableList.<ConnectorSplit>of());
}
// if this is an unpartitioned table, split into equal ranges
if (partitions.size() == 1) {
CassandraPartition cassandraPartition = partitions.get(0);
if (cassandraPartition.isUnpartitioned() || cassandraPartition.isIndexedColumnPredicatePushdown()) {
CassandraTable table = schemaProvider.getTable(cassandraTableHandle);
List<ConnectorSplit> splits = getSplitsByTokenRange(table, cassandraPartition.getPartitionId());
return new FixedSplitSource(connectorId, splits);
}
}
return new FixedSplitSource(connectorId, getSplitsForPartitions(cassandraTableHandle, partitions));
}
开发者ID:y-lan，项目名称:presto，代码行数:25，代码来源:CassandraSplitManager.java
示例14: testGetRecordsS3
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Test
public void testGetRecordsS3()
throws Exception
{
ConnectorTableHandle table = getTableHandle(tableS3);
List<ColumnHandle> columnHandles = ImmutableList.copyOf(metadata.getColumnHandles(SESSION, table).values());
Map<String, Integer> columnIndex = indexColumns(columnHandles);
List<ConnectorTableLayoutResult> tableLayoutResults = metadata.getTableLayouts(SESSION, table, new Constraint<>(TupleDomain.all(), bindings -> true), Optional.empty());
HiveTableLayoutHandle layoutHandle = (HiveTableLayoutHandle) getOnlyElement(tableLayoutResults).getTableLayout().getHandle();
assertEquals(layoutHandle.getPartitions().get().size(), 1);
ConnectorSplitSource splitSource = splitManager.getSplits(SESSION, layoutHandle);
long sum = 0;
for (ConnectorSplit split : getAllSplits(splitSource)) {
try (ConnectorPageSource pageSource = pageSourceProvider.createPageSource(SESSION, split, columnHandles)) {
MaterializedResult result = materializeSourceDataStream(SESSION, pageSource, getTypes(columnHandles));
for (MaterializedRow row : result) {
sum += (Long) row.getField(columnIndex.get("t_bigint"));
}
}
}
assertEquals(sum, 78300);
}
开发者ID:y-lan，项目名称:presto，代码行数:27，代码来源:AbstractTestHiveClientS3.java
示例15: testPartitionSchemaNonCanonical
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Test
public void testPartitionSchemaNonCanonical()
throws Exception
{
ConnectorSession session = newSession();
ConnectorTableHandle table = getTableHandle(tablePartitionSchemaChangeNonCanonical);
ColumnHandle column = metadata.getColumnHandles(session, table).get("t_boolean");
assertNotNull(column);
List<ConnectorTableLayoutResult> tableLayoutResults = metadata.getTableLayouts(session, table, new Constraint<>(TupleDomain.fromFixedValues(ImmutableMap.of(column, NullableValue.of(BOOLEAN, false))), bindings -> true), Optional.empty());
ConnectorTableLayoutHandle layoutHandle = getOnlyElement(tableLayoutResults).getTableLayout().getHandle();
assertEquals(getAllPartitions(layoutHandle).size(), 1);
assertEquals(getPartitionId(getAllPartitions(layoutHandle).get(0)), "t_boolean=0");
ConnectorSplitSource splitSource = splitManager.getSplits(session, layoutHandle);
ConnectorSplit split = getOnlyElement(getAllSplits(splitSource));
ImmutableList<ColumnHandle> columnHandles = ImmutableList.of(column);
try (ConnectorPageSource ignored = pageSourceProvider.createPageSource(session, split, columnHandles)) {
// TODO coercion of non-canonical values should be supported
fail("expected exception");
}
catch (PrestoException e) {
assertEquals(e.getErrorCode(), HIVE_INVALID_PARTITION_VALUE.toErrorCode());
}
}
开发者ID:y-lan，项目名称:presto，代码行数:27，代码来源:AbstractTestHiveClient.java
示例16: getSplits
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Override
public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHandle, ConnectorSession session, ConnectorTableLayoutHandle layout)
{
KinesisTableLayoutHandle kinesislayout = handleResolver.convertLayout(layout);
KinesisTableHandle kinesisTableHandle = kinesislayout.getTable();
InternalStreamDescription desc = this.getStreamDescription(kinesisTableHandle.getStreamName());
ImmutableList.Builder<ConnectorSplit> builder = ImmutableList.builder();
for (Shard shard : desc.getShards()) {
KinesisSplit split = new KinesisSplit(connectorId,
kinesisTableHandle.getStreamName(),
kinesisTableHandle.getMessageDataFormat(),
shard.getShardId(),
shard.getSequenceNumberRange().getStartingSequenceNumber(),
shard.getSequenceNumberRange().getEndingSequenceNumber());
builder.add(split);
}
return new FixedSplitSource(builder.build());
}
开发者ID:qubole，项目名称:presto-kinesis，代码行数:22，代码来源:KinesisSplitManager.java
示例17: getSplits
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Override
public ConnectorSplitSource getSplits(ConnectorTransactionHandle transactionHandle, ConnectorSession session, ConnectorTableLayoutHandle layout)
{
RestConnectorTableLayoutHandle layoutHandle = Types.checkType(layout, RestConnectorTableLayoutHandle.class, "layout");
List<HostAddress> addresses = nodeManager.getRequiredWorkerNodes().stream()
.map(Node::getHostAndPort)
.collect(toList());
return new FixedSplitSource(ImmutableList.of(
new RestConnectorSplit(layoutHandle.getTableHandle(), addresses)));
}
开发者ID:prestodb-rocks，项目名称:presto-rest，代码行数:13，代码来源:RestSplitManager.java
示例18: getSplits
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Override
public ConnectorSplitSource getSplits(ConnectorTransactionHandle transaction, ConnectorSession session, ConnectorTableLayoutHandle layout)
{
RaptorTableLayoutHandle handle = checkType(layout, RaptorTableLayoutHandle.class, "layout");
RaptorTableHandle table = handle.getTable();
TupleDomain<RaptorColumnHandle> effectivePredicate = toRaptorTupleDomain(handle.getConstraint());
return new RaptorSplitSource(table.getTableId(), effectivePredicate, table.getTransactionId());
}
开发者ID:y-lan，项目名称:presto，代码行数:9，代码来源:RaptorSplitManager.java
示例19: testNoHostForShard
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Test(expectedExceptions = PrestoException.class, expectedExceptionsMessageRegExp = "No host for shard .* found: \\[\\]")
public void testNoHostForShard()
throws InterruptedException
{
deleteShardNodes();
ConnectorTableLayoutResult layout = getOnlyElement(metadata.getTableLayouts(SESSION, tableHandle, Constraint.alwaysTrue(), Optional.empty()));
ConnectorSplitSource splitSource = getSplits(raptorSplitManager, layout);
getFutureValue(splitSource.getNextBatch(1000));
}
开发者ID:y-lan，项目名称:presto，代码行数:11，代码来源:TestRaptorSplitManager.java
示例20: testNoNodes
import com.facebook.presto.spi.ConnectorSplitSource; //导入依赖的package包/类
@Test(expectedExceptions = PrestoException.class, expectedExceptionsMessageRegExp = "No nodes available to run query")
public void testNoNodes()
throws InterruptedException, URISyntaxException
{
deleteShardNodes();
RaptorSplitManager raptorSplitManagerWithBackup = new RaptorSplitManager(new RaptorConnectorId("fbraptor"), ImmutableSet::of, shardManager, true);
ConnectorTableLayoutResult layout = getOnlyElement(metadata.getTableLayouts(SESSION, tableHandle, Constraint.alwaysTrue(), Optional.empty()));
ConnectorSplitSource splitSource = getSplits(raptorSplitManagerWithBackup, layout);
getFutureValue(splitSource.getNextBatch(1000), PrestoException.class);
}
开发者ID:y-lan，项目名称:presto，代码行数:12，代码来源:TestRaptorSplitManager.java
注：本文中的com.facebook.presto.spi.ConnectorSplitSource类示例整理自Github/MSDocs等源码及文档管理平台，相关代码片段筛选自各路编程大神贡献的开源项目，源码版权归原作者所有，传播和使用请参考对应项目的License；未经允许，请勿转载。 |
客服电话
APP下载
官方微信
请发表评论