• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    迪恩网络公众号

Java Datasets类代码示例

原作者: [db:作者] 来自: [db:来源] 收藏 邀请

本文整理汇总了Java中org.kitesdk.data.Datasets的典型用法代码示例。如果您正苦于以下问题:Java Datasets类的具体用法?Java Datasets怎么用?Java Datasets使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。



Datasets类属于org.kitesdk.data包,在下文中一共展示了Datasets类的20个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Java代码示例。

示例1: SavePolicy

import org.kitesdk.data.Datasets; //导入依赖的package包/类
private SavePolicy(Context context) {
  String uri = context.getString(CONFIG_KITE_ERROR_DATASET_URI);
  Preconditions.checkArgument(uri != null, "Must set "
      + CONFIG_KITE_ERROR_DATASET_URI + " when " + CONFIG_FAILURE_POLICY
      + "=save");
  if (Datasets.exists(uri)) {
    dataset = Datasets.load(uri, AvroFlumeEvent.class);
  } else {
    DatasetDescriptor descriptor = new DatasetDescriptor.Builder()
        .schema(AvroFlumeEvent.class)
        .build();
    dataset = Datasets.create(uri, descriptor, AvroFlumeEvent.class);
  }

  nEventsHandled = 0;
}
 
开发者ID:moueimei,项目名称:flume-release-1.7.0,代码行数:17,代码来源:SavePolicy.java


示例2: testOldConfig

import org.kitesdk.data.Datasets; //导入依赖的package包/类
@Test
public void testOldConfig() throws EventDeliveryException {
  config.put(DatasetSinkConstants.CONFIG_KITE_DATASET_URI, null);
  config.put(DatasetSinkConstants.CONFIG_KITE_REPO_URI, FILE_REPO_URI);
  config.put(DatasetSinkConstants.CONFIG_KITE_DATASET_NAME, DATASET_NAME);

  DatasetSink sink = sink(in, config);

  // run the sink
  sink.start();
  sink.process();
  sink.stop();

  Assert.assertEquals(
      Sets.newHashSet(expected),
      read(Datasets.load(FILE_DATASET_URI)));
  Assert.assertEquals("Should have committed", 0, remaining(in));
}
 
开发者ID:moueimei,项目名称:flume-release-1.7.0,代码行数:19,代码来源:TestDatasetSink.java


示例3: testDatasetUriOverridesOldConfig

import org.kitesdk.data.Datasets; //导入依赖的package包/类
@Test
public void testDatasetUriOverridesOldConfig() throws EventDeliveryException {
  // CONFIG_KITE_DATASET_URI is still set, otherwise this will cause an error
  config.put(DatasetSinkConstants.CONFIG_KITE_REPO_URI, "bad uri");
  config.put(DatasetSinkConstants.CONFIG_KITE_DATASET_NAME, "");

  DatasetSink sink = sink(in, config);

  // run the sink
  sink.start();
  sink.process();
  sink.stop();

  Assert.assertEquals(
      Sets.newHashSet(expected),
      read(Datasets.load(FILE_DATASET_URI)));
  Assert.assertEquals("Should have committed", 0, remaining(in));
}
 
开发者ID:moueimei,项目名称:flume-release-1.7.0,代码行数:19,代码来源:TestDatasetSink.java


示例4: testBatchSize

import org.kitesdk.data.Datasets; //导入依赖的package包/类
@Test
public void testBatchSize() throws EventDeliveryException {
  DatasetSink sink = sink(in, config);

  // release one record per process call
  config.put("kite.batchSize", "2");
  Configurables.configure(sink, config);

  sink.start();
  sink.process(); // process the first and second
  sink.roll(); // roll at the next process call
  sink.process(); // roll and process the third
  Assert.assertEquals(
      Sets.newHashSet(expected.subList(0, 2)),
      read(Datasets.load(FILE_DATASET_URI)));
  Assert.assertEquals("Should have committed", 0, remaining(in));
  sink.roll(); // roll at the next process call
  sink.process(); // roll, the channel is empty
  Assert.assertEquals(
      Sets.newHashSet(expected),
      read(Datasets.load(FILE_DATASET_URI)));
  sink.stop();
}
 
开发者ID:moueimei,项目名称:flume-release-1.7.0,代码行数:24,代码来源:TestDatasetSink.java


示例5: testTimedFileRolling

import org.kitesdk.data.Datasets; //导入依赖的package包/类
@Test
public void testTimedFileRolling()
    throws EventDeliveryException, InterruptedException {
  // use a new roll interval
  config.put("kite.rollInterval", "1"); // in seconds

  DatasetSink sink = sink(in, config);

  Dataset<GenericRecord> records = Datasets.load(FILE_DATASET_URI);

  // run the sink
  sink.start();
  sink.process();

  Assert.assertEquals("Should have committed", 0, remaining(in));

  Thread.sleep(1100); // sleep longer than the roll interval
  sink.process(); // rolling happens in the process method

  Assert.assertEquals(Sets.newHashSet(expected), read(records));

  // wait until the end to stop because it would close the files
  sink.stop();
}
 
开发者ID:moueimei,项目名称:flume-release-1.7.0,代码行数:25,代码来源:TestDatasetSink.java


示例6: testFileStoreWithSavePolicy

import org.kitesdk.data.Datasets; //导入依赖的package包/类
@Test
public void testFileStoreWithSavePolicy() throws EventDeliveryException {
  if (Datasets.exists(ERROR_DATASET_URI)) {
    Datasets.delete(ERROR_DATASET_URI);
  }
  config.put(DatasetSinkConstants.CONFIG_FAILURE_POLICY,
      DatasetSinkConstants.SAVE_FAILURE_POLICY);
  config.put(DatasetSinkConstants.CONFIG_KITE_ERROR_DATASET_URI,
      ERROR_DATASET_URI);
  DatasetSink sink = sink(in, config);

  // run the sink
  sink.start();
  sink.process();
  sink.stop();

  Assert.assertEquals(
      Sets.newHashSet(expected),
      read(Datasets.load(FILE_DATASET_URI)));
  Assert.assertEquals("Should have committed", 0, remaining(in));
}
 
开发者ID:moueimei,项目名称:flume-release-1.7.0,代码行数:22,代码来源:TestDatasetSink.java


示例7: testCommitOnBatch

import org.kitesdk.data.Datasets; //导入依赖的package包/类
@Test
public void testCommitOnBatch() throws EventDeliveryException {
  DatasetSink sink = sink(in, config);

  // run the sink
  sink.start();
  sink.process();

  // the transaction should commit during the call to process
  Assert.assertEquals("Should have committed", 0, remaining(in));
  // but the data won't be visible yet
  Assert.assertEquals(0,
      read(Datasets.load(FILE_DATASET_URI)).size());

  sink.stop();

  Assert.assertEquals(
      Sets.newHashSet(expected),
      read(Datasets.load(FILE_DATASET_URI)));
}
 
开发者ID:moueimei,项目名称:flume-release-1.7.0,代码行数:21,代码来源:TestDatasetSink.java


示例8: testCloseAndCreateWriter

import org.kitesdk.data.Datasets; //导入依赖的package包/类
@Test
public void testCloseAndCreateWriter() throws EventDeliveryException {
  config.put(DatasetSinkConstants.CONFIG_FLUSHABLE_COMMIT_ON_BATCH,
      Boolean.toString(false));
  config.put(DatasetSinkConstants.CONFIG_SYNCABLE_SYNC_ON_BATCH,
      Boolean.toString(false));
  DatasetSink sink = sink(in, config);

  // run the sink
  sink.start();
  sink.process();

  sink.closeWriter();
  sink.commitTransaction();
  sink.createWriter();

  Assert.assertNotNull("Writer should not be null", sink.getWriter());
  Assert.assertEquals("Should have committed", 0, remaining(in));

  sink.stop();

  Assert.assertEquals(
      Sets.newHashSet(expected),
      read(Datasets.load(FILE_DATASET_URI)));
}
 
开发者ID:moueimei,项目名称:flume-release-1.7.0,代码行数:26,代码来源:TestDatasetSink.java


示例9: testCloseWriter

import org.kitesdk.data.Datasets; //导入依赖的package包/类
@Test
public void testCloseWriter() throws EventDeliveryException {
  config.put(DatasetSinkConstants.CONFIG_FLUSHABLE_COMMIT_ON_BATCH,
      Boolean.toString(false));
  config.put(DatasetSinkConstants.CONFIG_SYNCABLE_SYNC_ON_BATCH,
      Boolean.toString(false));
  DatasetSink sink = sink(in, config);

  // run the sink
  sink.start();
  sink.process();

  sink.closeWriter();
  sink.commitTransaction();

  Assert.assertNull("Writer should be null", sink.getWriter());
  Assert.assertEquals("Should have committed", 0, remaining(in));

  sink.stop();

  Assert.assertEquals(
      Sets.newHashSet(expected),
      read(Datasets.load(FILE_DATASET_URI)));
}
 
开发者ID:moueimei,项目名称:flume-release-1.7.0,代码行数:25,代码来源:TestDatasetSink.java


示例10: testCreateWriter

import org.kitesdk.data.Datasets; //导入依赖的package包/类
@Test
public void testCreateWriter() throws EventDeliveryException {
  config.put(DatasetSinkConstants.CONFIG_FLUSHABLE_COMMIT_ON_BATCH,
      Boolean.toString(false));
  config.put(DatasetSinkConstants.CONFIG_SYNCABLE_SYNC_ON_BATCH,
      Boolean.toString(false));
  DatasetSink sink = sink(in, config);

  // run the sink
  sink.start();
  sink.process();

  sink.commitTransaction();
  sink.createWriter();
  Assert.assertNotNull("Writer should not be null", sink.getWriter());
  Assert.assertEquals("Should have committed", 0, remaining(in));

  sink.stop();

  Assert.assertEquals(0, read(Datasets.load(FILE_DATASET_URI)).size());
}
 
开发者ID:moueimei,项目名称:flume-release-1.7.0,代码行数:22,代码来源:TestDatasetSink.java


示例11: setUp

import org.kitesdk.data.Datasets; //导入依赖的package包/类
@BeforeMethod(alwaysRun = true)
public void setUp() {
  initMocks(this);
  mockStatic(KiteDatasetExecutor.class);
  mockStatic(Datasets.class);

  destroyer = new KiteToDestroyer() {
    @Override
    protected KiteDatasetExecutor getExecutor(String uri, Schema schema,
        FileFormat format) {
      return executorMock;
    }
  };

  linkConfig = new LinkConfiguration();
  toJobConfig = new ToJobConfiguration();
  toJobConfig.toJobConfig.uri = "dataset:file:/foo/bar";
  toJobConfig.toJobConfig.fileFormat = FileFormat.AVRO;
}
 
开发者ID:vybs,项目名称:sqoop-on-spark,代码行数:20,代码来源:TestKiteToDestroyer.java


示例12: dataset

import org.kitesdk.data.Datasets; //导入依赖的package包/类
/**
 * Ensures the given dataset exists, creating it if it doesn't
 * and updating the schema if necessary.
 */
protected void dataset(String uri, DatasetDescriptor descriptor) {

  try {

    Datasets.create(uri, descriptor);
  } catch (DatasetExistsException e) {

    Dataset existingDataset = Datasets.load(uri);

    DatasetDescriptor updated;

    // The given discriptor might not have a location,
    // so use the current one.
    if (descriptor.getLocation() == null) {
      updated = new DatasetDescriptor.Builder(descriptor)
          .location(existingDataset.getDescriptor().getLocation())
          .build();
    } else {

      updated = descriptor;
    }

    Datasets.update(uri, updated);
  }
}
 
开发者ID:rbrush,项目名称:kite-apps,代码行数:30,代码来源:AbstractApplication.java


示例13: run

import org.kitesdk.data.Datasets; //导入依赖的package包/类
@Override
public int run(String[] args) throws Exception {
  // Load the users dataset
  Dataset<Record> users = Datasets.load(
      "dataset:hdfs:/tmp/data/users", Record.class);

  // Get a reader for the dataset and read all the users
  DatasetReader<Record> reader = null;
  try {
    reader = users.newReader();
    for (GenericRecord user : reader) {
      System.out.println(user);
    }

  } finally {
    if (reader != null) {
      reader.close();
    }
  }

  return 0;
}
 
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:23,代码来源:ReadUserDatasetGeneric.java


示例14: run

import org.kitesdk.data.Datasets; //导入依赖的package包/类
@Override
public int run(String[] args) throws Exception {
  // Load the users dataset
  Dataset<Record> users = Datasets.load(
      "dataset:hive?dataset=users", Record.class);

  // Get a reader for the dataset and read all the users
  DatasetReader<Record> reader = null;
  try {
    reader = users.newReader();
    for (GenericRecord user : users.newReader()) {
      System.out.println(user);
    }

  } finally {
    if (reader != null) {
      reader.close();
    }
  }

  return 0;
}
 
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:23,代码来源:ReadHiveUserDatasetGeneric.java


示例15: run

import org.kitesdk.data.Datasets; //导入依赖的package包/类
@Override
public int run(String[] args) throws Exception {
  // Load the users dataset
  Dataset<Record> users = Datasets.load(
      "dataset:hdfs:/tmp/data/users", Record.class);

  // Get a reader for the dataset and read all the users
  DatasetReader<Record> reader = null;
  try {
    reader = users.with("favoriteColor", "green").newReader();
    for (GenericRecord user : reader) {
      System.out.println(user);
    }

  } finally {
    if (reader != null) {
      reader.close();
    }
  }

  return 0;
}
 
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:23,代码来源:ReadUserDatasetGenericOnePartition.java


示例16: run

import org.kitesdk.data.Datasets; //导入依赖的package包/类
@Override
public int run(String[] args) throws Exception {
  // Load the products dataset
  Dataset<Product> products = Datasets.load(
      "dataset:hdfs:/tmp/data/products", Product.class);

  // Get a reader for the dataset and read all the users
  DatasetReader<Product> reader = null;
  try {
    reader = products.newReader();
    for (Product product : reader) {
      System.out.println(product);
    }

  } finally {
    if (reader != null) {
      reader.close();
    }
  }

  return 0;
}
 
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:23,代码来源:ReadProductDatasetPojo.java


示例17: run

import org.kitesdk.data.Datasets; //导入依赖的package包/类
@Override
public int run(String[] args) throws Exception {

  // Load the events dataset
  Dataset<GenericRecord> events = Datasets.load("dataset:hive:/tmp/data/default/events");

  // Get a reader for the dataset and read all the events
  DatasetReader<GenericRecord> reader = events.newReader();
  try {
    for (GenericRecord event : reader) {
      System.out.println(event);
    }
  } finally {
    reader.close();
  }

  return 0;
}
 
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:19,代码来源:ReadDataset.java


示例18: run

import org.kitesdk.data.Datasets; //导入依赖的package包/类
@Override
public int run(String[] args) throws Exception {
  Dataset<Record> movies = Datasets.load(
      "dataset:hdfs:/tmp/data/movies", Record.class);

  DatasetReader<Record> reader = null;
  try {
    reader = movies.newReader();
    for (Record rec : reader) {
      System.err.println("Movie: " + rec);
    }

  } finally {
    if (reader != null) {
      reader.close();
    }
  }

  return 0;
}
 
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:21,代码来源:ReadMovies.java


示例19: run

import org.kitesdk.data.Datasets; //导入依赖的package包/类
@Override
public int run(List<String> args) throws Exception {

  String inputUri = uri;
  String outputUri = "dataset:hive?dataset=correlated_events";

  if (args.size() == 1) {
    outputUri = args.get(0);
  }

  Preconditions.checkState(Datasets.exists(inputUri),
      "input dataset doesn't exists");

  if (!Datasets.exists(outputUri)) {
    Datasets.create(outputUri, new DatasetDescriptor.Builder()
        .format("avro")
        .schema(CorrelatedEvents.class)
        .build());
  }
  CorrelateEventsTask task = new CorrelateEventsTask(inputUri, outputUri);
  task.run();

  return 0;
}
 
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:25,代码来源:CorrelateEvents.java


示例20: run

import org.kitesdk.data.Datasets; //导入依赖的package包/类
@Override
public int run(List<String> args) throws Exception {

  Preconditions.checkState(!Datasets.exists(uri),
      "events dataset already exists");

  DatasetDescriptor descriptor = new DatasetDescriptor.Builder()
      .schema(StandardEvent.class).build();

  View<StandardEvent> events = Datasets.create(uri, descriptor, StandardEvent.class);
  DatasetWriter<StandardEvent> writer = events.newWriter();
  try {
    while (System.currentTimeMillis() - baseTimestamp < 36000) {
      writer.write(generateRandomEvent());
    }
  } finally {
    writer.close();
  }

  System.out.println("Generated " + counter + " events");

  return 0;
}
 
开发者ID:kite-sdk,项目名称:kite-examples,代码行数:24,代码来源:CreateEvents.java



注:本文中的org.kitesdk.data.Datasets类示例整理自Github/MSDocs等源码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。


鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
上一篇:
Java BlockSand类代码示例发布时间:2022-05-22
下一篇:
Java ReassignLeaseOp类代码示例发布时间:2022-05-22
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap