Java VectorUDT类代码示例

OGeek|极客世界-中国程序员成长平台 › 门户 › 编程› Java›Java编程经验

原作者: [db:作者] 来自: [db:来源] 收藏邀请

本文整理汇总了Java中org.apache.spark.mllib.linalg.VectorUDT类的典型用法代码示例。如果您正苦于以下问题：Java VectorUDT类的具体用法？Java VectorUDT怎么用？Java VectorUDT使用的例子？那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。

VectorUDT类属于org.apache.spark.mllib.linalg包，在下文中一共展示了VectorUDT类的12个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于我们的系统推荐出更棒的Java代码示例。

示例1: start

import org.apache.spark.mllib.linalg.VectorUDT; //导入依赖的package包/类
private void start() {
	SparkSession spark = SparkSession.builder().appName("First Prediction").master("local").getOrCreate();

	StructType schema = new StructType(
			new StructField[] { new StructField("label", DataTypes.DoubleType, false, Metadata.empty()),
					new StructField("features", new VectorUDT(), false, Metadata.empty()), });

	// TODO this example is not working yet
}

开发者ID:jgperrin，项目名称:net.jgp.labs.spark，代码行数:10，代码来源:FirstPrediction.java

示例2: exportToJson

import org.apache.spark.mllib.linalg.VectorUDT; //导入依赖的package包/类
public static String exportToJson(Set<String> columns, StructType dfSchema) {
    //This would contain column name along with type of a dataframe

    List<Field> schema = new ArrayList<>();

    for (String column : columns) {
        StructField field = dfSchema.fields()[ dfSchema.fieldIndex(column) ];

        if (field.dataType() instanceof StringType) {
            schema.add(new Field(field.name(), STRING));
        } else if (field.dataType() instanceof BooleanType) {
            schema.add(new Field(field.name(), BOOLEAN));
        } else if (field.dataType() instanceof VectorUDT) {
            schema.add(new Field(field.name(), DOUBLE_ARRAY));
        } else if (field.dataType() instanceof DoubleType || field.dataType() instanceof DecimalType || field.dataType() instanceof FloatType ||
                field.dataType() instanceof IntegerType || field.dataType() instanceof LongType || field.dataType() instanceof ShortType) {
            schema.add(new Field(field.name(), DOUBLE));
        } else if (field.dataType() instanceof ArrayType) {
            if(((ArrayType)field.dataType()).elementType() instanceof StringType) {
                schema.add(new Field(field.name(), STRING_ARRAY));
            }else if(((ArrayType)field.dataType()).elementType() instanceof DoubleType) {
                schema.add(new Field(field.name(), DOUBLE_ARRAY));
            }else {
                throw new UnsupportedOperationException("Cannot support data of type " + field.dataType());
            }
        }
        else {
            throw new UnsupportedOperationException("Cannot support data of type " + field.dataType());
        }
    }
    return gson.toJson(schema);
}

开发者ID:flipkart-incubator，项目名称:spark-transformers，代码行数:33，代码来源:SchemaExporter.java

示例3: exportSchemaToJson

import org.apache.spark.mllib.linalg.VectorUDT; //导入依赖的package包/类
public static String exportSchemaToJson(StructType dfSchema) {
    //This would contain column name along with type of a dataframe

    List<Field> schema = new ArrayList<>();

    for (StructField field : dfSchema.fields()) {
        if (field.dataType() instanceof StringType) {
            schema.add(new Field(field.name(), STRING));
        } else if (field.dataType() instanceof BooleanType) {
            schema.add(new Field(field.name(), BOOLEAN));
        } else if (field.dataType() instanceof VectorUDT) {
            schema.add(new Field(field.name(), DOUBLE_ARRAY));
        } else if (field.dataType() instanceof DoubleType || field.dataType() instanceof DecimalType || field.dataType() instanceof FloatType ||
                field.dataType() instanceof IntegerType || field.dataType() instanceof LongType || field.dataType() instanceof ShortType) {
            schema.add(new Field(field.name(), DOUBLE));
        } else if (field.dataType() instanceof ArrayType) {
            if(((ArrayType)field.dataType()).elementType() instanceof StringType) {
                schema.add(new Field(field.name(), STRING_ARRAY));
            }else if(((ArrayType)field.dataType()).elementType() instanceof DoubleType) {
                schema.add(new Field(field.name(), DOUBLE_ARRAY));
            }else {
                throw new UnsupportedOperationException("Cannot support data of type " + field.dataType());
            }
        }
        else {
            throw new UnsupportedOperationException("Cannot support data of type " + field.dataType());
        }
    }
    return gson.toJson(schema);
}

开发者ID:flipkart-incubator，项目名称:spark-transformers，代码行数:31，代码来源:SchemaExporter.java

示例4: testVectorBinarizerDense

import org.apache.spark.mllib.linalg.VectorUDT; //导入依赖的package包/类
@Test
public void testVectorBinarizerDense() {
    // prepare data

    JavaRDD<Row> jrdd = sc.parallelize(Arrays.asList(
            RowFactory.create(0d, 1d, new DenseVector(new double[]{-2d, -3d, -4d, -1d, 6d, -7d, 8d, 0d, 0d, 0d, 0d, 0d})),
            RowFactory.create(1d, 2d, new DenseVector(new double[]{4d, -5d, 6d, 7d, -8d, 9d, -10d, 0d, 0d, 0d, 0d, 0d})),
            RowFactory.create(2d, 3d, new DenseVector(new double[]{-5d, 6d, -8d, 9d, 10d, 11d, 12d, 0d, 0d, 0d, 0d, 0d}))
    ));

    StructType schema = new StructType(new StructField[]{
            new StructField("id", DataTypes.DoubleType, false, Metadata.empty()),
            new StructField("value1", DataTypes.DoubleType, false, Metadata.empty()),
            new StructField("vector1", new VectorUDT(), false, Metadata.empty())
    });

    DataFrame df = sqlContext.createDataFrame(jrdd, schema);
    VectorBinarizer vectorBinarizer = new VectorBinarizer()
            .setInputCol("vector1")
            .setOutputCol("binarized")
            .setThreshold(2d);


    //Export this model
    byte[] exportedModel = ModelExporter.export(vectorBinarizer, df);

    //Import and get Transformer
    Transformer transformer = ModelImporter.importAndGetTransformer(exportedModel);
    //compare predictions
    Row[] sparkOutput = vectorBinarizer.transform(df).orderBy("id").select("id", "value1", "vector1", "binarized").collect();
    for (Row row : sparkOutput) {

        Map<String, Object> data = new HashMap<>();
        data.put(vectorBinarizer.getInputCol(), ((DenseVector) row.get(2)).toArray());
        transformer.transform(data);
        double[] output = (double[]) data.get(vectorBinarizer.getOutputCol());
        assertArrayEquals(output, ((DenseVector) row.get(3)).toArray(), 0d);
    }
}

开发者ID:flipkart-incubator，项目名称:spark-transformers，代码行数:40，代码来源:VectorBinarizerBridgeTest.java

示例5: testSchema1

import org.apache.spark.mllib.linalg.VectorUDT; //导入依赖的package包/类
/**
 * Output :
 [{"name":"id","datatype":"double"},{"name":"label","datatype":"double"},{"name":"features","datatype":"double []"}]
 * */
@Test
public void testSchema1() {
    StructType schema = new StructType(new StructField[]{
            new StructField("id", DataTypes.DoubleType, false, Metadata.empty()),
            new StructField("label", DataTypes.DoubleType, false, Metadata.empty()),
            new StructField("features", new VectorUDT(), false, Metadata.empty())
    });
    System.out.println(SchemaExporter.exportSchemaToJson(schema));
}

开发者ID:flipkart-incubator，项目名称:spark-transformers，代码行数:14，代码来源:SchemaExporterTest.java

示例6: testSchema3

import org.apache.spark.mllib.linalg.VectorUDT; //导入依赖的package包/类
/**
 * Output :
 [{"name":"id","datatype":"double"},{"name":"value1","datatype":"double"},{"name":"vector1","datatype":"double []"}]
 * */
@Test
public void testSchema3() {
    StructType schema = new StructType(new StructField[]{
            new StructField("id", DataTypes.DoubleType, false, Metadata.empty()),
            new StructField("value1", DataTypes.DoubleType, false, Metadata.empty()),
            new StructField("vector1", new VectorUDT(), false, Metadata.empty())
    });
    System.out.println(SchemaExporter.exportSchemaToJson(schema));
}

开发者ID:flipkart-incubator，项目名称:spark-transformers，代码行数:14，代码来源:SchemaExporterTest.java

示例7: testColumnExport1

import org.apache.spark.mllib.linalg.VectorUDT; //导入依赖的package包/类
/**
 * Output :
 [{"name":"features","datatype":"double []"},{"name":"id","datatype":"double"}]
 * */
@Test
public void testColumnExport1() {
    StructType schema = new StructType(new StructField[]{
            new StructField("id", DataTypes.DoubleType, false, Metadata.empty()),
            new StructField("label", DataTypes.DoubleType, false, Metadata.empty()),
            new StructField("features", new VectorUDT(), false, Metadata.empty())
    });
    System.out.println(SchemaExporter.exportToJson(new HashSet<String>(Arrays.asList("id", "features")),schema));
}

开发者ID:flipkart-incubator，项目名称:spark-transformers，代码行数:14，代码来源:SchemaExporterTest.java

示例8: testColumnExport3

import org.apache.spark.mllib.linalg.VectorUDT; //导入依赖的package包/类
/**
 * Output :
 [{"name":"id","datatype":"double"},{"name":"vector1","datatype":"double []"}]
 * */
@Test
public void testColumnExport3() {
    StructType schema = new StructType(new StructField[]{
            new StructField("id", DataTypes.DoubleType, false, Metadata.empty()),
            new StructField("value1", DataTypes.DoubleType, false, Metadata.empty()),
            new StructField("vector1", new VectorUDT(), false, Metadata.empty())
    });
    System.out.println(SchemaExporter.exportToJson(new HashSet<String>(Arrays.asList("id", "vector1")),schema));
}

开发者ID:flipkart-incubator，项目名称:spark-transformers，代码行数:14，代码来源:SchemaExporterTest.java

示例9: init

import org.apache.spark.mllib.linalg.VectorUDT; //导入依赖的package包/类
@Override
public StructField init(Evaluator evaluator){
	return DataTypes.createStructField(getColumnName(), new VectorUDT(), false);
}

开发者ID:jeremyore，项目名称:spark-pmml-import，代码行数:5，代码来源:ProbabilityColumnProducer.java

示例10: testVectorBinarizerSparse

import org.apache.spark.mllib.linalg.VectorUDT; //导入依赖的package包/类
@Test
public void testVectorBinarizerSparse() {
    // prepare data

    int[] sparseArray1 = {5, 6, 11, 4, 7, 9, 8, 14, 13};
    double[] sparseArray1Values = {-5d, 7d, 1d, -2d, -4d, -1d, 31d, -1d, -3d};

    int[] sparseArray2 = {2, 6, 1};
    double[] sparseArray2Values = {1d, 11d, 2d};

    int[] sparseArray3 = {4, 6, 1};
    double[] sparseArray3Values = {52d, 71d, 11d};

    int[] sparseArray4 = {4, 1, 2};
    double[] sparseArray4Values = {17d, 7d, 9d};

    JavaRDD<Row> jrdd = sc.parallelize(Arrays.asList(
            RowFactory.create(3d, 4d, new SparseVector(20, sparseArray1, sparseArray1Values)),
            RowFactory.create(4d, 5d, new SparseVector(20, sparseArray2, sparseArray2Values)),
            RowFactory.create(5d, 5d, new SparseVector(20, sparseArray3, sparseArray3Values)),
            RowFactory.create(6d, 5d, new SparseVector(20, sparseArray4, sparseArray4Values))
    ));

    StructType schema = new StructType(new StructField[]{
            new StructField("id", DataTypes.DoubleType, false, Metadata.empty()),
            new StructField("value1", DataTypes.DoubleType, false, Metadata.empty()),
            new StructField("vector1", new VectorUDT(), false, Metadata.empty())
    });

    DataFrame df = sqlContext.createDataFrame(jrdd, schema);
    VectorBinarizer vectorBinarizer = new VectorBinarizer()
            .setInputCol("vector1")
            .setOutputCol("binarized");


    //Export this model
    byte[] exportedModel = ModelExporter.export(vectorBinarizer, null);

    //Import and get Transformer
    Transformer transformer = ModelImporter.importAndGetTransformer(exportedModel);
    //compare predictions
    Row[] sparkOutput = vectorBinarizer.transform(df).orderBy("id").select("id", "value1", "vector1", "binarized").collect();
    for (Row row : sparkOutput) {

        Map<String, Object> data = new HashMap<>();
        data.put(vectorBinarizer.getInputCol(), ((SparseVector) row.get(2)).toArray());
        transformer.transform(data);
        double[] output = (double[]) data.get(vectorBinarizer.getOutputCol());
        assertArrayEquals(output, ((SparseVector)row.get(3)).toArray(), 0d);
    }
}

开发者ID:flipkart-incubator，项目名称:spark-transformers，代码行数:52，代码来源:VectorBinarizerBridgeTest.java

示例11: testVectorAssembler

import org.apache.spark.mllib.linalg.VectorUDT; //导入依赖的package包/类
@Test
public void testVectorAssembler() {
    // prepare data

    JavaRDD<Row> jrdd = sc.parallelize(Arrays.asList(
            RowFactory.create(0d, 1d, new DenseVector(new double[]{2d, 3d})),
            RowFactory.create(1d, 2d, new DenseVector(new double[]{3d, 4d})),
            RowFactory.create(2d, 3d, new DenseVector(new double[]{4d, 5d})),
            RowFactory.create(3d, 4d, new DenseVector(new double[]{5d, 6d})),
            RowFactory.create(4d, 5d, new DenseVector(new double[]{6d, 7d}))
    ));

    StructType schema = new StructType(new StructField[]{
            new StructField("id", DataTypes.DoubleType, false, Metadata.empty()),
            new StructField("value1", DataTypes.DoubleType, false, Metadata.empty()),
            new StructField("vector1", new VectorUDT(), false, Metadata.empty())
    });

    DataFrame df = sqlContext.createDataFrame(jrdd, schema);
    VectorAssembler vectorAssembler = new VectorAssembler()
            .setInputCols(new String[]{"value1", "vector1"})
            .setOutputCol("feature");


    //Export this model
    byte[] exportedModel = ModelExporter.export(vectorAssembler, null);

    String exportedModelJson = new String(exportedModel);
    //Import and get Transformer
    Transformer transformer = ModelImporter.importAndGetTransformer(exportedModel);
    //compare predictions
    Row[] sparkOutput = vectorAssembler.transform(df).orderBy("id").select("id", "value1", "vector1", "feature").collect();
    for (Row row : sparkOutput) {

        Map<String, Object> data = new HashMap<>();
        data.put(vectorAssembler.getInputCols()[0], row.get(1));
        data.put(vectorAssembler.getInputCols()[1], ((DenseVector) row.get(2)).toArray());
        transformer.transform(data);
        double[] output = (double[]) data.get(vectorAssembler.getOutputCol());
        assertArrayEquals(output, ((DenseVector) row.get(3)).toArray(), 0d);
    }
}

开发者ID:flipkart-incubator，项目名称:spark-transformers，代码行数:43，代码来源:VectorAssemblerBridgeTest.java

示例12: testChiSqSelector

import org.apache.spark.mllib.linalg.VectorUDT; //导入依赖的package包/类
@Test
public void testChiSqSelector() {
    // prepare data

    JavaRDD<Row> jrdd = sc.parallelize(Arrays.asList(
            RowFactory.create(0d, 0d, new DenseVector(new double[]{8d, 7d, 0d})),
            RowFactory.create(1d, 1d, new DenseVector(new double[]{0d, 9d, 6d})),
            RowFactory.create(2d, 1d, new DenseVector(new double[]{0.0d, 9.0d, 8.0d})),
            RowFactory.create(3d, 2d, new DenseVector(new double[]{8.0d, 9.0d, 5.0d}))
    ));

    double[] preFilteredData = {0.0d, 6.0d, 8.0d, 5.0d};

    StructType schema = new StructType(new StructField[]{
            new StructField("id", DataTypes.DoubleType, false, Metadata.empty()),
            new StructField("label", DataTypes.DoubleType, false, Metadata.empty()),
            new StructField("features", new VectorUDT(), false, Metadata.empty())
    });

    DataFrame df = sqlContext.createDataFrame(jrdd, schema);
    ChiSqSelector chiSqSelector = new ChiSqSelector();
    chiSqSelector.setNumTopFeatures(1);
    chiSqSelector.setFeaturesCol("features");
    chiSqSelector.setLabelCol("label");
    chiSqSelector.setOutputCol("output");

    ChiSqSelectorModel chiSqSelectorModel = chiSqSelector.fit(df);

    //Export this model
    byte[] exportedModel = ModelExporter.export(chiSqSelectorModel, null);

    String exportedModelJson = new String(exportedModel);

    //Import and get Transformer
    Transformer transformer = ModelImporter.importAndGetTransformer(exportedModel);

    //compare predictions
    Row[] sparkOutput = chiSqSelectorModel.transform(df).orderBy("id").select("id", "label", "features", "output").collect();
    for (Row row : sparkOutput) {
        Map<String, Object> data = new HashMap<>();
        data.put(chiSqSelectorModel.getFeaturesCol(), ((DenseVector) row.get(2)).toArray());
        transformer.transform(data);
        double[] output = (double[]) data.get(chiSqSelectorModel.getOutputCol());
        System.out.println(Arrays.toString(output));
        assertArrayEquals(output, ((DenseVector) row.get(3)).toArray(), 0d);
    }
}

开发者ID:flipkart-incubator，项目名称:spark-transformers，代码行数:48，代码来源:ChiSqSelectorBridgeTest.java

注：本文中的org.apache.spark.mllib.linalg.VectorUDT类示例整理自Github/MSDocs等源码及文档管理平台，相关代码片段筛选自各路编程大神贡献的开源项目，源码版权归原作者所有，传播和使用请参考对应项目的License；未经允许，请勿转载。

鲜花

握手

雷人

路过

鸡蛋

该文章已有0人参与评论

请发表评论

全部评论

专题导读

More+

10-27 六六分期app的软件客服如何联系？(六六分期

11-06 可心卡盟:win10系统火狐flash插件崩溃怎么

11-06 亲亲特价:怎么删除回收站图标

11-06 济南大学虚拟社区:鲁大师节能降温的具体办

11-06 xlueops.exe:无线网络安装向导

11-06 女斗合众国:win7系统cf与主机连接不稳定怎

11-06 0xc000022-[cf烟雾头]cf怎么调烟雾头

11-06 qizideyouhuo:应用程序无法正常启动0xc0000

11-06 ipz-185:win7系统vcf文件怎么打开

11-06 傻哥蹦迪:win10系统s4怎么打开usb调试

11-06 八神浩树gtaste:回收站清空了怎么恢复

11-06 妖尾之黑色守护:win10系统电脑没有1440x900

11-06 校园至尊魔王小说:win7系统浏览网页时字体

11-06 女斗合众国:win10系统访问共享文件夹提示请

11-06 tokyo hot n0654:恢复win7系统默认字体一招

11-06 雨酷仙境:设置win7系统转移临时文件夹腾出

11-06 阿穆纳伊之杖:win7系统开始菜单在右边还原

11-06 tunespotting:win10系统火狐flash插件总是

11-06 甘尔葛分析师：计谋网站seo关键词暴涨有什

11-06 蔡贵霖: 计谋网站seo关键词暴涨有什么秘密

11-06 博益网首页:ao3网页版进入不了解决方法

11-06 漏斗子专栏: 网站数据分析小白易懂精华篇

11-06 见证双虹怎么做:win7系统开启telnet命令的

11-06 颾狐蝶蜋:系统资源不足无法完成请求的服务

11-06 国光中学校歌:提交网站到alexa查询详细步骤

11-06 西安有情天:静态网页和动态网页的区别

11-06 红木雅尚斋:外部链接构造对网站的好处

11-06 前官礼遇：防止域名劫持–增强域安全性的10

11-06 密传二转答案: 中文分词算法有哪些

11-06 金泉家园邮编:百度快照劫持的表现及应对方

Java Set类代码示例发布时间：2022-05-22

Java CommonInvalidationConstants2类代码示例发布时间：2022-05-22

剪的笔顺,诠释剪的笔画,认识剪的部首

1 六六分期app的软件客服如何联系？(六六分期

六六分期app的软件客服如何联系？不知道吗？加qq群【895510560】即可！标题：六六分期

阅读：18210|2023-10-27

2 可心卡盟:win10系统火狐flash插件崩溃怎么

今天小编告诉大家如何处理win10系统火狐flash插件总是崩溃的问题，可能很多用户都不知

阅读：9656|2022-11-06

3 亲亲特价:怎么删除回收站图标

今天小编告诉大家如何对win10系统删除桌面回收站图标进行设置，可能很多用户都不知道

阅读：8168|2022-11-06

4 济南大学虚拟社区:鲁大师节能降温的具体办

今天小编告诉大家如何对win10系统电脑设置节能降温的设置方法，想必大家都遇到过需要

阅读：8543|2022-11-06

5 xlueops.exe:无线网络安装向导

我们在使用xp系统的过程中,经常需要对xp系统无线网络安装向导设置进行设置，可能很多

阅读：8449|2022-11-06

6 女斗合众国:win7系统cf与主机连接不稳定怎

今天小编告诉大家如何处理win7系统玩cf老是与主机连接不稳定的问题，可能很多用户都不

阅读：9375|2022-11-06

7 0xc000022-[cf烟雾头]cf怎么调烟雾头

电脑对日常生活的重要性小编就不多说了，可是一旦碰到win7系统设置cf烟雾头的问题，很

阅读：8418|2022-11-06

8 qizideyouhuo:应用程序无法正常启动0xc0000

我们在日常使用电脑的时候，有的小伙伴们可能在打开应用的时候会遇见提示应用程序无法

阅读：7855|2022-11-06

9 ipz-185:win7系统vcf文件怎么打开

今天小编告诉大家如何对win7系统打开vcf文件进行设置，可能很多用户都不知道怎么对win

阅读：8403|2022-11-06

10 傻哥蹦迪:win10系统s4怎么打开usb调试

今天小编告诉大家如何对win10系统s4开启USB调试模式进行设置，可能很多用户都不知道怎

阅读：7391|2022-11-06

客服电话

电子邮件

Java VectorUDT类代码示例

示例1: start

示例2: exportToJson

示例3: exportSchemaToJson

示例4: testVectorBinarizerDense

示例5: testSchema1

示例6: testSchema3

示例7: testColumnExport1

示例8: testColumnExport3

示例9: init

示例10: testVectorBinarizerSparse

示例11: testVectorAssembler

示例12: testChiSqSelector

请发表评论

全部评论

上一篇：

下一篇：

librespeed/speedtest: Self-hosted Speedt

transitive-bullshit/react-modern-library

avehtari/BDA_m_demos: Bayesian Data Anal

四维彩超怎么看性别？四维看男孩女孩诀窍

medfreeman/markdown-it-toc-and-anchor: m

剪的笔顺,诠释剪的笔画,认识剪的部首

六六分期app的软件客服如何联系？(六六分期

florent37/ViewAnimator: A fluent Android

florent37/Shrine-MaterialDesign2: implem

CVE-2020-36276

SimpleSoftwareIO/simple-sms: Send and re

关于我们

产品与服务

解决方案

139-2527-9053