本文整理汇总了Java中org.apache.spark.SparkFiles类的典型用法代码示例。如果您正苦于以下问题:Java SparkFiles类的具体用法?Java SparkFiles怎么用?Java SparkFiles使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。
SparkFiles类属于org.apache.spark包,在下文中一共展示了SparkFiles类的5个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Java代码示例。
示例1: loadNativeLib
import org.apache.spark.SparkFiles; //导入依赖的package包/类
void loadNativeLib(){
if(nativeLibPath.contains("/"))
System.load(nativeLibPath);
else
try{
System.load(SparkFiles.get(nativeLibPath));
} catch(Exception e){
System.load(nativeLibPath);
}
}
开发者ID:tudorv91,项目名称:SparkJNI,代码行数:11,代码来源:JniFunction.java
示例2: doMap
import org.apache.spark.SparkFiles; //导入依赖的package包/类
/**
* Imitates map part of hadoop streaming job.
* It executes provided script for every key in inputRecords rdd.
* <br/><br/>
* It is assumed that provided script will read records from standard input (one line for one record)
* and write mapped record into standard output (also one line for one record).
* Mapped record can be a key/value pair. In that case script should return key and value
* splitted by tab (\t) character in single line.
*/
public JavaPairRDD<String, String> doMap(JavaPairRDD<AvroKey<GenericRecord>, NullWritable> inputRecords, String scriptName, String args) {
JavaRDD<String> mappedRecords = inputRecords.keys().pipe("python " + SparkFiles.get(scriptName) + " " + args);
JavaPairRDD<String, String> outputRecords = mappedRecords
.mapToPair(line -> {
String[] splittedPair = line.split("\t");
return new Tuple2<String, String>(splittedPair[0], (splittedPair.length == 1) ? null : splittedPair[1]);
});
return outputRecords;
}
开发者ID:openaire,项目名称:iis,代码行数:22,代码来源:SparkPipeExecutor.java
示例3: doReduce
import org.apache.spark.SparkFiles; //导入依赖的package包/类
/**
* Imitates reduce part of hadoop streaming job.
* <br/><br/>
* It is assumed that provided script will read records from standard input (one line for one record)
* and group records with the same key into single record (reduce).
* Method assures that all input records with the same key will be transfered in adjacent lines.
* Reduced records should be written by script into standard output (one line for one record).
* Reduced records must be json strings of class provided as argument.
*/
public JavaPairRDD<AvroKey<GenericRecord>, NullWritable> doReduce(JavaPairRDD<String, String> inputRecords, String scriptName, String args, Class<? extends GenericRecord> outputClass) {
JavaRDD<String> reducedRecords = inputRecords.sortByKey()
.map(record -> record._1 + ((record._2 == null) ? "" : ("\t" + record._2)))
.pipe("python " + SparkFiles.get(scriptName) + " " + args);
JavaPairRDD<AvroKey<GenericRecord>, NullWritable> outputRecords = reducedRecords
.map(recordString -> AvroGsonFactory.create().fromJson(recordString, outputClass))
.mapToPair(record -> new Tuple2<AvroKey<GenericRecord>, NullWritable>(new AvroKey<>(record), NullWritable.get()));
return outputRecords;
}
开发者ID:openaire,项目名称:iis,代码行数:22,代码来源:SparkPipeExecutor.java
示例4: align
import org.apache.spark.SparkFiles; //导入依赖的package包/类
/**
* Performs read alignment on a RDD.
* @param unalignedReads the reads to align.
* @param pairedAlignment whether it should perform pair-end alignment ({@code true}) or single-end alignment ({@code false}).
* @return never {@code null}.
*/
public JavaRDD<GATKRead> align(final JavaRDD<GATKRead> unalignedReads, final boolean pairedAlignment) {
final Broadcast<SAMFileHeader> broadcastHeader = this.broadcastHeader;
final String indexFileName = this.indexFileName;
final boolean resolveIndexFileName = this.resolveIndexFileName;
return unalignedReads.mapPartitions(itr ->
new ReadAligner(resolveIndexFileName ? SparkFiles.get(indexFileName) : indexFileName, broadcastHeader.value(), pairedAlignment).apply(itr));
}
开发者ID:broadinstitute,项目名称:gatk,代码行数:14,代码来源:BwaSparkEngine.java
示例5: getScriptPath
import org.apache.spark.SparkFiles; //导入依赖的package包/类
private static String getScriptPath() {
String path = SparkFiles.get("scripts");
if (SystemUtils.IS_OS_WINDOWS) {
return path.replace("\\", "/");
}
return path;
}
开发者ID:openaire,项目名称:iis,代码行数:11,代码来源:DocumentClassificationJob.java
注:本文中的org.apache.spark.SparkFiles类示例整理自Github/MSDocs等源码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。 |
请发表评论