Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
435 views
in Technique[技术] by (71.8m points)

hadoop - Hadoop猪不能mkdir Java抛出IO异常(hadoop pig cannot mkdir java throw IO exception)

I have a very simple script example from hadoop real world solution cookbook and I try it on amazon cloudera clustertogov04 ami and it gives me the java exception of not able to mkdir??

(我有一个来自hadoop现实世界解决方案食谱的非常简单的脚本示例,我在亚马逊cloudera clustertogov04 ami上尝试过,它给了我无法mkdir的Java异常?)

but I have enough disk space??

(但是我有足够的磁盘空间?)

[ec2-user]$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvde1             8255928   3307252   4529300  43% /
tmpfs                  3757068         0   3757068   0% /dev/shm
/dev/xvdk            103212320    192116  97777324   1% /data

heres the script,command,error output

(这是脚本,命令,错误输出)

weblogs = load '/data2/weblogs/weblog_entries.txt' as
(md5:chararray,
url:chararray,
date:chararray,
time:chararray,
ip:chararray);
md5_grp = group weblogs by md5 parallel 4;
store md5_grp into '/data/weblogs/weblogs_md5_groups.bcp';


pig -x local -f pig02 2>err02

2013-06-20 19:57:29,499 [Thread-4] INFO org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 1 segments left of total size: 299132 bytes 2013-06-20 19:57:29,499 [Thread-4] INFO org.apache.hadoop.mapred.LocalJobRunner - 2013-06-20 19:57:29,519 [Thread-4] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local_0001 java.io.IOException: Mkdirs failed to create file:/data/weblogs/weblogs_md5_groups.bcp/_temporary/_attempt_local_0001_r_000000_0 at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:434) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:420) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:805) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:685) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextOutputFormat.getRecordWriter(PigTextOutputFormat.java:98) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.

(2013-06-20 19:57:29,499 [Thread-4] INFO org.apache.hadoop.mapred.Merger- 向下到最后一次合并遍历,总共剩余1个段:299132字节 2013-06-20 19 :57:29,499 [Thread-4]信息org.apache.hadoop.mapred.LocalJobRunner-2013-06-20 19:57:29,519 [Thread-4] WARN org.apache.hadoop.mapred.LocalJobRunner-job_local_0001 java.io .IOException:Mkdirs无法创建文件:/data/weblogs/weblogs_md5_groups.bcp/_temporary/_attempt_local_0001_r_000000_0 at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:434)在org.apache.hadoop.fs.ChecksumFile在org.apache.hadoop.fs.FileSystem.create(FileSystem.java:805)的org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)的.create(ChecksumFileSystem.java:420)。 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextOutputFormat.getRecordWriter(PigTextOutputFormat.java:98)位于org.apache.pig.backend处的apache.hadoop.fs.FileSystem.create(FileSystem.java:685)。 hadoop.executionengine.mapReduceLayer。)

PigOutputFormat.getRecordWriter(PigOutputFormat.java:84) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:582) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:433) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:309) 2013-06-20 19:57:33,176 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_local_0001 has failed!

(org.org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:433)的PigOutputFormat.getRecordWriter(PigOutputFormat.java:84)在org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:582)在org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:433) .apache.hadoop.mapred.LocalJobRunner $ Job.run(LocalJobRunner.java:309)2013-06-20 19:57:33,176 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher-作业job_local_0001失败!)

Stop running all dependent jobs

(停止运行所有依赖作业)
2013-06-20 19:57:33,180 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete

(2013-06-20 19:57:33,180 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher-100%完成)
2013-06-20 19:57:33,182 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!

(2013-06-20 19:57:33,182 [main]错误org.apache.pig.tools.pigstats.PigStatsUtil-1个地图缩小作业失败!)
2013-06-20 19:57:33,182 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode.

(2013-06-20 19:57:33,182 [main]信息org.apache.pig.tools.pigstats.SimplePigStats-检测到本地模式。)

Stats reported below may be incomplete

(以下报告的统计信息可能不完整)
2013-06-20 19:57:33,185 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

(2013-06-20 19:57:33,185 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats-脚本统计信息:)

HadoopVersion PigVersion UserId StartedAt FinishedAt Features

(HadoopVersion PigVersion用户ID StartedAt FinishedAt功能)
2.0.0-cdh4.1.2 0.10.0-cdh4.1.2 ec2-user 2013-06-20 19:57:27 2013-06-20 19:57:33 GROUP_BY

(2.0.0-cdh4.1.2 0.10.0-cdh4.1.2 ec2-user 2013-06-20 19:57:27 2013-06-20 19:57:33 GROUP_BY)

Failed!

(失败了!)

Pig Stack Trace ---------------

(猪栈痕迹---------------)
ERROR 2244: Job failed, hadoop does not return any error message

(错误2244:作业失败,hadoop不返回任何错误消息)

org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job failed, hadoop does not return any error message at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193)

(org.apache.pig.backend.executionengine.ExecException:错误2244:作业失败,Hadoop在org.apache的org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)上未返回任何错误消息.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193))
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)

(在org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)在org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84))
at org.apache.pig.Main.run(Main.java:430)

(在org.apache.pig.Main.run(Main.java:430))
at org.apache.pig.Main.main(Main.java:111) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597)

(在org.apache.pig.Main.main(Main.java:111)在sun.reflect.NativeMethodAccessorImpl.invoke0(在本机方法)在sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)在sun.reflect.DelegatingMethodAccessorImpl .invoke(DelegatingMethodAccessorImpl.java:25)at java.lang.reflect.Method.invoke(Method.java:597))

at org.apache.hadoop.util.RunJar.main(RunJar.java:208) (在org.apache.hadoop.util.RunJar.main(RunJar.java:208))

  ask by Andrew C. Kee translate from so

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

Looks like your Hadoop Job can't create the directory you've specified in your STORE

(看起来Hadoop作业无法创建您在STORE指定的目录)

Have you tried storing the output to a different location such as your home directory?

(您是否尝试过将输出存储到其他位置(例如主目录)?)

Also FYI, Pig won't save its output to a file called "weblogs_md5_groups.bcp", it will actually be creating a directory with that name.

(仅供参考,Pig不会将其输出保存到名为“ weblogs_md5_groups.bcp”的文件中,它实际上将使用该名称创建目录。)


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...