2.1Hadoop Eclipse Plugin配置及安装

it2025-04-24 10

Hadoop Eclipse 开发工具主要分为

1、根据Hadoop版本生成插件 2、安装Hadoop Eclipse插件 3、配置Hadoop目录 4、配置Hadoop连接 5、新一个MapReduce工程 WordCount.java MapReduce——WordCount问题总结参考：http://blog.sina.com.cn/s/blog_7fcb1aef0100zpux.html 正成功输入出后信息:

14/05/21 23:06:47 INFO input.FileInputFormat: Total input paths to process : 214/05/21 23:06:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable14/05/21 23:06:47 WARN snappy.LoadSnappy: Snappy native library not loaded14/05/21 23:06:48 INFO mapred.JobClient: Running job: job_201405220635_000914/05/21 23:06:49 INFO mapred.JobClient: map 0% reduce 0%14/05/21 23:06:59 INFO mapred.JobClient: map 50% reduce 0%14/05/21 23:07:00 INFO mapred.JobClient: map 100% reduce 0%14/05/21 23:07:09 INFO mapred.JobClient: map 100% reduce 33%14/05/21 23:07:11 INFO mapred.JobClient: map 100% reduce 100%14/05/21 23:07:13 INFO mapred.JobClient: Job complete: job_201405220635_000914/05/21 23:07:13 INFO mapred.JobClient: Counters: 2914/05/21 23:07:13 INFO mapred.JobClient: Job Counters 14/05/21 23:07:13 INFO mapred.JobClient: Launched reduce tasks=114/05/21 23:07:13 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=1738614/05/21 23:07:13 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=014/05/21 23:07:13 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=014/05/21 23:07:13 INFO mapred.JobClient: Launched map tasks=214/05/21 23:07:13 INFO mapred.JobClient: Data-local map tasks=214/05/21 23:07:13 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=1216014/05/21 23:07:13 INFO mapred.JobClient: File Output Format Counters 14/05/21 23:07:13 INFO mapred.JobClient: Bytes Written=1514/05/21 23:07:13 INFO mapred.JobClient: FileSystemCounters14/05/21 23:07:13 INFO mapred.JobClient: FILE_BYTES_READ=5214/05/21 23:07:13 INFO mapred.JobClient: HDFS_BYTES_READ=25214/05/21 23:07:13 INFO mapred.JobClient: FILE_BYTES_WRITTEN=17741914/05/21 23:07:13 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1514/05/21 23:07:13 INFO mapred.JobClient: File Input Format Counters 14/05/21 23:07:13 INFO mapred.JobClient: Bytes Read=2214/05/21 23:07:13 INFO mapred.JobClient: Map-Reduce Framework14/05/21 23:07:13 INFO mapred.JobClient: Map output materialized bytes=5814/05/21 23:07:13 INFO mapred.JobClient: Map input records=214/05/21 23:07:13 INFO mapred.JobClient: Reduce shuffle bytes=5814/05/21 23:07:13 INFO mapred.JobClient: Spilled Records=814/05/21 23:07:13 INFO mapred.JobClient: Map output bytes=3814/05/21 23:07:13 INFO mapred.JobClient: CPU time spent (ms)=561014/05/21 23:07:13 INFO mapred.JobClient: Total committed heap usage (bytes)=33633894414/05/21 23:07:13 INFO mapred.JobClient: Combine input records=414/05/21 23:07:13 INFO mapred.JobClient: SPLIT_RAW_BYTES=23014/05/21 23:07:13 INFO mapred.JobClient: Reduce input records=414/05/21 23:07:13 INFO mapred.JobClient: Reduce input groups=214/05/21 23:07:13 INFO mapred.JobClient: Combine output records=414/05/21 23:07:13 INFO mapred.JobClient: Physical memory (bytes) snapshot=42814668814/05/21 23:07:13 INFO mapred.JobClient: Reduce output records=214/05/21 23:07:13 INFO mapred.JobClient: Virtual memory (bytes) snapshot=214023372814/05/21 23:07:13 INFO mapred.JobClient: Map output records=4

错误解决方法：

14/05/21 23:14:11 INFO mapred.JobClient: Cleaning up the staging area hdfs://192.168.1.53:9000/app/hadoop/hadoop/tmp/mapred/staging/hadoop/.staging/job_201405220635_001014/05/21 23:14:11 ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory newout already existsException in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory newout already exists at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:973) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936) at org.apache.hadoop.mapreduce.Job.submit(Job.java:550) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580) at org.apache.hadoop.examples.WordCount.main(WordCount.java:93)

cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory newout already exists

说明目录已经存在了，需要删除后再执行，命令如下：

hadoop fs -rmr newout

转载于:https://www.cnblogs.com/scwanglijun/p/3742617.html

最新回复(0)