错误日志
ERROR tool.importTool: Encountered IOException running import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://mihaoyu151:9000/user/root/student already exists
[root@mihaoyu151 ~]# sqoop import > --connect jdbc:mysql://mihaoyu151:3306/mybatisdb > --username root > --password root > --table student > --hive-table kb15.student_mysql > --hive-import > --m 1 Warning: /opt/soft/sqoop146/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /opt/soft/sqoop146/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 21/11/23 09:19:56 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 21/11/23 09:19:56 WARN tool.baseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 21/11/23 09:19:56 INFO tool.baseSqoopTool: Using Hive-specific delimiters for output. You can override 21/11/23 09:19:56 INFO tool.baseSqoopTool: delimiters with --fields-terminated-by, etc. 21/11/23 09:19:56 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 21/11/23 09:19:56 INFO tool.CodeGenTool: Beginning code generation 21/11/23 09:19:56 INFO manager.SqlManager: Executing SQL statement: SELECt t.* FROM `student` AS t LIMIT 1 21/11/23 09:19:56 INFO manager.SqlManager: Executing SQL statement: SELECt t.* FROM `student` AS t LIMIT 1 21/11/23 09:19:56 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/soft/hadoop260 Note: /tmp/sqoop-root/compile/03e11c03cfa616424a2d39aed988c741/student.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 21/11/23 09:19:58 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/03e11c03cfa616424a2d39aed988c741/student.jar 21/11/23 09:19:58 WARN manager.MySQLManager: It looks like you are importing from mysql. 21/11/23 09:19:58 WARN manager.MySQLManager: This transfer can be faster! Use the --direct 21/11/23 09:19:58 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path. 21/11/23 09:19:58 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql) 21/11/23 09:19:58 INFO mapreduce.importJobbase: Beginning import of student SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/soft/hadoop260/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/soft/hbase120/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 21/11/23 09:19:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 21/11/23 09:19:58 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 21/11/23 09:19:59 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 21/11/23 09:19:59 INFO client.RMProxy: Connecting to ResourceManager at mihaoyu151/192.168.133.151:8032 21/11/23 09:19:59 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://mihaoyu151:9000/user/root/student already exists 21/11/23 09:19:59 ERROR tool.importTool: Encountered IOException running import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://mihaoyu151:9000/user/root/student already exists at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:270) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:143) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1325) at org.apache.sqoop.mapreduce.importJobbase.doSubmitJob(importJobbase.java:196) at org.apache.sqoop.mapreduce.importJobbase.runJob(importJobbase.java:169) at org.apache.sqoop.mapreduce.importJobbase.runimport(importJobbase.java:266) at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673) at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118) at org.apache.sqoop.tool.importTool.importTable(importTool.java:497) at org.apache.sqoop.tool.importTool.run(importTool.java:605) at org.apache.sqoop.Sqoop.run(Sqoop.java:143) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
解决办法
[root@mihaoyu151 ~]# hdfs dfs -rmr /user/root/student rmr: DEPRECATED: Please use 'rm -r' instead. 21/11/23 09:21:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Deleted /user/root/student [root@mihaoyu151 ~]# sqoop import > --connect jdbc:mysql://mihaoyu151:3306/mybatisdb > --username root > --password root > --table student > --hive-table kb15.student_mysql > --hive-import > --m 1 Warning: /opt/soft/sqoop146/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /opt/soft/sqoop146/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 21/11/23 09:21:24 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 21/11/23 09:21:24 WARN tool.baseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 21/11/23 09:21:24 INFO tool.baseSqoopTool: Using Hive-specific delimiters for output. You can override 21/11/23 09:21:24 INFO tool.baseSqoopTool: delimiters with --fields-terminated-by, etc. 21/11/23 09:21:24 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 21/11/23 09:21:24 INFO tool.CodeGenTool: Beginning code generation 21/11/23 09:21:24 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student` AS t LIMIT 1 21/11/23 09:21:24 INFO manager.SqlManager: Executing SQL statement: SELECt t.* FROM `student` AS t LIMIT 1 21/11/23 09:21:24 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/soft/hadoop260 Note: /tmp/sqoop-root/compile/c2780f73626821864f7ba23e7758fe9d/student.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 21/11/23 09:21:26 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/c2780f73626821864f7ba23e7758fe9d/student.jar 21/11/23 09:21:26 WARN manager.MySQLManager: It looks like you are importing from mysql. 21/11/23 09:21:26 WARN manager.MySQLManager: This transfer can be faster! Use the --direct 21/11/23 09:21:26 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path. 21/11/23 09:21:26 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql) 21/11/23 09:21:26 INFO mapreduce.importJobbase: Beginning import of student SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/soft/hadoop260/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/soft/hbase120/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 21/11/23 09:21:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 21/11/23 09:21:27 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 21/11/23 09:21:27 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 21/11/23 09:21:27 INFO client.RMProxy: Connecting to ResourceManager at mihaoyu151/192.168.133.151:8032 21/11/23 09:21:29 INFO db.DBInputFormat: Using read commited transaction isolation 21/11/23 09:21:29 INFO mapreduce.JobSubmitter: number of splits:1 21/11/23 09:21:29 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1637624153438_0005 21/11/23 09:21:29 INFO impl.YarnClientImpl: Submitted application application_1637624153438_0005 21/11/23 09:21:29 INFO mapreduce.Job: The url to track the job: http://mihaoyu151:8088/proxy/application_1637624153438_0005/ 21/11/23 09:21:29 INFO mapreduce.Job: Running job: job_1637624153438_0005 21/11/23 09:21:37 INFO mapreduce.Job: Job job_1637624153438_0005 running in uber mode : false 21/11/23 09:21:37 INFO mapreduce.Job: map 0% reduce 0% 21/11/23 09:21:43 INFO mapreduce.Job: map 100% reduce 0% 21/11/23 09:21:43 INFO mapreduce.Job: Job job_1637624153438_0005 completed successfully 21/11/23 09:21:43 INFO mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=161559 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=87 HDFS: Number of bytes written=331 HDFS: Number of read operations=4 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Other local map tasks=1 Total time spent by all maps in occupied slots (ms)=2891 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=2891 Total vcore-milliseconds taken by all map tasks=2891 Total megabyte-milliseconds taken by all map tasks=2960384 Map-Reduce framework Map input records=9 Map output records=9 Input split bytes=87 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=40 CPU time spent (ms)=590 Physical memory (bytes) snapshot=117817344 Virtual memory (bytes) snapshot=2731085824 Total committed heap usage (bytes)=60882944 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=331 21/11/23 09:21:43 INFO mapreduce.importJobbase: Transferred 331 bytes in 15.5214 seconds (21.3254 bytes/sec) 21/11/23 09:21:43 INFO mapreduce.importJobbase: Retrieved 9 records. 21/11/23 09:21:43 INFO manager.SqlManager: Executing SQL statement: SELECt t.* FROM `student` AS t LIMIT 1 21/11/23 09:21:43 INFO hive.Hiveimport: Loading uploaded data into Hive 21/11/23 09:21:45 INFO hive.Hiveimport: SLF4J: Class path contains multiple SLF4J bindings. 21/11/23 09:21:45 INFO hive.Hiveimport: SLF4J: Found binding in [jar:file:/opt/soft/hbase120/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] 21/11/23 09:21:45 INFO hive.Hiveimport: SLF4J: Found binding in [jar:file:/opt/soft/hadoop260/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] 21/11/23 09:21:45 INFO hive.Hiveimport: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 21/11/23 09:21:45 INFO hive.Hiveimport: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 21/11/23 09:21:45 INFO hive.Hiveimport: 2021-11-23 09:21:45,188 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 21/11/23 09:21:46 INFO hive.Hiveimport: SLF4J: Class path contains multiple SLF4J bindings. 21/11/23 09:21:46 INFO hive.Hiveimport: SLF4J: Found binding in [jar:file:/opt/soft/hadoop260/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] 21/11/23 09:21:46 INFO hive.Hiveimport: SLF4J: Found binding in [jar:file:/opt/soft/hbase120/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] 21/11/23 09:21:46 INFO hive.Hiveimport: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 21/11/23 09:21:46 INFO hive.Hiveimport: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 21/11/23 09:21:47 INFO hive.Hiveimport: 21/11/23 09:21:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 21/11/23 09:21:47 INFO hive.Hiveimport: 21/11/23 09:21:47 WARN conf.HiveConf: HiveConf of name hive.server2.thrift.client.user does not exist 21/11/23 09:21:47 INFO hive.Hiveimport: 21/11/23 09:21:47 WARN conf.HiveConf: HiveConf of name hive.server2.thrift.client.password does not exist 21/11/23 09:21:47 INFO hive.Hiveimport: 21/11/23 09:21:47 INFO hive.Hiveimport: Logging initialized using configuration in file:/opt/soft/hive110/conf/hive-log4j.properties 21/11/23 09:21:53 INFO hive.Hiveimport: OK 21/11/23 09:21:53 INFO hive.Hiveimport: Time taken: 4.928 seconds 21/11/23 09:21:53 INFO hive.Hiveimport: Loading data to table kb15.student_mysql 21/11/23 09:21:54 INFO hive.Hiveimport: Table kb15.student_mysql stats: [numFiles=1, totalSize=331] 21/11/23 09:21:54 INFO hive.Hiveimport: OK 21/11/23 09:21:54 INFO hive.Hiveimport: Time taken: 0.716 seconds 21/11/23 09:21:54 INFO hive.Hiveimport: Hive import complete. 21/11/23 09:21:54 INFO hive.Hiveimport: Export directory is contains the _SUCCESS file only, removing the directory.
