Sqoop增量导入失败
问题描述:
我有下面的表格:mergetab在MYSQL中。Sqoop增量导入失败
id | name | city | rectime |
---+-----------+-------------+---------------------+
1 | Sidhartha | Hyderabad | 2017-04-18 15:31:22 |
2 | Saketh | Bengaluru | 2017-04-18 15:32:37 |
3 | Sunny | Mumbai | 2017-04-18 15:32:57 |
4 | Bobby | Delhi | 2017-04-18 15:33:15 |
我已经在里面插入一条记录,它看起来如下:
id | name | city | rectime |
---+-----------+-------------+---------------------+
1 | Sidhartha | Hyderabad | 2017-04-18 15:31:22 |
2 | Saketh | Bengaluru | 2017-04-18 15:32:37 |
3 | Sunny | Mumbai | 2017-04-18 15:32:57 |
4 | Bobby | Delhi | 2017-04-18 15:33:15 |
5 | Madhavi | Dharmavaram | 2017-04-18 16:57:09 |
我试着用下面的命令做了sqoop增量导入:
sqoop import --connect jdbc:mysql://127.0.0.1/mydb --table mergetab --username root --password cloudera --hive-import --hive-table mergetab --incremental-append --check-column id --last-value $(hive -S -e "select max(id) from mergetab");
即时得到的以下错误,我无法理解我的sqoop命令有什么问题。 错误:
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Error parsing arguments for import:
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: WARN:
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: The
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: method
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: class
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: org.apache.commons.logging.impl.SLF4JLogFactory#release()
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: was
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: invoked.
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: WARN:
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: Please
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: see
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: http://www.slf4j.org/codes.html#release
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: for
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: an
我在Cloudera VM中练习这个。任何人都可以让我在这里做什么错误?
答
与您的代码行--last-value $(hive -S -e "select max(id) from mergetab")
的问题是,蜂巢总会打印头/即使在静音模式下记录信息,所以最后的值总是会得到价值一样 -
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hive/2.1.0/libexec/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/2.7.3/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
5
你需要或者写一些扭曲shell脚本提取的最后一行,也可以使用直线与showHeader
和outputformat
如下图所示先手直接分配给变量
--last value $(beeline --showHeader=false --outputformat=tsv2 -e "your query")
你只是试图运行'蜂巢-S -e”的值.. .wtf ...“命令在as中地狱,看看它产生了什么样的垃圾? –
我试过了。 FAILED:ParseException行1:0无法识别'hive'附近的输入 - ''S'。 但即使没有'-S',我仍然收到相同的错误信息。 – Sidhartha