最近在写一个hadoop的job,其中依赖了guava,我需要的版本是27.2-jre
然而,不幸的是,Hadoop集群上是2.7.2,其中也有guava版本,11.0.2
更坑的是,guava在16之后有break change,无法向下兼容。
导致不管怎么运行,都会出现:
Error: com.google.common.hash.Funnels.stringFunnel(Ljava/nio/charset/Charset;)Lcom/google/common/hash/Funnel; No Such Method
Error: com.google.common.hash.Funnels.stringFunnel(Ljava/nio/charset/Charset;)Lcom/google/common/hash/Funnel; No Such Method
Error: com.google.common.hash.Funnels.stringFunnel(Ljava/nio/charset/Charset;)Lcom/google/common/hash/Funnel; No Such Method
所以这种是无法通过libjar方式解决的。
除了升级集群这种伤筋动骨的方案外(运维应该不会答应的),比较Hack的方式,就是把依赖的包的package进行重命名,从而绕过冲突。
1 使用jarjar
这是Google给的工具,貌似已经无法使用了
2 使用Maven Shade插件
是的,又是这个神奇的插件,可以做fat jar,做各种奇怪的事情,也可以做 包名替换,非常简单:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.4</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<excludes>
<exclude>org.apache.hadoop</exclude>
</excludes>
</artifactSet>
<transformers>
.....
</transformers>
<relocations>
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>com.hack.common</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.4</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<excludes>
<exclude>org.apache.hadoop</exclude>
</excludes>
</artifactSet>
<transformers>
.....
</transformers>
<relocations>
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>com.hack.common</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.2.4</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <artifactSet> <excludes> <exclude>org.apache.hadoop</exclude> </excludes> </artifactSet> <transformers> ..... </transformers> <relocations> <relocation> <pattern>com.google.common</pattern> <shadedPattern>com.hack.common</shadedPattern> </relocation> </relocations> </configuration> </execution> </executions> </plugin> </plugins> </build>
就是上面这样,使用relocations规则,即可搞定,不需要改动任何代码,打好的包中,会自动替换包名(包括你的代码和 你 依赖的包)。