Flink问题汇总

flink-1.13.1+hadoop-3.2.1关于guava包冲突问题

相关报错

Caused by: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
  at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357) ~[hadoop-common-3.2.1.jar:?]
  at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338) ~[hadoop-common-3.2.1.jar:?]
  at org.apache.hadoop.conf.Configuration.setClass(Configuration.java:2713) ~[hadoop-common-3.2.1.jar:?]
  at org.apache.hadoop.ipc.RPC.setProtocolEngine(RPC.java:205) ~[hadoop-common-3.2.1.jar:?]
  at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.<init>(ApplicationMasterProtocolPBClientImpl.java:58) ~[hadoop-yarn-common-3.2.1.jar:?]
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_241]
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_241]
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_241]
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_241]
  at org.apache.hadoop.yarn.factories.impl.pb.RpcClientFactoryPBImpl.getClient(RpcClientFactoryPBImpl.java:78) ~[hadoop-yarn-common-3.2.1.jar:?]
  ... 40 more

解决方法：

将hadoop 3.2.1中自带的guava-27.0-jre.jar拷贝到flink的lib目录下即可。

Flink SQL过长导致上报自定义metrics失败

问题描述：

解决办法：

sql太长，导致上报的自定义metrics中task_name过长，导致写入表失败，我们临时修改代码，将task_nane设置为固定值，解决了该问题。

flink集成kafka问题

问题描述

Caused by: java.lang.ClassCastException: cannot assign instance of org.apache.commons.collections.map.LinkedMap to field org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.pendingOffsetsToCommit of type org.apache.commons.collections.map.LinkedMap in instance of org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer
at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1405)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2291)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:566)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:552)
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:540)
at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:501)
at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:224)
... 4 more

解决方法：

Kafka库与Flink的反向类加载方法不兼容，修改 conf/flink-conf.yaml 并重启Flink
classloader.resolve-order: parent-first