Hi, there:
The "FLIP-108: Add GPU support in Flink"[1] is now working in
progress. However, we met a problem with
"RuntimeContext#getExternalResourceInfos" if we want to leverage the
Plugin[2] mechanism in Flink.
The interface is:
The problem is now:
public interface RuntimeContext {
/**
* Get the specific external resource information by the resourceName.
*/
<T extends ExternalResourceInfo> Set<T>
getExternalResourceInfos(String resourceName, Class<T>
externalResourceType);
}
The problem is that the mainClassLoader does not recognize the
subclasses of ExternalResourceInfo. Those ExternalResourceInfo is
located in ExternalResourceDriver jar and has been isolated from
mainClassLoader by PluginManager. So, ClassNotFoundExeption will be
thrown out.
The solution could be:
- Not leveraging the plugin mechanism. Just load drivers to
mainClassLoader. The drawback is that user needs to handle the
dependency conflict.
- Force user to build two separate jars. One for the
ExternalResourceDriver, the other for the ExternalResourceInfo. The
jar including ExternalResourceInfo class should be added to “/lib”
dir. This approach probably makes sense but might annoy user.
- Change the RuntimeContext#getExternalResourceInfos, let it return
ExternalResourceInfo and add something like “Properties getInfo()” to
ExternalResourceInfo interface. The contract for resolving the return
value would be specified by the driver provider and user. The Flink
core does not need to be aware of the concrete implementation:
public interface RuntimeContext {
/**
* Get the specific external resource information by the resourceName.
*/
Set<ExternalResourceInfo> getExternalResourceInfos(String resourceName);
}
public interface ExternalResourceInfo {
Properties getInfo();
}
>From my side, I prefer the third approach:
- Regarding usability, it frees user from handling dependency or
packaging two jars.
- It decouples the Flink's mainClassLoader from the concrete
implementation of ExternalResourceInfo.
Looking forward to your feedback.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-108%3A+Add+GPU+support+in+Flink
[2] https://ci.apache.org/projects/flink/flink-docs-master/ops/plugins.html
Best,
Yangze Guo
