本地模式下（local），java连接spark调用scala语言打成的jar包

一，环境构建

1、spark版本

2、Scala语言打成的jar，因为Scala和java都是运行在jvm上的字节码类语言，java可以直接调用。，当然该包中包含了一些算法，LR、GBDT、决策树、随机森林，以便于数据的训练。

3、maven webapp项目，pom引入包如下：

<!-- Begin：spark包 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId>
<version>${hadoop.client.version}</version> </dependency> -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.11</artifactId>
<version>${spark.version}</version>
</dependency>

<!-- End:spark包 -->

版本是：<spark.version>2.2.0</spark.version>

3.spark连接工具类：

package cfca.xfraud.afmls.util.spark;

import org.apache.spark.SparkContext;
import org.apache.spark.sql.SparkSession;
import cfca.xfraud.afmls.common.Constants;
import cfca.xfraud.afmls.exception.CodeException;
import cfca.xfraud.dataprocess.DataProcess;
import cfca.xfraud.model.DesicionTree;
import cfca.xfraud.model.GBDTree;
import cfca.xfraud.model.LR;
import cfca.xfraud.model.RFTree;
/**
* @Author zhangzhongqiu
* @Description spark工具类
* @version 1.2.0
* @since jdk1.8
* @date 2018-03-06
*
*
*/
public class SparkUtil {
private static  SparkSession sparkSession;
static{
SparkContext sparkContext = new SparkContext(Constants.SPARK_LOCAL_PATTERN, Constants.APP_NAME);
sparkSession = new SparkSession(sparkContext);
}
/**
* 数据预处理运行状态
* @param appName
* @param dataPattern
* @return
* @throws Exception
* @throws CodeException
*/
public static boolean runDataProcessStatus(int dataPattern,String trainTime) throws Exception {
boolean isSuccess=false;
try {
isSuccess = new DataProcess(sparkSession).runDataProcess(dataPattern,trainTime);
} catch (Exception e) {
throw e;
}
return isSuccess;
}

/**
* 数据训练过程
* @param algorithmPattern
* @param depth
* @param trainTime
* @return
* @throws Exception
*/
public static boolean runTrainData(String algorithmPattern,int depth,String trainTime) throws Exception{
boolean isRunSuccess=false;

try {

if(Constants.STR_ZERO.equals(algorithmPattern)){
isRunSuccess = new LR(sparkSession).lrClassificationModel(20,0.2,trainTime);

}

if(Constants.STR_ONE.equals(algorithmPattern)){
isRunSuccess = new DesicionTree(sparkSession).decisionTreeClassificationModel(20,trainTime);

}

if(Constants.STR_TWO.equals(algorithmPattern)){
isRunSuccess=new GBDTree(sparkSession).GBDTreeClassificationModel(6,20,trainTime);

}

if(Constants.STR_THREE.equals(algorithmPattern)){
isRunSuccess = new RFTree(sparkSession).rfTreeClassificationModel(100,trainTime);
}
} catch (Exception e) {
throw e;
}
return isRunSuccess;
}
}

主要是建立local模式下的sparkSession作为凭证，进行操作，蓝色字体来自于Scala包中的代码，只需要在相应的类中提供函数入口就好了。

原文地址http://www.bieryun.com/2457.html