首页 文章 精选 留言 我的

精选列表

搜索[install],共10017篇文章
优秀的个人博客,低调大师

Mahout 0.12.2 Install

Mahout 是一个很强大的数据挖掘工具,是一个分布式机器学习算法的集合,包括被称为Taste的分布式协同过滤的实现、分类、聚类等。Mahout最大的优点就是基于hadoop实现,把很多以前运行于单机上的算法,转化为了MapReduce模式,这样大大提升了算法可处理的数据量和处理性能。 http://www.ha97.com/5803.html 1.下载解压[root@sht-sgmhadoopnn-01 hadoop]# wget http://apache.fayea.com/mahout/0.12.2/apache-mahout-distribution-0.12.2.tar.gz[root@sht-sgmhadoopnn-01 hadoop]# tar -xzvf apache-mahout-distribution-0.12.2.tar.gz [root@sht-sgmhadoopnn-01 hadoop]# ln -s /hadoop/apache-mahout-distribution-0.12.2 mahout 2.配置环境变量[root@sht-sgmhadoopnn-01 hadoop]# vi /etc/profile................export MAHOUT_HOME=/hadoop/mahoutexport MAHOUT_CONF_DIR=$MAHOUT_HOME/conf export PATH=$MAHOUT_HOME/bin:$PATH[root@sht-sgmhadoopnn-01 hadoop]# source /etc/profile 3.运行mahout 测试[root@sht-sgmhadoopnn-01 ~]# mahoutMAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.Running on hadoop, using /hadoop/hadoop-2.7.2/bin/hadoop and HADOOP_CONF_DIR=/hadoop/hadoop-2.7.2/etc/hadoopMAHOUT-JOB: /hadoop/mahout/mahout-examples-0.12.2-job.jarAn example program must be given as the first argument.Valid program names are: arff.vector: : Generate Vectors from an ARFF file or directory baumwelch: : Baum-Welch algorithm for unsupervised HMM training canopy: : Canopy clustering cat: : Print a file or resource as the logistic regression models would see it cleansvd: : Cleanup and verification of SVD output clusterdump: : Dump cluster output to text clusterpp: : Groups Clustering Output In Clusters cmdump: : Dump confusion matrix in HTML or text formats cvb: : LDA via Collapsed Variation Bayes (0th deriv. approx) cvb0_local: : LDA via Collapsed Variation Bayes, in memory locally. describe: : Describe the fields and target variable in a data set evaluateFactorization: : compute RMSE and MAE of a rating matrix factorization against probes fkmeans: : Fuzzy K-means clustering hmmpredict: : Generate random sequence of observations by given HMM itemsimilarity: : Compute the item-item-similarities for item-based collaborative filtering kmeans: : K-means clustering lucene.vector: : Generate Vectors from a Lucene index matrixdump: : Dump matrix in CSV format matrixmult: : Take the product of two matrices parallelALS: : ALS-WR factorization of a rating matrix qualcluster: : Runs clustering experiments and summarizes results in a CSV recommendfactorized: : Compute recommendations using the factorization of a rating matrix recommenditembased: : Compute recommendations using item-based collaborative filtering regexconverter: : Convert text files on a per line basis based on regular expressions resplit: : Splits a set of SequenceFiles into a number of equal splits rowid: : Map SequenceFile<Text,VectorWritable> to {SequenceFile<IntWritable,VectorWritable>, SequenceFile<IntWritable,Text>} rowsimilarity: : Compute the pairwise similarities of the rows of a matrix runAdaptiveLogistic: : Score new production data using a probably trained and validated AdaptivelogisticRegression model runlogistic: : Run a logistic regression model against CSV data seq2encoded: : Encoded Sparse Vector generation from Text sequence files seq2sparse: : Sparse Vector generation from Text sequence files seqdirectory: : Generate sequence files (of Text) from a directory seqdumper: : Generic Sequence File dumper seqmailarchives: : Creates SequenceFile from a directory containing gzipped mail archives seqwiki: : Wikipedia xml dump to sequence file spectralkmeans: : Spectral k-means clustering split: : Split Input data into test and train sets splitDataset: : split a rating dataset into training and probe parts ssvd: : Stochastic SVD streamingkmeans: : Streaming k-means clustering svd: : Lanczos Singular Value Decomposition testnb: : Test the Vector-based Bayes classifier trainAdaptiveLogistic: : Train an AdaptivelogisticRegression model trainlogistic: : Train a logistic regression using stochastic gradient descent trainnb: : Train the Vector-based Bayes classifier transpose: : Take the transpose of a matrix validateAdaptiveLogistic: : Validate an AdaptivelogisticRegression model against hold-out data set vecdist: : Compute the distances between a set of Vectors (or Cluster or Canopy, they must fit in memory) and a list of Vectors vectordump: : Dump vectors from a sequence file to text viterbi: : Viterbi decoding of hidden states from given output states sequence[root@sht-sgmhadoopnn-01 ~]#

优秀的个人博客,低调大师

Spark2.0.0 Install And Examples

1.Scala 2.11.8 下载解压[root@sht-sgmhadoopnn-01 hadoop]# wget http://downloads.lightbend.com/scala/2.11.8/scala-2.11.8.tgz[root@sht-sgmhadoopnn-01 hadoop]# tar xzvf scala-2.11.8.tgz[root@sht-sgmhadoopnn-01 hadoop]# mv scala-2.11.8 scala[root@sht-sgmhadoopnn-01 hadoop]# 2.将scala文件夹同步到集群其他机器[root@sht-sgmhadoopnn-01 hadoop]# scp -r scala root@sht-sgmhadoopnn-02:/hadoop/[root@sht-sgmhadoopnn-01 hadoop]# scp -r scala root@sht-sgmhadoopdn-01:/hadoop/[root@sht-sgmhadoopnn-01 hadoop]# scp -r scala root@sht-sgmhadoopdn-02:/hadoop/[root@sht-sgmhadoopnn-01 hadoop]# scp -r scala root@sht-sgmhadoopdn-03:/hadoop/ 3.在集群的每台机器配置环境变量,生效###在文件末尾添加两行[root@sht-sgmhadoopnn-01 hadoop]# vi /etc/profileexport SCALA_HOME=/hadoop/scalaexport PATH=$SCALA_HOME/bin:$PATH [root@sht-sgmhadoopnn-01 hadoop]# scp -r /etc/profile root@sht-sgmhadoopnn-02:/etc/profile[root@sht-sgmhadoopnn-01 hadoop]# scp -r /etc/profile root@sht-sgmhadoopdn-01:/etc/profile[root@sht-sgmhadoopnn-01 hadoop]# scp -r /etc/profile root@sht-sgmhadoopdn-02:/etc/profile[root@sht-sgmhadoopnn-01 hadoop]# scp -r /etc/profile root@sht-sgmhadoopdn-03:/etc/profile [root@sht-sgmhadoopnn-01 hadoop]# source /etc/profile[root@sht-sgmhadoopnn-02 hadoop]# source /etc/profile[root@sht-sgmhadoopdn-01 hadoop]# source /etc/profile[root@sht-sgmhadoopdn-02 hadoop]# source /etc/profile[root@sht-sgmhadoopdn-03 hadoop]# source /etc/profile ---------------------------------------------------------------------------------------------------------------------1.Spark2.0.0下载解压[root@sht-sgmhadoopnn-01 hadoop]# wget http://apache.website-solution.net/spark/spark-2.0.0/spark-2.0.0-bin-hadoop2.7.tgz[root@sht-sgmhadoopnn-01 hadoop]# tar xzvf spark-2.0.0-bin-hadoop2.7.tgz[root@sht-sgmhadoopnn-01 hadoop]# mv spark-2.0.0-bin-hadoop2.7 spark 2.配置spark-env.sh[root@sht-sgmhadoopnn-01 conf]# pwd/hadoop/spark/conf[root@sht-sgmhadoopnn-01 conf]# cp spark-env.sh.template spark-env.sh[root@sht-sgmhadoopnn-01 conf]# ###添加以下5行[root@sht-sgmhadoopnn-01 conf]# vi spark-env.sh export SCALA_HOME=/hadoop/scalaexport JAVA_HOME=/usr/java/jdk1.7.0_67-clouderaexport SPARK_MASTER_IP=172.16.101.55export SPARK_WORKER_MEMORY=1gexport SPARK_PID_DIR=/hadoop/pidexport HADOOP_CONF_DIR=/hadoop/hadoop/etc/hadoop 3.配置slaves文件[root@sht-sgmhadoopnn-01 conf]# cp slaves.template slaves[root@sht-sgmhadoopnn-01 conf]# vi slaves sht-sgmhadoopdn-01sht-sgmhadoopdn-02sht-sgmhadoopdn-03 4.将spark文件夹copy到配置slaves文件的机器上[root@sht-sgmhadoopnn-01 hadoop]# scp -r spark root@sht-sgmhadoopdn-01:/hadoop/[root@sht-sgmhadoopnn-01 hadoop]# scp -r spark root@sht-sgmhadoopdn-02:/hadoop/[root@sht-sgmhadoopnn-01 hadoop]# scp -r spark root@sht-sgmhadoopdn-03:/hadoop/ 5.在集群的每台机器配置环境变量,生效[root@sht-sgmhadoopnn-01 hadoop]# vi /etc/profileexport SPARK_HOME=/hadoop/scalaexport PATH=$SPARK_HOME/bin:$PATH [root@sht-sgmhadoopnn-01 hadoop]# scp -r /etc/profile root@sht-sgmhadoopnn-02:/etc/profile[root@sht-sgmhadoopnn-01 hadoop]# scp -r /etc/profile root@sht-sgmhadoopdn-01:/etc/profile[root@sht-sgmhadoopnn-01 hadoop]# scp -r /etc/profile root@sht-sgmhadoopdn-02:/etc/profile[root@sht-sgmhadoopnn-01 hadoop]# scp -r /etc/profile root@sht-sgmhadoopdn-03:/etc/profile [root@sht-sgmhadoopnn-01 hadoop]# source /etc/profile[root@sht-sgmhadoopnn-02 hadoop]# source /etc/profile[root@sht-sgmhadoopdn-01 hadoop]# source /etc/profile[root@sht-sgmhadoopdn-02 hadoop]# source /etc/profile[root@sht-sgmhadoopdn-03 hadoop]# source /etc/profile 6.启动spark[root@sht-sgmhadoopnn-01 sbin]# ./start-all.shstarting org.apache.spark.deploy.master.Master, logging to /hadoop/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-sht-sgmhadoopnn-01.outsht-sgmhadoopdn-01: starting org.apache.spark.deploy.worker.Worker, logging to /hadoop/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-sht-sgmhadoopdn-01.telenav.cn.outsht-sgmhadoopdn-02: starting org.apache.spark.deploy.worker.Worker, logging to /hadoop/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-sht-sgmhadoopdn-02.telenav.cn.outsht-sgmhadoopdn-03: starting org.apache.spark.deploy.worker.Worker, logging to /hadoop/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-sht-sgmhadoopdn-03.telenav.cn.out[root@sht-sgmhadoopnn-01 sbin]# 7.web查看http://sht-sgmhadoopnn-01:8080/ [root@sht-sgmhadoopnn-01 sbin]# jps27169 HMaster26233 NameNode26641 ResourceManager2312 Jps26542 DFSZKFailoverController2092 Master27303 RunJar26989 JobHistoryServer [root@sht-sgmhadoopdn-01 ~]# jps19907 Worker2086 jar17265 DataNode17486 NodeManager20055 Jps17377 JournalNode17697 HRegionServer3671 QuorumPeerMain 8.运行WordCount案例[root@sht-sgmhadoopnn-01 hadoop]# vi wordcount.txthello abc 123abc hadoop hello hdfsspark yarn123 abc hello hdfs spark wjp wjp abc hello [root@sht-sgmhadoopnn-01 bin]# spark-shellscala> scala> val textfile = sc.textFile("file:///hadoop/wordcount.txt")textfile: org.apache.spark.rdd.RDD[String] = file:///hadoop/wordcount.txt MapPartitionsRDD[1] at textFile at :24 scala> val count=textfile.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_+_)count: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[4] at reduceByKey at :26 scala> count.collect()res0: Array[(String, Int)] = Array((hello,4), (123,2), (yarn,1), (abc,4), (wjp,2), (spark,2), (hadoop,1), (hdfs,2)) scala> ###val file=sc.textFile("hadoop fs -ls hdfs://172.16.101.56:8020/wordcount.txt") val file = sc.textFile("hdfs://namenode:8020/path/to/input")val counts = file.flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _)counts.saveAsTextFile("hdfs://namenode:8020/output") --------------------------------------------------------------------------------------------------------------------------------------------------------a.本地模式两线程运行#[root@sht-sgmhadoopdn-01 ~]# ./bin/run-example SparkPi 2>&1 | grep "Pi is roughly" [root@sht-sgmhadoopnn-01 spark]# ./bin/run-example SparkPi 10 --master local[2] b.Spark Standalone 集群模式运行[root@sht-sgmhadoopnn-01 spark]# ./bin/spark-submit \ --class org.apache.spark.examples.SparkPi \--master spark://sht-sgmhadoopnn-01:7077 \examples/jars/spark-examples_2.11-2.0.0.jar \100 c.注意 Spark on YARN 支持两种运行模式,分别为yarn-cluster和yarn-client,具体的区别可以看这篇博文,从广义上讲,yarn-cluster适用于生产环境;而yarn-client适用于交互和调试,也就是希望快速地看到application的输出。 #Spark on YARN 集群上 yarn-cluster 模式运行[root@sht-sgmhadoopnn-01 spark]# ./bin/spark-submit \--class org.apache.spark.examples.SparkPi \--master yarn-cluster ./examples/jars/spark-examples_2.11-2.0.0.jar \10 spark-submit \--class org.apache.spark.examples.SparkPi \--master yarn-cluster \--num-executors 3 \--driver-memory 4g \--executor-memory 2g \--executor-cores 1 \$SPARK_HOME/examples/jars/spark-examples_2.11-2.0.0.jar \10

资源下载

更多资源
Mario

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长,特征是大鼻子、头戴帽子、身穿背带裤,还留着胡子。与他的双胞胎兄弟路易基一起,长年担任任天堂的招牌角色。

Nacos

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称,一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集,帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

Rocky Linux

Rocky Linux

Rocky Linux(中文名:洛基)是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版,作为CentOS稳定版停止维护后与RHEL(Red Hat Enterprise Linux)完全兼容的开源替代方案,由社区拥有并管理,支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性,采用模块化包装和SELinux安全架构,默认包含GNOME桌面环境及XFS文件系统,支持十年生命周期更新。

WebStorm

WebStorm

WebStorm 是jetbrains公司旗下一款JavaScript 开发工具。目前已经被广大中国JS开发者誉为“Web前端开发神器”、“最强大的HTML5编辑器”、“最智能的JavaScript IDE”等。与IntelliJ IDEA同源,继承了IntelliJ IDEA强大的JS部分的功能。

用户登录
用户注册