英特尔深度学习框架BigDL——a distributed deep learning library for Apache Spark
BigDL: Distributed Deep Learning on Apache Spark What is BigDL? BigDL is a distributed deep learning library for Apache Spark; with BigDL, users can write their deep learning applications as standard Spark programs, which can directly run on top of existing Spark or Hadoop clusters. Rich deep learning support.Modeled afterTorch, BigDL provides comprehensive support for deep learning, including numeric computing (viaTensor) and high levelneural networks; in addition, users can load pre-trainedCaffeorTorchmodels into Spark programs using BigDL. Extremely high performance.To achieve high performance, BigDL usesIntel MKLand multi-threaded programming in each Spark task. Consequently, it is orders of magnitude faster than out-of-box open sourceCaffe,TorchorTensorFlowon a single-node Xeon (i.e., comparable with mainstream GPU). Efficiently scale-out.BigDL can efficiently scale out to perform data analytics at "Big Data scale", by leveragingApache Spark(a lightning fast distributed data processing framework), as well as efficient implementations of synchronous SGD and all-reduce communications on Spark. Why BigDL? You may want to write your deep learning programs using BigDL if: You want to analyze a large amount of data on the same Big Data (Hadoop/Spark) cluster where the data are stored (in, say, HDFS, HBase, Hive, etc.). You want to add deep learning functionalities (either training or prediction) to your Big Data (Spark) programs and/or workflow. You want to leverage existing Hadoop/Spark clusters to run your deep learning applications, which can be then dynamically shared with other workloads (e.g., ETL, data warehouse, feature engineering, classical machine learning, graph analytics, etc.) How to use BigDL? More information can be found at the BigDL project website: https://bigdl-project.github.io/ In particular, you can check out theGetting Started pagefor a quick overview of how to use BigDL For step-by-step deep leaning tutorials on BigDL (using Python), you can check out theBigDL Tutorials project You can join theBigDL Google Group(or subscribe to theMail List) for more questions and discussions on BigDL You can post bug reports and feature requests at theIssue Page 本文转自张昺华-sky博客园博客，原文链接：http://www.cnblogs.com/bonelee/p/7349523.html，如需转载请自行联系原作者
