首页 文章 精选 留言 我的

精选列表

搜索[AI原生],共10000篇文章
优秀的个人博客,低调大师

原生技术之Docker入门

1. 为什么需要容器? 下图是一个比较传统的软件架构: 做过java的同学可能对上图的架构方式比较了解,我们通常会将一个应用程序生成一个war包,放到一个tomcat容器当中并在一台虚拟机(VM)中启动运行,然后配置nginx的负载均衡策略,将来自用户的请求转发到某个tomcat应用上,这种基于主机或虚拟机部署的应用会存在以下几个问题: 可移植性差 需要事先安装应用所需要的运行环境,比如java应用所需要的jdk或者jre,如果需要重新部署一个应用,就需要重新初始化环境再安装应用,过程繁琐; 另外如果一个应用需要jdk7的运行环境另外一个应用需要jdk8,那在一台主机上就很难满足; 可维护性差 如果tomcat应用本身或者所在的虚拟机操作系统出现问题时,则需要人工干预,比如配置nginx转发规则、执行重启操作等; 可扩展性差 应用的负载有高有低,不够稳定,当前应用负载大的时候,我们需要增加应用的数量,当应用负载降低的时候,我们需要降低应用的数量; 无法资源隔离 如果一台虚拟机部署多个应用,不同的应用或者进程之间会相互影响; ... 我们接下来就来看一下我们是如何一步步的解决这些问题的。 首先是容器化,我们选择的方案是Docker。 Docker将应用程序与该程序的依赖,打包成一个容器镜像,运行这个文件就会生成虚拟容器。程序在这个虚拟容器里运行,就好像运行在真实的物理机上,并且每个容器之间资源互相隔离而且都有自己的文件系统,这样容器之间进程不会相互影响,可以通过下图来进行对比基于虚拟机和基于容器部署应用的区别: 2. Docker介绍 2.1 Docker架构 Docker是客户端-服务器架构的应用,主要由以下部分组成: 服务端是一个名为dockerd守护进程,用来监听REST API请求并管理Docker对象,比如镜像、容器、存储卷及网络等。 命令行客户端(CLI),也就是我们平常在控制台输入的docker命令行,通过调用REST API进行控制Docker daemon或者同其进行集成。 镜像仓库(Docker Registries),镜像仓库用来存储Docker镜像。 以下是Docker的架构示意图: 2.2 Docker对象 IMAGES 镜像一般是通过指令创建的只读文件,用来生成容器。一般一个镜像是基于另外一个镜像并添加一些额外的指令创建的,可以通过一个名为Dockerfile的文件来生成一个镜像,在Dockerfile中的每一行指令会生成一层(layer)。当Dockerfile有改动需要重新生成镜像时,只需要重新生成改变的那些层就可以,这样就可以使得镜像文件更加轻量、快速构建。 CONTAINERS 容器是通过镜像文件生成的运行实例。可以通过REST API或者docker client进行创建、启动、停止、移动或者删除一个容器。 SERVICE 用来管理和扩展多个容器,需要同docker swarm一起工作 2.3 底层技术 Docker采用go语言编写,并且使用了Linux内核中的几个特性来实现其功能,主要有如下: Namespaces Docker通过Namespaces来提供隔离的工作空间(Workspace),当你运行一个容器的时候,Docker为这个容器创建了数个不同类型的Namespaces,主要有以下类型: pid namespace:提供进程隔离功能 net namespace:管理网络接口 ipc namespace:内部资源访问控制 (IPC:Inter Process Communication) mnt namespace:管理文件系统挂载 uts namespace: 内核隔离以及版本识别(UTS:Unix Timesharing System) CGroups(Control Groups) Docker通过CGroup来限定容器只能使用特定的资源。举例来讲,Docker可以限制某个容器只能使用多少cpu及内存资源。 UnionFS(Union File System) 一种文件系统类型,可以运行在其他文件系统上,通过创建不同的层来使得容器文件系统更加轻量和快速。还有其他几种类似的文件系统,包括AUFS、btrfs、vfs和DeviceMapper。 3. Docker的安装部署 以下命令是在Centos7上的命令,其他操作系统会存在一些差异 yum install docker:通过yum下载docker相关的依赖 systemctl enable docker: 开机运行systemctl start docker: 启动docker服务 执行完上述操作,docker服务已经在运行了,可以通过执行 docker version 和 docker info 命令查看docker的版本以及相关的信息。 4. Docker的使用 4.1 Dockerfile文件 我们之前有提到Docker可以将应用程序打包成一个镜像,那么如何生成镜像文件呢?这就需要用到Dockerfile文件。它是一个文本文件,用来配置镜像,Docker根据该文件生成二进制的镜像文件。以下是一个Dockerfile文件示例: # 该镜像文件继承官方的nginx镜像,冒号表示标签,这里标签是latest,表示最新的版本 FROM nginx:latest # 将_book目录下的文件copy至镜像文件的/var/www/public目录 COPY _book /var/www/public/ COPY nginx_app.conf/etc/nginx/conf.d/ nginx_app.conf # 将容器的8080端口暴露出来,允许外部连接这个端口 EXPOSE 8080 # 容器启动后执行 nginx -g daemon off 命令 CMD ["nginx", "-g", "daemon off;"] 4.2 创建镜像文件 有了Dockerfile文件以后,就可以用docker build命令创建镜像文件了。 docker build -t zcloud-document:0.0.1. docker image ls 如果运行成功,就可以看到新生成的镜像文件zcloud-document了。 4.3 生成容器 # 生成容器 docker run -p 8080:8080 -it zcloud-document:0.0.1 docker ps # 重新生成一个新的镜像标签,并指向原来的镜像 docker tag zcloud-document:0.0.1 10.0.0.183:5000/zcloud/zcloud-document:0.0.1 # 推送到私有镜像仓库 docker push 10.0.0.183:5000/zcloud/zcloud-document:0.0.1 关于Docker其他的一些操作命令,大家可以自行查阅,网上介绍的文章也比较多,参考文章:Docker 入门教程(https://docs.docker.com/get-started/)

优秀的个人博客,低调大师

Java原生实现定时器

public static void main(String[] args) { //新加定时器 //这个方法schedule(TimerTask task, Date firstTime, long period) //获得当天的日期 SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd"); Date date = new Date(); //定义开始时间字符 String timeStr = "15:27:00"; timeStr = sdf.format(date)+" "+timeStr; //获得当天的指定时间的date对象 sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"); try{ date = sdf.parse(timeStr); //判断今天的执行时间是否已经过去,如果过去则改为明天 if(date.getTime() date = new Date(date.getTime()+24*60*60*1000); } }catch(Exception e){ e.printStackTrace(); } Timer timer = new Timer(); //timer.schedule(new testTask(), date, 246060*1000); timer.schedule(new testTask(),date,246060*1000); while(true){ try{ int ch = System.in.read(); if(ch-'c'==0){ timer.cancel(); } } catch(Exception e){ e.printStackTrace(); } } //new roomdhtranSys().commoneMethod(); //System.out.println("结束!"); } package com; import java.text.SimpleDateFormat;import java.util.Date;import java.util.TimerTask; public class testTask extends TimerTask { public void run(){ new roomdhtranSys().commoneMethod(); }}

优秀的个人博客,低调大师

《Cloud Native 》云原生技术汇总

An awesome & curated list of best applications and tools for Cloud Native. This Awesome Repository is highly inspired from cncf's landscape & Awesome. Items marked with Open-Source Software are open-source software. Items marked with Freeware are free. --> Cloud Native Services Components Cloud Provisioning Runtime Orchestration & Management Application Definition & Development Platform Serverless Observability & Analysis Cloud Public - A public cloud is a pool of virtual resources—developed from hardware owned and managed by a third-party company—that is automatically provisioned and allocated among multiple clients through a self-service interface. Alibaba Cloud - Alibaba Cloud develops highly scalable cloud computing and data management services. Amazon Web Services - Amazon Web Services provides information technology infrastructure services to businesses in the form of web services. Azure Cloud - Microsoft is a software corporation that develops, manufactures, licenses, supports, and sells a range of software products and services. Baidu Cloud - Baidu is a Chinese website and search engine that enables individuals to obtain information and find what they need. DigitalOcean - DigitalOcean is an IaaS company that delivers a seamless way for developers and businesses to deploy and scale any application in the cloud. Fujitsu K5 - Fujitsu provides information technology and communications solutions. Google Cloud - Google is a multinational corporation that is specialized in internet-related services and products. Huawei Cloud - Huawei Technologies provides infrastructure application software and devices with wireline, wireless, and IP technologies. IBM Cloud - IBM is an IT technology and consulting firm providing computer hardware, software, and infrastructure and hosting services. Oracle Cloud - Oracle is a computer technology corporation developing and marketing computer hardware systems and enterprise software products. Joyent Cloud - Your Cloud, Your Way Packet Cloud - Packet is a bare metal cloud built for developers. 8 minute deploys, no hypervisor, & full automation support from 15 global data centers. Tencent Cloud - Tencent is a Chinese internet service portal offering value-added internet, mobile, telecom, and online advertising services. Citrix Cloud - Move Faster, Work Better, Lower IT Costs Private - The private cloud is defined as computing services offered either over the Internet or a private internal network and only to select users instead of the general public. Also called an internal or corporate cloud, private cloud computing gives businesses many of the benefits of a public cloud - including self-service, scalability, and elasticity - with the additional control and customization available from dedicated resources over a computing infrastructure hosted on-premises. Openstack - Repository containing OpenStack repositories Scaleway - Scaleway is the world's first Cloud Computing IaaS platform Foreman - an application that automates the lifecycle of servers Digital Bebar - Digital Rebar Provision is a simple but powerful Golang executable that provides a complete API-driven DHCP/PXE/TFTP provisioning system. MAAS - Official MAAS repository mirror. (Do not submit pull requests or bugs here; use Launchpad instead.) VMware - VMware is a software company providing cloud and virtualization services. Hybrid - A hybrid cloud is a computing environment that combines a public cloud and a private cloud by allowing data and applications to be shared between them. Ensono - Complete hybrid IT services – from cloud to mainframe. Operate for today. Optimize for tomorrow. Dellemc - Dell EMC is a powerful part of Dell Technologies' commitment to your transformation Hpe - Hybrid Cloud Solutions Scalr - The Hybrid Cloud Management Platform IBM Z hybrid cloud Rackspace Hybrid Cloud Microsoft Hybrid Cloud VMware Hybrid Cloud AWS Hybrid Cloud Provisioning Container Registries Container Registry is a private Docker repository that works with popular continuous delivery systems. - [ECR](https://aws.amazon.com/cn/ecr/) - Amazon Elastic Container Registry (ECR) is a secure, fully-managed Docker container registry that makes it easy for developers to store, manage, and deploy Docker container images. - [Azure Registry](https://azure.microsoft.com/en-us/services/container-registry/) - Manage a Docker private registry as a first-class Azure resource. - [Codefresh Registry](https://codefresh.io/registry-beta/) - Codefresh is a Docker-native CI/CD platform.Instantly build, test and deploy Docker images. - [Docker Registry](https://docs.docker.com/registry/) - Docker Trusted Registry (DTR) is a commercial product that enables complete image management workflow, featuring LDAP integration, image signing, security scanning, and integration with Universal Control Plane. DTR is offered as an add-on to Docker Enterprise subscriptions of Standard or higher. - [Google Container Registry](https://cloud.google.com/container-registry/) - High-speed, private Docker image storage on Google Cloud Platform - [Harbor](http://vmware.github.io/harbor/) - An Enterprise-class Container Registry Server based on Docker Distribution. - [JFrog Artifactory](https://jfrog.com/artifactory/) - Enterprise Universal Artifact Manager. - [Portus](http://port.us.org/) - Portus is an open source authorization service and user interface for the next generation Docker Registry. - [Project Atomic](https://www.projectatomic.io/) - Atomic Host provides immutable infrastructure for deploying to hundreds or thousands of servers in your private or public cloud. - [QUAY Enterprise](https://coreos.com/quay-enterprise/) - One container registry for your entire enterprise. Host Management & Tooling Host management tool - [Ansible](https://www.ansible.com/) - Ansible is designed around the way people work and the way people work together. - [Chef](https://www.chef.io) - Ship better software, faster.Enable collaboration and continuous automation across your infrastructure, applications, and compliance for all your apps and infrastructure. - [LniuxKit](https://github.com/linuxkit/linuxkit) - A toolkit for building secure, portable and lean operating systems for containers - [CFEngine](https://cfengine.com/) - CFEngine Community - [Puppet](https://puppet.com/) - Server automation framework and application - [Rundeck](https://www.rundeck.com/) - Enable Self-Service Operations: Give specific users access to your existing tools, services, and scripts - [Saltstack](https://saltstack.com/) - Intelligent automation for a software-defined world Infrastructure Automation Infrastructure automation makes servers and VM management more flexible, efficient, and scalable by converting management tasks and policy into code. - [AWS CloudFormation](https://aws.amazon.com/cn/cloudformation/) - [HOSH](https://bosh.io) - BOSH is an open source tool for release engineering, deployment, lifecycle management, and monitoring of distributed systems. - [Helm](https://helm.sh/) - Helm is the best way to find, share, and use software built for Kubernetes. - [Infrakit](https://github.com/docker/infrakit) - A toolkit for creating and managing declarative, self-healing infrastructure. - [Juju](https://jujucharms.com/) - Juju is an open source application modelling tool. Deploy, configure, scale and operate your software on public and private clouds. - [Cloud Coreo](https://www.cloudcoreo.com/) - A Platform for Modern Cloud Teams - [Cloudify](https://cloudify.co/) - Radically Simplifying Multi-Cloud Orchestration - [Kubicorn](http://kubicorn.io/) - Create, manage, snapshot, and scale Kubernetes infrastructure in the public cloud. - [ManageIQ](http://manageiq.org/) - Discover, Optimize, and Control your Hybrid IT - [Terraform](https://www.terraform.io/) - Write, Plan, and Create Infrastructure as Code Key Management Key management is the name of management of cryptographic keys in a cryptosystem. Secure Images Secure your images so that you maintain control of how they are displayed on the Internet. - [Notary](https://github.com/theupdateframework/notary) - Notary is a project that allows anyone to have trust over arbitrary collections of data - [TUF](https://theupdateframework.github.io/) - A framework for securing software update systems - [Aqua](https://www.aquasec.com/) - The Aqua Container Security Platform provides development-to-production lifecycle controls for securing containerized applications that run on-premises or in the cloud, on Windows or Linux, supporting multiple orchestration environments. - [Clair](https://coreos.com/clair) - Clair is an open source project for the static analysis of vulnerabilities in appc and docker containers. - [OpenSCAP](https://www.open-scap.org/) - Discover a wide array of tools for managing system security and standards compliance. - [Twistlock](https://www.twistlock.com/) - Container Security for Docker, Kubernetes and Beyond - [Anchore](https://anchore.com/) - An open source complete solution for compliance, certification, security scanning, and auditing of public and private container images. - [anchore.io](https://anchore.io/) - Discover, Analyze, and Certify Container Images. - [Black Duck](https://www.blackducksoftware.com/) - Complete Visibility. Automated Control. - [NeuVector](https://neuvector.com/) - Continuous Network Security for Kubernetes Containers - [Sonatype Nexus](https://www.sonatype.com/) - The world's best way to organize, store, and distribute software components. Runtime Cloud-Native Network Network Segmentation and Policy,SDN & APIs (eg CNI, libnetwork) Incubating CNCF Projects CNI - Container Network Interface - networking for Linux containers CNCF Member Products/Projects Aporeto - Cloud Native Security for Containers and Microservices Cannl - Policy based networking for cloud native applications Contiv - Container networking for various use cases Flannel - flannel is a network fabric for containers, designed for Kubernetes NSX - VMware is a software company providing cloud and virtualization services. Open vSwitch - Open vSwitch is a multilayer software switch licensed under the open source Apache 2 license. OpenContrial - An open-source network virtualization platform for the cloud. Project Calico - Cloud native application connectivity and network policy Weave Net - Simple, resilient multi-host Docker networking and more. Non-CNCF Member Products/Projects Aviatrix - The company develops software that enables enterprises to build hybrid clouds by easily Big Switch Networks - Big Switch Networks is the Next-Generation Data Center Networking Company, designing intelligent, agile and flexible networks Cilium - HTTP, gRPC, and Kafka Aware Security and Networking for Containers with BPF and XDP Cumulus - Cumulus Networks, a software company, designs, and sells Linux operating systems for networking hardware. GuardiCoreCentra - GuardiCore provides network security solutions for software defined data centers. MidoNet - MidoNet is an Open Source network virtualization system for Openstack clouds Nuage Networks - Nuage Networks Fundamentals: Software Defined Networking for the Datacenter and Beyond. Plumgrid - PLUMgrid is involved in virtual networking and SDN/NFV to deliver cloud infrastructure solutions that transform businesses. Romana - The Romana Project - Installation scripts, documentation, issue tracker and wiki. Start here. SnapRoute - SnapRoute is an open networking stack company. Cloud-Native Storage Volume Drivers/Plugins,Local Storage Management,Remote Storage Access Sandbox CNCF Projects Rook - File, Block, and Object Storage Services for your Cloud-Native Environments CNCF Member Products/Projects Ceph - Ceph is a unified, distributed storage system designed for excellent performance, reliability and scalability. Container Storage Interface - Container Storage Interface (CSI) Specification. Dell EMC - IT and Workforce Transformation. Made real every day. Diamanti - Diamanti is the first container platform with plug and play network and persistent storage that seamlessly integrates the most widely adopted software stack - standard open source Kubernetes and Docker - so there is no vendor lock-in. QoS on network and storage maximizes container density. Gluster - Gluster is free and open source softeare scalable network filesystem. Hatchway - Persistent Storage for Cloud Native Applications Kasten - Kasten is on a mission to dramatically simplify operational management of stateful cloud-native applications. Manta - Structural variant and indel caller for mapped sequencing data Minio - Minio is a high performance distributed object storage server, designed for large-scale private cloud infrastructure. Minio is widely deployed across the world with over 64.2M+ docker pulls. NetApp - NetApp HCI. All New and Available Now. OpenEBS - OpenEBS is an open source storage platform that provides persistent and containerized block storage for DevOps and container environments. Portworx - The Solution for Stateful Containers in Production. Designed for DevOps. Rex-Ray - REX-Ray is an open source, storage management solution designed to support container runtimes such as Docker and Mesos. StorageOS - Enterprise persistent storage for containers and the cloud. Non-CNCF Member Products/Projects Datera - Datera is an application-driven data infrastructure company. Hedving - Modern storage for the modern business. Infinit - The Elle coroutine-based asynchronous C++ development framework. LeoFS - The LeoFS Storage System OpenIO - OpenIO Software Defined Storage Pure Storage - Pure Storage is an all-flash enterprise storage company that enables broad deployment of flash in data centers. Quobyte - Data Center File System. Fast and Reliable Software Storage Robin Systems - Data-Centric Compute and Storage Containerization Infrastructure Software Sheepdog - Distributed Storage System for QEMU Springpath - Springpath is hyperconvergence software that turns standard servers of choice into a single pool of compute and storage resources. Swift - OpenStack Storage (Swift) Container Runtime The new CF Container Runtime gives you more granular control and management of containers with Kubernetes. Incubating CNCF Projects containerd rkt - rkt is a pod-native container engine for Linux. It is composable, secure, and built on standards. CNCF Member Products/Projects CRI-O - Open Container Initiative-based implementation of Kubernetes Container Runtime Interface Intel Clear Containers - OCI (Open Containers Initiative) compatible runtime using Virtual Machines Ixd - Daemon based on liblxc offering a REST API to manage containers Pouch - Pouch is an open-source project created to promote the container technology movement. runc - CLI tool for spawning and running containers according to the OCI specification SmartOS - Converged Container and Virtual Machine HypervisorNon-CNCF Member Products/Projects Kata Containers - Kata Containers runtimes RunV - Hypervisor-based Runtime for OCI Singularity - Singularity: Application containers for Linux Orchestration & Management Scheduling & Orchestration Graduated CNCF Projects Kubernetes - Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications CNCF Member Products/Projects ECS - Amazon Web Services provides information technology infrastructure services to businesses in the form of web services. Docker Swarm - Swarm: a Docker-native clustering system Microsoft Azure Service Fabric - Service Fabric is a distributed systems platform for packaging, deploying, and managing stateless and stateful distributed applications and containers at large scale. Non-CNCF Member Products/Projects Mesos - Mirror of Apache Mesos Nomad - Nomad is a flexible, enterprise-grade cluster scheduler designed to easily integrate into existing workflows. Coordination & Service Discovery Incubating CNCF Projects CoreDNS - CoreDNS is a DNS server that chains plugins. CNCF Member Products/Projects ContainerPilot - A service for autodiscovery and configuration of applications running in containers etcD - Distributed reliable key-value store for the most critical data of a distributed system VMware Haret - A strongly consistent distributed coordination system, built using proven protocols & implemented in Rust. Non-CNCF Member Products/Projects Apache Zookeeper - Apache ZooKeeper is an effort to develop and maintain an open-source server which enables highly reliable distributed coordination. Consul - Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure. Eureka - AWS Service registry for resilient mid-tier load balancing and failover. SkyDNS - DNS service discovery for etcd SmartStack - A transparent service discovery framework for connecting an SOA Service Management Envoy - C++ front/service proxy gRPC - The C based gRPC (C++, Python, Ruby, Objective-C, PHP, C#) Linkerd - Production-grade feature-rich service mesh for any platform 3Scale - 3scale api gateway reloaded Ambassador - open source Kubernetes-native API gateway for microservices built on the Envoy Proxy Avi Networks - Avi Networks is a Silicon Valley startup with proven track record in building virtualization, networking and software solutions. Conduit - Ultralight service mesh for Kubernetes F5 - F5 Networks provides application delivery networking technology that optimizes the delivery of network-based applications. Heptio Contour - Contour is a Kubernetes ingress controller for Lyft's Envoy proxy. Kong - ? The Microservice API Gateway NGINX - application delivery for the modern web Open Service Broker API - Open Service Broker API Specification Turbine Labs - Turbine Labs Apache Thrift - Mirror of Apache Thrift Avro - Apache Avro Backplane - A service that unifies discovery, routing, and load balancing for web servers written in any language, running in any cloud or datacenter. HAProxy - The Reliable, High Performance TCP/HTTP Load Balancer Hystrix - Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable. Istio - An open platform to connect, manage, and secure microservices. Netflix Zuul - Zuul is a gateway service that provides dynamic routing, monitoring, resiliency, security, and more. Open Policy Agent (OPA) - An open source project to policy-enable your service. Ribbon - Ribbon is a Inter Process Communication (remote procedure calls) library with built in software load balancers. Traefik - Træfik, a modern reverse proxy Vamp - Vamp - canary releasing and autoscaling for microservice systems. Application Definition & Development Database & Data Warehouse Incubating CNCF Projects Vitess - Vitess is a database clustering system for horizontal scaling of MySQL. CNCF Member Products/Projects Cloudhbase - Lightweight, embedded, syncable NoSQL database engine for iOS (and Mac!) apps. IBM DB2 - IBM is an IT technology and consulting firm providing computer hardware, software, and infrastructure and hosting services. Iguazio - iguazio's Continuous Analytics Data Platform has redesigned the data stack to accelerate performance in big data, IoT and cloud-native apps. Infinispan - Infinispan is an open source data grid platform and highly scalable NoSQL cloud data store. Microsoft SQL Server - Microsoft is a software corporation that develops, manufactures, licenses, supports, and sells a range of software products and services. MySQL - MySQL Server, the world's most popular open source database, and MySQL Cluster, a real-time, open source transactional database. Oracle - Oracle is a computer technology corporation developing and marketing computer hardware systems and enterprise software products. RethinkDB - The open-source database for the realtime web. SQL Data Warehouse - Microsoft is a software corporation that develops, manufactures, licenses, supports, and sells a range of software products and services. YugaByte DB - YugaByteDB is a transactional, high-performance database for building distributed cloud services. It currently supports Redis API (as a true DB) and Cassandra API, with SQL coming very soon. Non-CNCF Member Products/Projects ArangoDB - ? ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions. BigchainDB - Meet BigchainDB. The blockchain database. CarbonData - Mirror of Apache CarbonData Cassandra - Mirror of Apache Cassandra CockroachDB - CockroachDB - the open source, cloud-native SQL database. Crate.io - CrateDB is a distributed SQL database that makes it simple to store and analyze massive amounts of machine data in real-time. Druid - Column oriented distributed data store ideal for powering interactive applications. Hadoop - Mirror of Apache Hadoop MariaDB - MariaDB server is a community developed fork of MySQL server. Started by core members of the original MySQL team, MariaDB actively works with outside developers to deliver the most featureful, stable, and sanely licensed open SQL server in the industry. MemSQL - A real-time data warehouse you can run everywhere MongoDB - MongoDB is a document database with the scalability and flexibility that you want with the querying and indexing that you need NomsDB - The versioned, forkable, syncable database OrientDB - OrientDB is the most versatile DBMS supporting Graph, Document, Reactive, Full-Text, Geospatial and Key-Value models in one Multi-Model product. OrientDB can run distributed (Multi-Master), supports SQL, ACID Transactions, Full-Text indexing and Reactive Queries. OrientDB Community Edition is Open Source using a liberal Apache 2 license. Pachyderm - Reproducible Data Science at Scale! Pilosa - Pilosa is an open source, distributed bitmap index that dramatically accelerates queries across multiple, massive data sets. PostgreSQL - PostgreSQL is a powerful, open source object-relational database system. Presto - Distributed SQL query engine for big data Qubole - Qubole delivers a Self-Service Platform for Big Data Analytics built on Amazon, Microsoft, Google and Oracle Clouds. Redis - Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, HyperLogLogs, Bitmaps. Scylla - NoSQL data store using the seastar framework, compatible with Apache Cassandra Snowflake - Snowflake is the only data warehouse built for the cloud. Software AG - Software AG provides business process management, data management, and consulting services worldwide. Starburst - Starburst (www.starburstdata.com) is the enterprise Presto company offering an SQL-on-Anything analytics platform. TiDB - TiDB is a distributed HTAP database compatible with the MySQL protocol. Vertica - Vertica Systems develops data management solutions for storing databases and allowing clients to conduct real-time and ad hoc queries. Streaming Incubating CNCF Projects NATS - High-Performance server for NATS, the cloud native messaging system. CNCF Member Products/Projects Amazon Kinesis - Amazon Web Services provides information technology infrastructure services to businesses in the form of web services. CloudEvents - CloudEvents Specification Google Cloud Dataflow - Google is a multinational corporation that is specialized in internet-related services and products. Heron - Heron is a realtime, distributed, fault-tolerant stream processing engine from Twitter Non-CNCF Member Products/Projects Apache Apex - Mirror of Apache Apex core Apache NiFi - Mirror of Apache NiFi Apache RocketMQ - Mirror of Apache RocketMQ Apache Spark - Mirror of Apache Spark Apache Storm - Mirror of Apache Storm Flink - Mirror of Apache Flink Kafka - Mirror of Apache Kafka Pulsar - Pulsar - distributed pub-sub messaging system RabbitMQ - RabbitMQ is the most widely deployed open source message broker. StreamSets - StreamSets DataCollector - Continuous big data ingest infrastructure. Source Code Management GitHub - GitHub is a web-based Git repository hosting service offering distributed revision control and source code management functionality of Git. GitLab - GitLab CE | Please open new issues in our issue tracker on GitLab.com Visual Studio Team Services - Microsoft is a software corporation that develops, manufactures, licenses, supports, and sells a range of software products and services. Bitbucket - Atlassian provides collaboration software for teams with products including JIRA, Confluence, HipChat, Bitbucket, and Stash. Application Definition Bitnami - Loved by Devs, Trusted by Ops. Easy to use cloud images, containers, and VMs that work on any platform Docker Compose - Define and run multi-container applications with Docker Habitat - Modern applications with built-in automation OpenAPI - The OpenAPI Specification Repository Telepresence - Local development against a remote Kubernetes or OpenShift cluster Apache Brooklyn - Apache Brooklyn KubeVirt - A virtualization API and runtime add-on for Kubernetes in order to define and manage virtual machines. Packer - Packer is a tool for creating identical machine images for multiple platforms from a single source configuration. CI & CD Continuous integration and continuous delivery are two approaches to software development that are designed to improve code quality and enable rapid delivery and deployment of code. CNCF Member Products/Projects Argo - Get stuff done with container-native workflows for Kubernetes. Cloud 66 Skycap - Ops tools for Devs. Build, deliver, deploy and manage any applications on any cloud or server. Cloudbees - CloudBees offers CloudBees Jenkins Enterprise, an enterprise-grade continuous delivery platform powered by Jenkins. Codefresh - Codefresh is a continuous delivery and collaboration platform for containers and microservices. Codeship - CloudBees offers CloudBees Jenkins Enterprise, an enterprise-grade continuous delivery platform powered by Jenkins. Concourse - BOSH release and development workspace for Concourse ContainerOps - DevOps Orchestration Platform Habitus - A Build Flow Tool for Docker Runner - GitLab Runner is the open source project that is used to run your jobs and send the results back to GitLab. Weave Flux - A tool for deploying container images to Kubernetes services Wercker - The Wercker CLI can be used to execute pipelines locally for both local development and easy introspection. Non-CNCF Member Products/Projects Appveyor - Appveyor Systems Inc. aim is to give powerful continuous integration and deployment tools to every .NET developer. Bamboo - Atlassian provides collaboration software for teams with products including JIRA, Confluence, HipChat, Bitbucket, and Stash. BuddyBuild - Buddybuild is a Vancouver-based app tools company focused on continuous integration and debugging tools. Buildkite - The Buildkite Agent is an open-source toolkit written in Golang for securely running build jobs on any device or network CircleCI - CircleCI provides software teams the confidence to build, test, and deploy across numerous platforms. Distelli - True Continuous Delivery from Source Control to Servers. Drone - Drone is a Continuous Delivery platform built on Docker, written in Go Jenkins - Build great things at any scale Octopus Deploy - Octopus Deploy is a user-friendly release management OpenStack Zuul CI - The Gatekeeper, or a project gating system Semaphore - Hosted continuous integration and deployment service Shippable - Shippable helps companies ship code faster by giving them a powerful continuous integration platform built natively on Docker. Solano Labs - Continuous Integration & Deployment Spinnaker - Spinnaker is an open source, multi-cloud continuous delivery platform for releasing software changes with high velocity and confidence. Travis - The Ember web client for Travis CI XL Deploy - XebiaLabs develops enterprise-scale Continuous Delivery and DevOps software. Platform Certified Kubernetes - Distribution Apprenda Kismatic Enterprise Toolkit (KET) Appscode Pharmer Caicloud Compass Canonical Distribution of Kubernetes Cloud Foundry Container Runtime CoreOS bootkube DaoCloud Enterprise Diamanti Converged Container Infrastructure Docker EE/CE Ghostcloud EcOS Giant Swarm Managed Kubernetes Google Kube-Up Heptio Quickstart for Kubernetes IBM Cloud Private inwinSTACK kube-ansible Joyent Triton for Kubernetes kube-spawn Kublr Loodse Kubermatic Mesosphere Kubernetes on DC/OS Mirantis Cloud Platform Netease Container Service Dedicated OpenShift Oracle Linux Container Services Oracle Terraform Kubernetes Installer Pivotal Container Service (PKS) Platform9 Managed Kubernetes QFusion Rancher Samsung Kraken StackPointCloud SUSE CaaS Platform Tectonic VMWare Pivotal Container Service (PKS) Weaveworks kubeadm WiseCloud Typhoon Certified Kubernetes - Platform Alauda EE Alibaba Cloud Container Service Azure (ACS) Engine Azure Container Service (AKS) Baidu Cloud Container Engine BoCloud BeyondcentContainer Cisco Container Platform EasyStack Kubernetes Service (EKS) eKing Cloud Container Platform Google Kubernetes Engine (GKE) HarmonyCloud Container Platform Hasura Huawei Cloud Container Engine (CCE) IBM Cloud Container Service Nirmata Managed Kubernetes Oracle Container Engine SAP Certified Gardener Tencent Cloud Container Service (CCS) TenxCloud Container Engine (TCE) ZTE TECS Non-Certified Kubernetes Amazon Elastic Container Service for Kubernetes (EKS) Cloud 66 Maestro Containership Gravitatonal Telekube Huawei FusionStage Navops Supergiant goPaddle Stratoscale Symphony PaaS & Container Service Apcera - Ericsson is a technology company that provides and operates telecommunications networks, television and video systems, and related services. Cloud Foundry Application Runtime - Cloud Foundry Application Runtime utilizes containers as part of its DNA, and has since before Docker popularized containers. The new CF Container Runtime gives you more granular control and management of containers with Kubernetes. Datawire - An early stage startup that's focused on making it easy for developers to build resilient microservices. Exoscale - Exoscale is the cloud hosting platform for SaaS companies, developers and systems administrators. Galactic Fog - Build Future-Proof Applications. Simplify integration. Run applications anywhere. Adapt to changes instantly. Heroku - Salesforce is a global cloud computing company that develops CRM solutions and provides business software on a subscription basis. Atomist - Atomist make microservice applications Easy and fun to build, through a cloud-based service. Clouber - Clouber is a provider of mCenter, an Application Modernization/Migration / Management platform across hybrid (public and private) clouds. Convox - Launch a Private Cloud in Minutes. Empire - A PaaS built on top of Amazon EC2 Container Service (ECS) Flynn - A next generation open source platform as a service (PaaS Hyper - Hyper.sh is a secure container cloud service. JHipster - Open Source application generator for creating Spring Boot + Angular projects in seconds! Kontena - The developer friendly container and micro services platform. Works on any cloud, easy to setup, simple to use. Lightbend - Lightbend (formerly Typesafe) is dedicated to helping developers build Reactive applications on the JVM. No Code - The best way to write secure and reliable applications. Write nothing; deploy nowhere. PaaSTA - An open, distributed platform as a service Platform.sh - Platform.sh is an automated, continuous-deployment high-availability cloud hosting solution Portainer - Simple management UI for Docker Scalingo - Scalingo a Docker Platform Service Transform your code as Docker container & run it on our cloud, making it instantly available & scalable. Tsuru - Open source, extensible and Docker-based Platform as a Service (PaaS). Serverless Security PureSec - PureSec is the world's leading Serverless Security Runtime Environment Snyk - Snyk is a security company helping to monitor app vulnerabilities. Libraries Python Lambda - A toolkit for developing and deploying serverless Python code in AWS Lambda. Tools Architect - ? cloud function signatures for http handlers, pubsub, scheduled functions and table triggers Dashbird - AWS Lambda monitoring & debugging platform. Serverless observability & troubleshooting. Serverless monitoring. IOpipe - IOpipe provides a toolbox for developing, monitoring, and operating serverless applications. Microcule - SDK and CLI for spawning streaming stateless HTTP microservices in multiple programming languages Node Lambda - Command line tool to locally run and deploy your node.js application to Amazon Lambda Stackery - Run serverless in production with Stackery's serverless operations console. Thundra - IT Alert and Notifications Management Frameworks AWS Chalice - Python Serverless Microframework for AWS SAM Local - AWS Serverless Application Model (AWS SAM) prescribes rules for expressing Serverless applications on AWS. Serverless - Serverless Framework – Build web, mobile and IoT applications with serverless architectures using AWS Lambda, Azure Functions, Google CloudFunctions & more! Spring Cloud Function - Pivotal is a software company that provides digital transformation technology and services. Apex - Build, deploy, and manage AWS Lambda functions with ease (with Go support!). Bustle Shep - A framework for building JavaScript Applications with AWS API Gateway and Lambda ClaudisJS - Deploy Node.js projects to AWS Lambda and API Gateway easily Dawson - A serverless web framework for Node.js on AWS (CloudFormation, CloudFront, API Gateway, Lambda) Flogo - Ultralight Edge Microservices Framework Gordon - λ Gordon is a tool to create, wire and deploy AWS Lambdas using CloudFormation GunIO Zappa - Serverless Python KappaIO - What precedes Lambda Mitoc Group Deep - Full-stack JavaScript Framework for Cloud-Native Web Applications (perfect for Serverless use cases) Sparta - A GO FRAMEWORK FOR AWS LAMBDA Platforms CNCF Member Products/Projects AWS Lambda - Amazon Web Services provides information technology infrastructure services to businesses in the form of web services. Azure Functions - Microsoft is a software corporation that develops, manufactures, licenses, supports, and sells a range of software products and services. Google Cloud Functions - https://cloud.google.com/functions/ IBM Cloud Functions - IBM is an IT technology and consulting firm providing computer hardware, software, and infrastructure and hosting services. Twilio Functions - Twilio is a cloud communication company that enables users to use standard web languages to build voice, VoIP, and SMS apps via a web API. Non-CNCF Member Products/Projects Algorithmia - Algorithmia is an open marketplace for algorithms, enabling developers to create tomorrows smart applications today. Apache OpenWhisk - Apache OpenWhisk is a serverless event-based programming service and an Apache Incubator project. AppScale - AppScale is an easy-to-manage serverless platform for building and running scalable web and mobile applications on any infrastructure. Clay - Rapid Prototyping for Developers Hyper Func - Hyper.sh is a secure container cloud service. Iron.io - Iron.io is a scalable cloud-based message queue and processing platform for building distributed cloud applications. Nano Lambda - Explore deploying code in lambda.Run server-side code with an API call. Overclock - Overclock Labs develops protocols, tools, and infrastructure to make foundational elements of the internet open, decentralized, and simple OVH Functions - OVH.com is an independent French company that offers web, dedicated, and cloud hosting solutions. PubNub Functions - The PubNub Data Stream Network enables mobile and web developers to build and scale realtime apps. Spotinst Functions - Our SaaS optimization platform delivers significant cost reduction for AWS and GCE, while maintaining high availability and performance. StdLib - StdLib Service Creation, Deployment, and Management Tools Syncano - A serverless application platform to build powerful realtime apps more efficiently. Weblab - Microservices at your fingertips Webtask - Webtasks is a simple, lightweight, and secure way of running isolated backend code that removed or reduces the need for a backend. Zeit Now - Now – Realtime Global Deployments Hybrid Platforms Galactic Fog Gestalt - Build Future-Proof Applications. Simplify integration. Run applications anywhere. Adapt to changes instantly. Nuclio - High-Performance Serverless event and data processing platform Binaris - A high-performance serverless platform for interactive and real-time applications. Cloudboost - One Complete NoSQL Database Service for your app. Fn - The container native, cloud agnostic serverless platform. fx - fx is a tool to help you do Function as a Service with painless on your own servers LunchBadger - LunchBadger is a multi-cloud platform for microservices and serverless. Kubernetes-Native Platforms Fission - Fast Serverless Functions for Kubernetes Oracle Application Container Cloud - Oracle is a computer technology corporation developing and marketing computer hardware systems and enterprise software products. Riff - riff is for functions Funktion - a CLI tool for working with funktion Kubeless - Kubernetes Native Serverless Framework OpenFAAS - OpenFaaS - Serverless Functions Made Simple for Docker & Kubernetes OpenLambda - An open source serverless computing platform PubNub - The PubNub Data Stream Network enables mobile and web developers to build and scale realtime apps. Observability & Analysis Monitoring CNCF Member Products/Projects Prometheus - The Prometheus monitoring system and time series database. Amazon CloudWatch - Amazon Web Services provides information technology infrastructure services to businesses in the form of web services. Datadog - Datadog offers a cloud-scale monitoring service. Dynatrace - Dynatrace transform how Web and non-Web business-critical applications are monitored, managed, and optimized throughout their lifecycle. Google Stackdriver - Google is a multinational corporation that is specialized in internet-related services and products. Grafana - The tool for beautiful monitoring and metric analytics & dashboards for Graphite, InfluxDB & Prometheus & More InfluxDB - Scalable datastore for metrics, events, and real-time analytics Instana - Instana is an APM solution that automatically monitors dynamic modern apps. Lighstep - LightStep's mission is to cut through the scale and complexity of today's software to help organizations stay in control of their systems. Log Analytics - Microsoft is a software corporation that develops, manufactures, licenses, supports, and sells a range of software products and services. Netsil - Observability and Monitoring for Modern Cloud Applications SignalFX - Advanced monitoring platform for modern applications Snap - A powerful open telemetry framework.Easily collect, process, and publish telemetry data at scale. SysDig - Linux system exploration and troubleshooting tool with first class support for containers Weave Cloud - Weaveworks provides a simple and consistent way to connect and manage containers and microservices. Non-CNCF Member Products/Projects AppDynamics - AppDynamics develops application performance management (APM) solutions that deliver problem resolution for highly distributed applications. AppNeta - AppNeta is the only app performance monitoring company with solutions for apps you develop, SaaS apps you use & networks that deliver them. Axibase - Purpose-built solution for analyzing and reporting on massive volumes of time-series data collected at high frequency. Catchpoint Systems - Catchpoint is a leading digital experience intelligence company. Centreon - Centreon is a network, system, applicative supervision and monitoring tool. Cobe - Cobe delivers an aggregated view of every element related to your business. CoScale - Full stack performance monitoring. Built for container and microservices applications. Powered by anomaly detection. Graphite - A highly scalable real-time graphing system Honeybadger - Exception, uptime, and performance monit. Icinga - Monitoring as code IronDB - Realtime Monitoring and Analytics Librato - Real time operations analytics for metrics from any source Meros - Meros is creating enterprise monitoring and management tools for Docker Nagios - The Industry Standard In IT Infrastructure Monitoring. New Relic - New Relic is a leading digital intelligence company, delivering full-stack visibility and analytics to enterprises around the world. NodeSource - Building products focused on Node.js security and performance for the Enterprise. OpBeat - Opbeat is joining forces with Elastic. OpenTSDB - A scalable, distributed Time Series Database. OpsClarity - Intelligent Monitoring for Modern Applications and Data Infrastructure Outlyer - Infrastructure monitoring platform made for DevOps and microservices. Rocana - Rocana is a San Francisco, CA-based provider of root cause analysis software company Sensu - Monitoring for today's infrastructure. Sentry - Sentry is a cross-platform crash reporting and aggregation platform. Server Density - Monitoring agent for Server Density (Linux, FreeBSD and OS X) StackRox - StackRox delivers the industry's only adaptive threat protection for containers. StackState - The market-leading Algorithmic IT Operations platform Tingyun - Observability and Analysis, Monitoring Wavefront - Wavefront is a hosted platform for ingesting, storing, visualizing and alerting on time series data. Zabbix - The Ultimate Enterprise - class Monitoring Platform Logging Fluentd - Fluentd: Unified Logging Layer (project under CNCF) Humio - Log everything, answer anything Splunk - Splunk provides operational intelligence software that monitors, reports, and analyzes real-time machine data. Elastic - Open Source, Distributed, RESTful Search Engine. Graylog - Free and open source log management Loggly - Loggly parses your log files, shows you the code in GitHub which caused the log errors. 10,000+ customers, including 1/3 of the Fortune 500. Logz - Logz.io is an enterprise-grade ELK as a service with alerts, unlimited scalability, and predictive fault detection. Loom Systems - Predict & Prevent Problems in the Digital Business Sematext - Sematext is a Search and Big Data analytics products and services company. Sumo Logic - Sumo Logic, a log management and analytics service, transforms big data into sources of operations, security and compliance intelligence. Tracing Jaeger - CNCF Jaeger, a Distributed Tracing System OpenTracing - OpenTracing API for Go Spring Cloud Sleuth - Distributed tracing for spring cloud Appdash - Application tracing system for Go, based on Google's Dapper. SkyWalking - A distributed tracing system, and APM ( Application Performance Monitoring ) Zipkin - Zipkin is a distributed tracing system Contribute Contributions are most welcome, please adhere to the contribution guidelines. ⬆ back to top License This work is licensed under a Creative Commons Attribution 4.0 International License.

优秀的个人博客,低调大师

hadoop 原生MapReduce 实现数据连接

其实很简单,输入两个文件,一个作为基础数据(学生信息文件),一个是分数信息文件。 学生信息文件:存放学生数据:包括学号,学生名称 分数信息数据:存放学生的分数信息:包括学号,学科,分数。 我们将通过M/R实现根据学号,进行数据关联,最终结果为:学生名称,学科,分数。 模拟数据 学生数据 [hadoop@hadoop11 student_data]$ cat students.txt 1 Randy 2 Tom 3 kitty 4 Lucy 5 Lily 6 Bruce 7 King 8 Jay 9 Melody 10 Kimy ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// 分数数据 [hadoop@hadoop11 student_data]$ cat scores.txt 1 English 89 2 English 77 3 English 54 4 English 98 5 English 83 6 English 99 7 English 30 8 English 76 9 English 56 10 English 88 1 Math 79 2 Math 37 3 Math 65 4 Math 88 5 Math 89 6 Math 59 7 Math 60 8 Math 86 9 Math 56 10 Math 68 1 China 89 2 China 67 3 China 84 4 China 68 5 China 43 6 China 89 7 China 70 8 China 96 9 China 56 10 China 78 /////////////////////////////////////////////////////////////////////////////////////////////////////// 实现 1)两个文本解析器,分别解析两个文本文件。 本文转自 randy_shandong 51CTO博客,原文链接:http://blog.51cto.com/dba10g/1565697,如需转载请自行联系原作者

资源下载

更多资源
Mario

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长,特征是大鼻子、头戴帽子、身穿背带裤,还留着胡子。与他的双胞胎兄弟路易基一起,长年担任任天堂的招牌角色。

腾讯云软件源

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题,腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构,目前腾讯云软件源站支持公网访问和内网访问。

Sublime Text

Sublime Text

Sublime Text具有漂亮的用户界面和强大的功能,例如代码缩略图,Python的插件,代码段等。还可自定义键绑定,菜单和工具栏。Sublime Text 的主要功能包括:拼写检查,书签,完整的 Python API , Goto 功能,即时项目切换,多选择,多窗口等等。Sublime Text 是一个跨平台的编辑器,同时支持Windows、Linux、Mac OS X等操作系统。

WebStorm

WebStorm

WebStorm 是jetbrains公司旗下一款JavaScript 开发工具。目前已经被广大中国JS开发者誉为“Web前端开发神器”、“最强大的HTML5编辑器”、“最智能的JavaScript IDE”等。与IntelliJ IDEA同源,继承了IntelliJ IDEA强大的JS部分的功能。

用户登录
用户注册