首页 文章 精选 留言 我的

精选列表

搜索[MLA 技术],共10000篇文章
优秀的个人博客,低调大师

技术分享 | Koa框架源码分析

一、Why Koa ? koa 是由 Express 原班人马打造的,致力于成为一个更小、更富有表现力、更健壮的 Web 框架。 使用 koa 编写 web 应用,通过组合不同的 generator,可以免除重复繁琐的回调函数嵌套, 并极大地提升错误处理的效率。koa 不在内核方法中绑定任何中间件, 它仅仅提供了一个轻量优雅的函数库,使得编写 Web 应用变得得心应手。 async await写法,让异步更加优雅。(代码是写给人看的,顺便让机器执行而已) 良好的抽象,简洁的api,简单、吊炸天的中间件机制。 强大的社区,GayHub上数不胜数的中间件。 二、小试牛刀之源码文件一览 npm上download下koa之后打开koa/lib文件夹可以看到四个文件,足以证明koa框架是多么简洁。(不过koa框架依赖了很多node_modules) application.js 入口文件,也是骨架文件,创建一个服务。 context.js app 的 context 对象, 传入中间件的上下文对象。 request.js app 的请求对象,包含请求相关的一些属性。 response.js app 的响应对象,包含响应相关的一些属性。 下面大致总的分为三章节来分享一下koa源码。 1、Hello Koa 首先看一下原生node和koa分别是如何创建一个server。 /** *原生Node创建server */ const http = require('http'); const server = http.createServer((req, res) => { res.statusCode = 200; res.setHeader('Content-Type', 'text/plain'); res.end('Hello World\n'); }); server.listen(3000); /** *koa创建server */ const Koa = require('koa'); const app = new Koa(); app.use(ctx => { ctx.body = 'Hello Koa'; }); app.listen(3000); 看一下源码中koa究竟都做了什么 //1.首先创建了一个Application的类,继承了Emitter类,然后暴露出去。 module.exports = class Application extends Emitter { constructor() { super(); } } //2.类下面有一个listen方法,创建http服务 listen(...args) { debug('listen'); const server = http.createServer(this.callback()); return server.listen(...args); } 在Hello Koa中,引入这个类,然后实例化这个类,在使用下面的listen即搭建起了一个简单的koa服务。 然后讲一下初始化koa的时候,这个类的下面有一个use方法,也就是我们添加中间件的方法。 use(fn) { //如果use里面传的不是方法,直接错误抛出去 if (typeof fn !== 'function') throw new TypeError('middleware must be a function!'); //如果是星号函数,用一个库转成async await方法(去兼容koa1的中间件) if (isGeneratorFunction(fn)) { //如果是generator函数,用convert转成async函数,然后执行。也是就是koa2能支持koa1的中间件。 deprecate('Support for generators will be removed in v3. ' + 'See the documentation for examples of how to convert old middleware ' + 'https://github.com/koajs/koa/blob/master/docs/migration.md'); fn = convert(fn); //转async await方法 } debug('use %s', fn._name || fn.name || '-'); this.middleware.push(fn); //koa把所有的中间件都放在一个数组里面 return this; } 如代码所示,koa把所有的中间件放在一个数组里面,然后koa2也是兼容generator函数,为了兼容koa1的中间件。 callback方法Koa处理请求的方法。 callback() { const fn = compose(this.middleware); //组装中间件 if (!this.listeners('error').length) this.on('error', this.onerror); const handleRequest = (req, res) => { const ctx = this.createContext(req, res); return this.handleRequest(ctx, fn); }; return handleRequest; } 接下来就是看一下callback中compose如何处理了中间价和中间件的执行机制 Koa中间件(middleware)实现探索 首先用一段简单的代码了解一下中间件执行顺序。 const Koa = require('koa') const app = new Koa() app.use(async function m1(ctx, next) { console.log('1') await next(); console.log('1 end') }) app.use(async function m2(ctx, next) { console.log('2') await next(); console.log('2 end') }) app.use(async function m3(ctx, next) { console.log('3') ctx.body = 'Hello World' console.log('3 end') }) app.listen(3000); // 打印的顺序是 // 1、2、3、3end、2end、1end koa通过compose方法把中间件封装成一个promise对象,然后递归执行。使中间件数组变成洋葱圈,能递归线性处理上下文。 下面是官方的配图: 我们主要看一下这个库: function compose (middleware) { return function (context, next) { let index = -1 return dispatch(0) function dispatch (i) { if (i <= index) return Promise.reject(new Error('next() called multiple times')) index = i let fn = middleware[i] if (i === middleware.length) fn = next if (!fn) return Promise.resolve() try { return Promise.resolve(fn(context, dispatch.bind(null, i + 1))); /** dispatch.bind(null, i + 1) 利用了js的bind继承 其实可以写成 function() { dispatch(i + 1) } 一个等待被执行的函数,然后用next()去执行下一个中间件。 */ } catch (err) { return Promise.reject(err) } } } } //通过调用第一个函数,层层递归调用,然后达到洋葱模型 //大白话讲一下就是,compose方法把所有的中间的执行的时候,把第一个中间件封装成了一个psomise对象执行,然后同时在中间件里面写next()方法去递归执行下一个中间件。 这里的 Promise.resolve(fn(..)) 帮助我们异步执行的中间件函数,这里的next函数就解释了为什么Koa的中间件调用是递归执行的,它递归调用了 dispatch 函数来遍历数组中的,同时,所有的中间件函数享有同一个 ctx。 然后我根据compose处理后的中间件,大致中间件执行中理想的样子如图: (async function(){ // middleware1 console.log('1'); await (async function(){ // middleware2 console.log('2'); await ( async function(){ // middleware3 ... console.log('3'); console.log('3 end'); // ...middleware3 }) console.log('2 end') // ...middleware2 }) console.log('1 end'); // ...middleware1 })() 组装成promise的中间件然后赋值到fn上去执行。 fn(ctx).then(handleResponse).catch(onerror)。 来看看这一句,fn 之前说过了,是所有的中间件函数的 “集合”, 用这一个中间件来表示整个处理过程。 同时 fn 也是一个 async 函数,执行结果返回一个 promise 对象, 同时 handleResponse 作为其 resolved 函数,onerror 是 rejected 函数。 也就是说koa中捕捉错误不是用try catch 而且是用promise中的catch。 3、 context的来源 context 使用node原生的 http 的监听回调函数中的 req res 来进行进一步的封装,意味着对于每一个 http 请求,koa都会创建一个 context 并共享给所有的全局中间件使用,当所有的中间件执行完过后,会将最后要返回的所有数据统一再交还给 res 进行返回,所以我们在每一个中间件中才能够从 ctx 中取得自己所需要的 req 中的数据进行处理,最后 ctx 再把要返回的 body 给原生的 res 进行返回。 每一个请求都有唯一一个 context 对象,所有的关于请求和返回的东西都统一放在里面 createContext 方法将 req res 进一步封装。 const context = Object.create(this.context); // 创建一个对象,使之拥有context的原型方法,后面以此类推 const request = context.request = Object.create(this.request); const response = context.response = Object.create(this.response); context.app = request.app = response.app = this; context.req = request.req = response.req = req; context.res = request.res = response.res = res; request.ctx = response.ctx = context; request.response = response; response.request = request; 本着一个请求一个context的原则,context 必须是作为一个临时对象而存在,所有的东西都必须封进一个对象中,因此 app req res 三个属性就此诞生。 三、讲到最后 本文主要围绕“koa2的骨架”、“剖析koa中间件”、“ctx的来源”这三个模块来阐述了koa的原理和实现。并根据思路流程渐进梳理讲解了一些细节思路和比较关键的内容点,以及通过展示部分关键代码讲述了koa中间件的执行原理。文中肯定会有一些不够严谨的思考和错误,欢迎大家指正,有兴趣欢迎一起探讨和改进~ 最后,感谢您的阅读! 原文发布时间为:2018年06月24日 原文作者: 掘金 本文来源: 掘金 如需转载请联系原作者

优秀的个人博客,低调大师

《Cloud Native 》云原生技术汇总

An awesome & curated list of best applications and tools for Cloud Native. This Awesome Repository is highly inspired from cncf's landscape & Awesome. Items marked with Open-Source Software are open-source software. Items marked with Freeware are free. --> Cloud Native Services Components Cloud Provisioning Runtime Orchestration & Management Application Definition & Development Platform Serverless Observability & Analysis Cloud Public - A public cloud is a pool of virtual resources—developed from hardware owned and managed by a third-party company—that is automatically provisioned and allocated among multiple clients through a self-service interface. Alibaba Cloud - Alibaba Cloud develops highly scalable cloud computing and data management services. Amazon Web Services - Amazon Web Services provides information technology infrastructure services to businesses in the form of web services. Azure Cloud - Microsoft is a software corporation that develops, manufactures, licenses, supports, and sells a range of software products and services. Baidu Cloud - Baidu is a Chinese website and search engine that enables individuals to obtain information and find what they need. DigitalOcean - DigitalOcean is an IaaS company that delivers a seamless way for developers and businesses to deploy and scale any application in the cloud. Fujitsu K5 - Fujitsu provides information technology and communications solutions. Google Cloud - Google is a multinational corporation that is specialized in internet-related services and products. Huawei Cloud - Huawei Technologies provides infrastructure application software and devices with wireline, wireless, and IP technologies. IBM Cloud - IBM is an IT technology and consulting firm providing computer hardware, software, and infrastructure and hosting services. Oracle Cloud - Oracle is a computer technology corporation developing and marketing computer hardware systems and enterprise software products. Joyent Cloud - Your Cloud, Your Way Packet Cloud - Packet is a bare metal cloud built for developers. 8 minute deploys, no hypervisor, & full automation support from 15 global data centers. Tencent Cloud - Tencent is a Chinese internet service portal offering value-added internet, mobile, telecom, and online advertising services. Citrix Cloud - Move Faster, Work Better, Lower IT Costs Private - The private cloud is defined as computing services offered either over the Internet or a private internal network and only to select users instead of the general public. Also called an internal or corporate cloud, private cloud computing gives businesses many of the benefits of a public cloud - including self-service, scalability, and elasticity - with the additional control and customization available from dedicated resources over a computing infrastructure hosted on-premises. Openstack - Repository containing OpenStack repositories Scaleway - Scaleway is the world's first Cloud Computing IaaS platform Foreman - an application that automates the lifecycle of servers Digital Bebar - Digital Rebar Provision is a simple but powerful Golang executable that provides a complete API-driven DHCP/PXE/TFTP provisioning system. MAAS - Official MAAS repository mirror. (Do not submit pull requests or bugs here; use Launchpad instead.) VMware - VMware is a software company providing cloud and virtualization services. Hybrid - A hybrid cloud is a computing environment that combines a public cloud and a private cloud by allowing data and applications to be shared between them. Ensono - Complete hybrid IT services – from cloud to mainframe. Operate for today. Optimize for tomorrow. Dellemc - Dell EMC is a powerful part of Dell Technologies' commitment to your transformation Hpe - Hybrid Cloud Solutions Scalr - The Hybrid Cloud Management Platform IBM Z hybrid cloud Rackspace Hybrid Cloud Microsoft Hybrid Cloud VMware Hybrid Cloud AWS Hybrid Cloud Provisioning Container Registries Container Registry is a private Docker repository that works with popular continuous delivery systems. - [ECR](https://aws.amazon.com/cn/ecr/) - Amazon Elastic Container Registry (ECR) is a secure, fully-managed Docker container registry that makes it easy for developers to store, manage, and deploy Docker container images. - [Azure Registry](https://azure.microsoft.com/en-us/services/container-registry/) - Manage a Docker private registry as a first-class Azure resource. - [Codefresh Registry](https://codefresh.io/registry-beta/) - Codefresh is a Docker-native CI/CD platform.Instantly build, test and deploy Docker images. - [Docker Registry](https://docs.docker.com/registry/) - Docker Trusted Registry (DTR) is a commercial product that enables complete image management workflow, featuring LDAP integration, image signing, security scanning, and integration with Universal Control Plane. DTR is offered as an add-on to Docker Enterprise subscriptions of Standard or higher. - [Google Container Registry](https://cloud.google.com/container-registry/) - High-speed, private Docker image storage on Google Cloud Platform - [Harbor](http://vmware.github.io/harbor/) - An Enterprise-class Container Registry Server based on Docker Distribution. - [JFrog Artifactory](https://jfrog.com/artifactory/) - Enterprise Universal Artifact Manager. - [Portus](http://port.us.org/) - Portus is an open source authorization service and user interface for the next generation Docker Registry. - [Project Atomic](https://www.projectatomic.io/) - Atomic Host provides immutable infrastructure for deploying to hundreds or thousands of servers in your private or public cloud. - [QUAY Enterprise](https://coreos.com/quay-enterprise/) - One container registry for your entire enterprise. Host Management & Tooling Host management tool - [Ansible](https://www.ansible.com/) - Ansible is designed around the way people work and the way people work together. - [Chef](https://www.chef.io) - Ship better software, faster.Enable collaboration and continuous automation across your infrastructure, applications, and compliance for all your apps and infrastructure. - [LniuxKit](https://github.com/linuxkit/linuxkit) - A toolkit for building secure, portable and lean operating systems for containers - [CFEngine](https://cfengine.com/) - CFEngine Community - [Puppet](https://puppet.com/) - Server automation framework and application - [Rundeck](https://www.rundeck.com/) - Enable Self-Service Operations: Give specific users access to your existing tools, services, and scripts - [Saltstack](https://saltstack.com/) - Intelligent automation for a software-defined world Infrastructure Automation Infrastructure automation makes servers and VM management more flexible, efficient, and scalable by converting management tasks and policy into code. - [AWS CloudFormation](https://aws.amazon.com/cn/cloudformation/) - [HOSH](https://bosh.io) - BOSH is an open source tool for release engineering, deployment, lifecycle management, and monitoring of distributed systems. - [Helm](https://helm.sh/) - Helm is the best way to find, share, and use software built for Kubernetes. - [Infrakit](https://github.com/docker/infrakit) - A toolkit for creating and managing declarative, self-healing infrastructure. - [Juju](https://jujucharms.com/) - Juju is an open source application modelling tool. Deploy, configure, scale and operate your software on public and private clouds. - [Cloud Coreo](https://www.cloudcoreo.com/) - A Platform for Modern Cloud Teams - [Cloudify](https://cloudify.co/) - Radically Simplifying Multi-Cloud Orchestration - [Kubicorn](http://kubicorn.io/) - Create, manage, snapshot, and scale Kubernetes infrastructure in the public cloud. - [ManageIQ](http://manageiq.org/) - Discover, Optimize, and Control your Hybrid IT - [Terraform](https://www.terraform.io/) - Write, Plan, and Create Infrastructure as Code Key Management Key management is the name of management of cryptographic keys in a cryptosystem. Secure Images Secure your images so that you maintain control of how they are displayed on the Internet. - [Notary](https://github.com/theupdateframework/notary) - Notary is a project that allows anyone to have trust over arbitrary collections of data - [TUF](https://theupdateframework.github.io/) - A framework for securing software update systems - [Aqua](https://www.aquasec.com/) - The Aqua Container Security Platform provides development-to-production lifecycle controls for securing containerized applications that run on-premises or in the cloud, on Windows or Linux, supporting multiple orchestration environments. - [Clair](https://coreos.com/clair) - Clair is an open source project for the static analysis of vulnerabilities in appc and docker containers. - [OpenSCAP](https://www.open-scap.org/) - Discover a wide array of tools for managing system security and standards compliance. - [Twistlock](https://www.twistlock.com/) - Container Security for Docker, Kubernetes and Beyond - [Anchore](https://anchore.com/) - An open source complete solution for compliance, certification, security scanning, and auditing of public and private container images. - [anchore.io](https://anchore.io/) - Discover, Analyze, and Certify Container Images. - [Black Duck](https://www.blackducksoftware.com/) - Complete Visibility. Automated Control. - [NeuVector](https://neuvector.com/) - Continuous Network Security for Kubernetes Containers - [Sonatype Nexus](https://www.sonatype.com/) - The world's best way to organize, store, and distribute software components. Runtime Cloud-Native Network Network Segmentation and Policy,SDN & APIs (eg CNI, libnetwork) Incubating CNCF Projects CNI - Container Network Interface - networking for Linux containers CNCF Member Products/Projects Aporeto - Cloud Native Security for Containers and Microservices Cannl - Policy based networking for cloud native applications Contiv - Container networking for various use cases Flannel - flannel is a network fabric for containers, designed for Kubernetes NSX - VMware is a software company providing cloud and virtualization services. Open vSwitch - Open vSwitch is a multilayer software switch licensed under the open source Apache 2 license. OpenContrial - An open-source network virtualization platform for the cloud. Project Calico - Cloud native application connectivity and network policy Weave Net - Simple, resilient multi-host Docker networking and more. Non-CNCF Member Products/Projects Aviatrix - The company develops software that enables enterprises to build hybrid clouds by easily Big Switch Networks - Big Switch Networks is the Next-Generation Data Center Networking Company, designing intelligent, agile and flexible networks Cilium - HTTP, gRPC, and Kafka Aware Security and Networking for Containers with BPF and XDP Cumulus - Cumulus Networks, a software company, designs, and sells Linux operating systems for networking hardware. GuardiCoreCentra - GuardiCore provides network security solutions for software defined data centers. MidoNet - MidoNet is an Open Source network virtualization system for Openstack clouds Nuage Networks - Nuage Networks Fundamentals: Software Defined Networking for the Datacenter and Beyond. Plumgrid - PLUMgrid is involved in virtual networking and SDN/NFV to deliver cloud infrastructure solutions that transform businesses. Romana - The Romana Project - Installation scripts, documentation, issue tracker and wiki. Start here. SnapRoute - SnapRoute is an open networking stack company. Cloud-Native Storage Volume Drivers/Plugins,Local Storage Management,Remote Storage Access Sandbox CNCF Projects Rook - File, Block, and Object Storage Services for your Cloud-Native Environments CNCF Member Products/Projects Ceph - Ceph is a unified, distributed storage system designed for excellent performance, reliability and scalability. Container Storage Interface - Container Storage Interface (CSI) Specification. Dell EMC - IT and Workforce Transformation. Made real every day. Diamanti - Diamanti is the first container platform with plug and play network and persistent storage that seamlessly integrates the most widely adopted software stack - standard open source Kubernetes and Docker - so there is no vendor lock-in. QoS on network and storage maximizes container density. Gluster - Gluster is free and open source softeare scalable network filesystem. Hatchway - Persistent Storage for Cloud Native Applications Kasten - Kasten is on a mission to dramatically simplify operational management of stateful cloud-native applications. Manta - Structural variant and indel caller for mapped sequencing data Minio - Minio is a high performance distributed object storage server, designed for large-scale private cloud infrastructure. Minio is widely deployed across the world with over 64.2M+ docker pulls. NetApp - NetApp HCI. All New and Available Now. OpenEBS - OpenEBS is an open source storage platform that provides persistent and containerized block storage for DevOps and container environments. Portworx - The Solution for Stateful Containers in Production. Designed for DevOps. Rex-Ray - REX-Ray is an open source, storage management solution designed to support container runtimes such as Docker and Mesos. StorageOS - Enterprise persistent storage for containers and the cloud. Non-CNCF Member Products/Projects Datera - Datera is an application-driven data infrastructure company. Hedving - Modern storage for the modern business. Infinit - The Elle coroutine-based asynchronous C++ development framework. LeoFS - The LeoFS Storage System OpenIO - OpenIO Software Defined Storage Pure Storage - Pure Storage is an all-flash enterprise storage company that enables broad deployment of flash in data centers. Quobyte - Data Center File System. Fast and Reliable Software Storage Robin Systems - Data-Centric Compute and Storage Containerization Infrastructure Software Sheepdog - Distributed Storage System for QEMU Springpath - Springpath is hyperconvergence software that turns standard servers of choice into a single pool of compute and storage resources. Swift - OpenStack Storage (Swift) Container Runtime The new CF Container Runtime gives you more granular control and management of containers with Kubernetes. Incubating CNCF Projects containerd rkt - rkt is a pod-native container engine for Linux. It is composable, secure, and built on standards. CNCF Member Products/Projects CRI-O - Open Container Initiative-based implementation of Kubernetes Container Runtime Interface Intel Clear Containers - OCI (Open Containers Initiative) compatible runtime using Virtual Machines Ixd - Daemon based on liblxc offering a REST API to manage containers Pouch - Pouch is an open-source project created to promote the container technology movement. runc - CLI tool for spawning and running containers according to the OCI specification SmartOS - Converged Container and Virtual Machine HypervisorNon-CNCF Member Products/Projects Kata Containers - Kata Containers runtimes RunV - Hypervisor-based Runtime for OCI Singularity - Singularity: Application containers for Linux Orchestration & Management Scheduling & Orchestration Graduated CNCF Projects Kubernetes - Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications CNCF Member Products/Projects ECS - Amazon Web Services provides information technology infrastructure services to businesses in the form of web services. Docker Swarm - Swarm: a Docker-native clustering system Microsoft Azure Service Fabric - Service Fabric is a distributed systems platform for packaging, deploying, and managing stateless and stateful distributed applications and containers at large scale. Non-CNCF Member Products/Projects Mesos - Mirror of Apache Mesos Nomad - Nomad is a flexible, enterprise-grade cluster scheduler designed to easily integrate into existing workflows. Coordination & Service Discovery Incubating CNCF Projects CoreDNS - CoreDNS is a DNS server that chains plugins. CNCF Member Products/Projects ContainerPilot - A service for autodiscovery and configuration of applications running in containers etcD - Distributed reliable key-value store for the most critical data of a distributed system VMware Haret - A strongly consistent distributed coordination system, built using proven protocols & implemented in Rust. Non-CNCF Member Products/Projects Apache Zookeeper - Apache ZooKeeper is an effort to develop and maintain an open-source server which enables highly reliable distributed coordination. Consul - Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure. Eureka - AWS Service registry for resilient mid-tier load balancing and failover. SkyDNS - DNS service discovery for etcd SmartStack - A transparent service discovery framework for connecting an SOA Service Management Envoy - C++ front/service proxy gRPC - The C based gRPC (C++, Python, Ruby, Objective-C, PHP, C#) Linkerd - Production-grade feature-rich service mesh for any platform 3Scale - 3scale api gateway reloaded Ambassador - open source Kubernetes-native API gateway for microservices built on the Envoy Proxy Avi Networks - Avi Networks is a Silicon Valley startup with proven track record in building virtualization, networking and software solutions. Conduit - Ultralight service mesh for Kubernetes F5 - F5 Networks provides application delivery networking technology that optimizes the delivery of network-based applications. Heptio Contour - Contour is a Kubernetes ingress controller for Lyft's Envoy proxy. Kong - ? The Microservice API Gateway NGINX - application delivery for the modern web Open Service Broker API - Open Service Broker API Specification Turbine Labs - Turbine Labs Apache Thrift - Mirror of Apache Thrift Avro - Apache Avro Backplane - A service that unifies discovery, routing, and load balancing for web servers written in any language, running in any cloud or datacenter. HAProxy - The Reliable, High Performance TCP/HTTP Load Balancer Hystrix - Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable. Istio - An open platform to connect, manage, and secure microservices. Netflix Zuul - Zuul is a gateway service that provides dynamic routing, monitoring, resiliency, security, and more. Open Policy Agent (OPA) - An open source project to policy-enable your service. Ribbon - Ribbon is a Inter Process Communication (remote procedure calls) library with built in software load balancers. Traefik - Træfik, a modern reverse proxy Vamp - Vamp - canary releasing and autoscaling for microservice systems. Application Definition & Development Database & Data Warehouse Incubating CNCF Projects Vitess - Vitess is a database clustering system for horizontal scaling of MySQL. CNCF Member Products/Projects Cloudhbase - Lightweight, embedded, syncable NoSQL database engine for iOS (and Mac!) apps. IBM DB2 - IBM is an IT technology and consulting firm providing computer hardware, software, and infrastructure and hosting services. Iguazio - iguazio's Continuous Analytics Data Platform has redesigned the data stack to accelerate performance in big data, IoT and cloud-native apps. Infinispan - Infinispan is an open source data grid platform and highly scalable NoSQL cloud data store. Microsoft SQL Server - Microsoft is a software corporation that develops, manufactures, licenses, supports, and sells a range of software products and services. MySQL - MySQL Server, the world's most popular open source database, and MySQL Cluster, a real-time, open source transactional database. Oracle - Oracle is a computer technology corporation developing and marketing computer hardware systems and enterprise software products. RethinkDB - The open-source database for the realtime web. SQL Data Warehouse - Microsoft is a software corporation that develops, manufactures, licenses, supports, and sells a range of software products and services. YugaByte DB - YugaByteDB is a transactional, high-performance database for building distributed cloud services. It currently supports Redis API (as a true DB) and Cassandra API, with SQL coming very soon. Non-CNCF Member Products/Projects ArangoDB - ? ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions. BigchainDB - Meet BigchainDB. The blockchain database. CarbonData - Mirror of Apache CarbonData Cassandra - Mirror of Apache Cassandra CockroachDB - CockroachDB - the open source, cloud-native SQL database. Crate.io - CrateDB is a distributed SQL database that makes it simple to store and analyze massive amounts of machine data in real-time. Druid - Column oriented distributed data store ideal for powering interactive applications. Hadoop - Mirror of Apache Hadoop MariaDB - MariaDB server is a community developed fork of MySQL server. Started by core members of the original MySQL team, MariaDB actively works with outside developers to deliver the most featureful, stable, and sanely licensed open SQL server in the industry. MemSQL - A real-time data warehouse you can run everywhere MongoDB - MongoDB is a document database with the scalability and flexibility that you want with the querying and indexing that you need NomsDB - The versioned, forkable, syncable database OrientDB - OrientDB is the most versatile DBMS supporting Graph, Document, Reactive, Full-Text, Geospatial and Key-Value models in one Multi-Model product. OrientDB can run distributed (Multi-Master), supports SQL, ACID Transactions, Full-Text indexing and Reactive Queries. OrientDB Community Edition is Open Source using a liberal Apache 2 license. Pachyderm - Reproducible Data Science at Scale! Pilosa - Pilosa is an open source, distributed bitmap index that dramatically accelerates queries across multiple, massive data sets. PostgreSQL - PostgreSQL is a powerful, open source object-relational database system. Presto - Distributed SQL query engine for big data Qubole - Qubole delivers a Self-Service Platform for Big Data Analytics built on Amazon, Microsoft, Google and Oracle Clouds. Redis - Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, HyperLogLogs, Bitmaps. Scylla - NoSQL data store using the seastar framework, compatible with Apache Cassandra Snowflake - Snowflake is the only data warehouse built for the cloud. Software AG - Software AG provides business process management, data management, and consulting services worldwide. Starburst - Starburst (www.starburstdata.com) is the enterprise Presto company offering an SQL-on-Anything analytics platform. TiDB - TiDB is a distributed HTAP database compatible with the MySQL protocol. Vertica - Vertica Systems develops data management solutions for storing databases and allowing clients to conduct real-time and ad hoc queries. Streaming Incubating CNCF Projects NATS - High-Performance server for NATS, the cloud native messaging system. CNCF Member Products/Projects Amazon Kinesis - Amazon Web Services provides information technology infrastructure services to businesses in the form of web services. CloudEvents - CloudEvents Specification Google Cloud Dataflow - Google is a multinational corporation that is specialized in internet-related services and products. Heron - Heron is a realtime, distributed, fault-tolerant stream processing engine from Twitter Non-CNCF Member Products/Projects Apache Apex - Mirror of Apache Apex core Apache NiFi - Mirror of Apache NiFi Apache RocketMQ - Mirror of Apache RocketMQ Apache Spark - Mirror of Apache Spark Apache Storm - Mirror of Apache Storm Flink - Mirror of Apache Flink Kafka - Mirror of Apache Kafka Pulsar - Pulsar - distributed pub-sub messaging system RabbitMQ - RabbitMQ is the most widely deployed open source message broker. StreamSets - StreamSets DataCollector - Continuous big data ingest infrastructure. Source Code Management GitHub - GitHub is a web-based Git repository hosting service offering distributed revision control and source code management functionality of Git. GitLab - GitLab CE | Please open new issues in our issue tracker on GitLab.com Visual Studio Team Services - Microsoft is a software corporation that develops, manufactures, licenses, supports, and sells a range of software products and services. Bitbucket - Atlassian provides collaboration software for teams with products including JIRA, Confluence, HipChat, Bitbucket, and Stash. Application Definition Bitnami - Loved by Devs, Trusted by Ops. Easy to use cloud images, containers, and VMs that work on any platform Docker Compose - Define and run multi-container applications with Docker Habitat - Modern applications with built-in automation OpenAPI - The OpenAPI Specification Repository Telepresence - Local development against a remote Kubernetes or OpenShift cluster Apache Brooklyn - Apache Brooklyn KubeVirt - A virtualization API and runtime add-on for Kubernetes in order to define and manage virtual machines. Packer - Packer is a tool for creating identical machine images for multiple platforms from a single source configuration. CI & CD Continuous integration and continuous delivery are two approaches to software development that are designed to improve code quality and enable rapid delivery and deployment of code. CNCF Member Products/Projects Argo - Get stuff done with container-native workflows for Kubernetes. Cloud 66 Skycap - Ops tools for Devs. Build, deliver, deploy and manage any applications on any cloud or server. Cloudbees - CloudBees offers CloudBees Jenkins Enterprise, an enterprise-grade continuous delivery platform powered by Jenkins. Codefresh - Codefresh is a continuous delivery and collaboration platform for containers and microservices. Codeship - CloudBees offers CloudBees Jenkins Enterprise, an enterprise-grade continuous delivery platform powered by Jenkins. Concourse - BOSH release and development workspace for Concourse ContainerOps - DevOps Orchestration Platform Habitus - A Build Flow Tool for Docker Runner - GitLab Runner is the open source project that is used to run your jobs and send the results back to GitLab. Weave Flux - A tool for deploying container images to Kubernetes services Wercker - The Wercker CLI can be used to execute pipelines locally for both local development and easy introspection. Non-CNCF Member Products/Projects Appveyor - Appveyor Systems Inc. aim is to give powerful continuous integration and deployment tools to every .NET developer. Bamboo - Atlassian provides collaboration software for teams with products including JIRA, Confluence, HipChat, Bitbucket, and Stash. BuddyBuild - Buddybuild is a Vancouver-based app tools company focused on continuous integration and debugging tools. Buildkite - The Buildkite Agent is an open-source toolkit written in Golang for securely running build jobs on any device or network CircleCI - CircleCI provides software teams the confidence to build, test, and deploy across numerous platforms. Distelli - True Continuous Delivery from Source Control to Servers. Drone - Drone is a Continuous Delivery platform built on Docker, written in Go Jenkins - Build great things at any scale Octopus Deploy - Octopus Deploy is a user-friendly release management OpenStack Zuul CI - The Gatekeeper, or a project gating system Semaphore - Hosted continuous integration and deployment service Shippable - Shippable helps companies ship code faster by giving them a powerful continuous integration platform built natively on Docker. Solano Labs - Continuous Integration & Deployment Spinnaker - Spinnaker is an open source, multi-cloud continuous delivery platform for releasing software changes with high velocity and confidence. Travis - The Ember web client for Travis CI XL Deploy - XebiaLabs develops enterprise-scale Continuous Delivery and DevOps software. Platform Certified Kubernetes - Distribution Apprenda Kismatic Enterprise Toolkit (KET) Appscode Pharmer Caicloud Compass Canonical Distribution of Kubernetes Cloud Foundry Container Runtime CoreOS bootkube DaoCloud Enterprise Diamanti Converged Container Infrastructure Docker EE/CE Ghostcloud EcOS Giant Swarm Managed Kubernetes Google Kube-Up Heptio Quickstart for Kubernetes IBM Cloud Private inwinSTACK kube-ansible Joyent Triton for Kubernetes kube-spawn Kublr Loodse Kubermatic Mesosphere Kubernetes on DC/OS Mirantis Cloud Platform Netease Container Service Dedicated OpenShift Oracle Linux Container Services Oracle Terraform Kubernetes Installer Pivotal Container Service (PKS) Platform9 Managed Kubernetes QFusion Rancher Samsung Kraken StackPointCloud SUSE CaaS Platform Tectonic VMWare Pivotal Container Service (PKS) Weaveworks kubeadm WiseCloud Typhoon Certified Kubernetes - Platform Alauda EE Alibaba Cloud Container Service Azure (ACS) Engine Azure Container Service (AKS) Baidu Cloud Container Engine BoCloud BeyondcentContainer Cisco Container Platform EasyStack Kubernetes Service (EKS) eKing Cloud Container Platform Google Kubernetes Engine (GKE) HarmonyCloud Container Platform Hasura Huawei Cloud Container Engine (CCE) IBM Cloud Container Service Nirmata Managed Kubernetes Oracle Container Engine SAP Certified Gardener Tencent Cloud Container Service (CCS) TenxCloud Container Engine (TCE) ZTE TECS Non-Certified Kubernetes Amazon Elastic Container Service for Kubernetes (EKS) Cloud 66 Maestro Containership Gravitatonal Telekube Huawei FusionStage Navops Supergiant goPaddle Stratoscale Symphony PaaS & Container Service Apcera - Ericsson is a technology company that provides and operates telecommunications networks, television and video systems, and related services. Cloud Foundry Application Runtime - Cloud Foundry Application Runtime utilizes containers as part of its DNA, and has since before Docker popularized containers. The new CF Container Runtime gives you more granular control and management of containers with Kubernetes. Datawire - An early stage startup that's focused on making it easy for developers to build resilient microservices. Exoscale - Exoscale is the cloud hosting platform for SaaS companies, developers and systems administrators. Galactic Fog - Build Future-Proof Applications. Simplify integration. Run applications anywhere. Adapt to changes instantly. Heroku - Salesforce is a global cloud computing company that develops CRM solutions and provides business software on a subscription basis. Atomist - Atomist make microservice applications Easy and fun to build, through a cloud-based service. Clouber - Clouber is a provider of mCenter, an Application Modernization/Migration / Management platform across hybrid (public and private) clouds. Convox - Launch a Private Cloud in Minutes. Empire - A PaaS built on top of Amazon EC2 Container Service (ECS) Flynn - A next generation open source platform as a service (PaaS Hyper - Hyper.sh is a secure container cloud service. JHipster - Open Source application generator for creating Spring Boot + Angular projects in seconds! Kontena - The developer friendly container and micro services platform. Works on any cloud, easy to setup, simple to use. Lightbend - Lightbend (formerly Typesafe) is dedicated to helping developers build Reactive applications on the JVM. No Code - The best way to write secure and reliable applications. Write nothing; deploy nowhere. PaaSTA - An open, distributed platform as a service Platform.sh - Platform.sh is an automated, continuous-deployment high-availability cloud hosting solution Portainer - Simple management UI for Docker Scalingo - Scalingo a Docker Platform Service Transform your code as Docker container & run it on our cloud, making it instantly available & scalable. Tsuru - Open source, extensible and Docker-based Platform as a Service (PaaS). Serverless Security PureSec - PureSec is the world's leading Serverless Security Runtime Environment Snyk - Snyk is a security company helping to monitor app vulnerabilities. Libraries Python Lambda - A toolkit for developing and deploying serverless Python code in AWS Lambda. Tools Architect - ? cloud function signatures for http handlers, pubsub, scheduled functions and table triggers Dashbird - AWS Lambda monitoring & debugging platform. Serverless observability & troubleshooting. Serverless monitoring. IOpipe - IOpipe provides a toolbox for developing, monitoring, and operating serverless applications. Microcule - SDK and CLI for spawning streaming stateless HTTP microservices in multiple programming languages Node Lambda - Command line tool to locally run and deploy your node.js application to Amazon Lambda Stackery - Run serverless in production with Stackery's serverless operations console. Thundra - IT Alert and Notifications Management Frameworks AWS Chalice - Python Serverless Microframework for AWS SAM Local - AWS Serverless Application Model (AWS SAM) prescribes rules for expressing Serverless applications on AWS. Serverless - Serverless Framework – Build web, mobile and IoT applications with serverless architectures using AWS Lambda, Azure Functions, Google CloudFunctions & more! Spring Cloud Function - Pivotal is a software company that provides digital transformation technology and services. Apex - Build, deploy, and manage AWS Lambda functions with ease (with Go support!). Bustle Shep - A framework for building JavaScript Applications with AWS API Gateway and Lambda ClaudisJS - Deploy Node.js projects to AWS Lambda and API Gateway easily Dawson - A serverless web framework for Node.js on AWS (CloudFormation, CloudFront, API Gateway, Lambda) Flogo - Ultralight Edge Microservices Framework Gordon - λ Gordon is a tool to create, wire and deploy AWS Lambdas using CloudFormation GunIO Zappa - Serverless Python KappaIO - What precedes Lambda Mitoc Group Deep - Full-stack JavaScript Framework for Cloud-Native Web Applications (perfect for Serverless use cases) Sparta - A GO FRAMEWORK FOR AWS LAMBDA Platforms CNCF Member Products/Projects AWS Lambda - Amazon Web Services provides information technology infrastructure services to businesses in the form of web services. Azure Functions - Microsoft is a software corporation that develops, manufactures, licenses, supports, and sells a range of software products and services. Google Cloud Functions - https://cloud.google.com/functions/ IBM Cloud Functions - IBM is an IT technology and consulting firm providing computer hardware, software, and infrastructure and hosting services. Twilio Functions - Twilio is a cloud communication company that enables users to use standard web languages to build voice, VoIP, and SMS apps via a web API. Non-CNCF Member Products/Projects Algorithmia - Algorithmia is an open marketplace for algorithms, enabling developers to create tomorrows smart applications today. Apache OpenWhisk - Apache OpenWhisk is a serverless event-based programming service and an Apache Incubator project. AppScale - AppScale is an easy-to-manage serverless platform for building and running scalable web and mobile applications on any infrastructure. Clay - Rapid Prototyping for Developers Hyper Func - Hyper.sh is a secure container cloud service. Iron.io - Iron.io is a scalable cloud-based message queue and processing platform for building distributed cloud applications. Nano Lambda - Explore deploying code in lambda.Run server-side code with an API call. Overclock - Overclock Labs develops protocols, tools, and infrastructure to make foundational elements of the internet open, decentralized, and simple OVH Functions - OVH.com is an independent French company that offers web, dedicated, and cloud hosting solutions. PubNub Functions - The PubNub Data Stream Network enables mobile and web developers to build and scale realtime apps. Spotinst Functions - Our SaaS optimization platform delivers significant cost reduction for AWS and GCE, while maintaining high availability and performance. StdLib - StdLib Service Creation, Deployment, and Management Tools Syncano - A serverless application platform to build powerful realtime apps more efficiently. Weblab - Microservices at your fingertips Webtask - Webtasks is a simple, lightweight, and secure way of running isolated backend code that removed or reduces the need for a backend. Zeit Now - Now – Realtime Global Deployments Hybrid Platforms Galactic Fog Gestalt - Build Future-Proof Applications. Simplify integration. Run applications anywhere. Adapt to changes instantly. Nuclio - High-Performance Serverless event and data processing platform Binaris - A high-performance serverless platform for interactive and real-time applications. Cloudboost - One Complete NoSQL Database Service for your app. Fn - The container native, cloud agnostic serverless platform. fx - fx is a tool to help you do Function as a Service with painless on your own servers LunchBadger - LunchBadger is a multi-cloud platform for microservices and serverless. Kubernetes-Native Platforms Fission - Fast Serverless Functions for Kubernetes Oracle Application Container Cloud - Oracle is a computer technology corporation developing and marketing computer hardware systems and enterprise software products. Riff - riff is for functions Funktion - a CLI tool for working with funktion Kubeless - Kubernetes Native Serverless Framework OpenFAAS - OpenFaaS - Serverless Functions Made Simple for Docker & Kubernetes OpenLambda - An open source serverless computing platform PubNub - The PubNub Data Stream Network enables mobile and web developers to build and scale realtime apps. Observability & Analysis Monitoring CNCF Member Products/Projects Prometheus - The Prometheus monitoring system and time series database. Amazon CloudWatch - Amazon Web Services provides information technology infrastructure services to businesses in the form of web services. Datadog - Datadog offers a cloud-scale monitoring service. Dynatrace - Dynatrace transform how Web and non-Web business-critical applications are monitored, managed, and optimized throughout their lifecycle. Google Stackdriver - Google is a multinational corporation that is specialized in internet-related services and products. Grafana - The tool for beautiful monitoring and metric analytics & dashboards for Graphite, InfluxDB & Prometheus & More InfluxDB - Scalable datastore for metrics, events, and real-time analytics Instana - Instana is an APM solution that automatically monitors dynamic modern apps. Lighstep - LightStep's mission is to cut through the scale and complexity of today's software to help organizations stay in control of their systems. Log Analytics - Microsoft is a software corporation that develops, manufactures, licenses, supports, and sells a range of software products and services. Netsil - Observability and Monitoring for Modern Cloud Applications SignalFX - Advanced monitoring platform for modern applications Snap - A powerful open telemetry framework.Easily collect, process, and publish telemetry data at scale. SysDig - Linux system exploration and troubleshooting tool with first class support for containers Weave Cloud - Weaveworks provides a simple and consistent way to connect and manage containers and microservices. Non-CNCF Member Products/Projects AppDynamics - AppDynamics develops application performance management (APM) solutions that deliver problem resolution for highly distributed applications. AppNeta - AppNeta is the only app performance monitoring company with solutions for apps you develop, SaaS apps you use & networks that deliver them. Axibase - Purpose-built solution for analyzing and reporting on massive volumes of time-series data collected at high frequency. Catchpoint Systems - Catchpoint is a leading digital experience intelligence company. Centreon - Centreon is a network, system, applicative supervision and monitoring tool. Cobe - Cobe delivers an aggregated view of every element related to your business. CoScale - Full stack performance monitoring. Built for container and microservices applications. Powered by anomaly detection. Graphite - A highly scalable real-time graphing system Honeybadger - Exception, uptime, and performance monit. Icinga - Monitoring as code IronDB - Realtime Monitoring and Analytics Librato - Real time operations analytics for metrics from any source Meros - Meros is creating enterprise monitoring and management tools for Docker Nagios - The Industry Standard In IT Infrastructure Monitoring. New Relic - New Relic is a leading digital intelligence company, delivering full-stack visibility and analytics to enterprises around the world. NodeSource - Building products focused on Node.js security and performance for the Enterprise. OpBeat - Opbeat is joining forces with Elastic. OpenTSDB - A scalable, distributed Time Series Database. OpsClarity - Intelligent Monitoring for Modern Applications and Data Infrastructure Outlyer - Infrastructure monitoring platform made for DevOps and microservices. Rocana - Rocana is a San Francisco, CA-based provider of root cause analysis software company Sensu - Monitoring for today's infrastructure. Sentry - Sentry is a cross-platform crash reporting and aggregation platform. Server Density - Monitoring agent for Server Density (Linux, FreeBSD and OS X) StackRox - StackRox delivers the industry's only adaptive threat protection for containers. StackState - The market-leading Algorithmic IT Operations platform Tingyun - Observability and Analysis, Monitoring Wavefront - Wavefront is a hosted platform for ingesting, storing, visualizing and alerting on time series data. Zabbix - The Ultimate Enterprise - class Monitoring Platform Logging Fluentd - Fluentd: Unified Logging Layer (project under CNCF) Humio - Log everything, answer anything Splunk - Splunk provides operational intelligence software that monitors, reports, and analyzes real-time machine data. Elastic - Open Source, Distributed, RESTful Search Engine. Graylog - Free and open source log management Loggly - Loggly parses your log files, shows you the code in GitHub which caused the log errors. 10,000+ customers, including 1/3 of the Fortune 500. Logz - Logz.io is an enterprise-grade ELK as a service with alerts, unlimited scalability, and predictive fault detection. Loom Systems - Predict & Prevent Problems in the Digital Business Sematext - Sematext is a Search and Big Data analytics products and services company. Sumo Logic - Sumo Logic, a log management and analytics service, transforms big data into sources of operations, security and compliance intelligence. Tracing Jaeger - CNCF Jaeger, a Distributed Tracing System OpenTracing - OpenTracing API for Go Spring Cloud Sleuth - Distributed tracing for spring cloud Appdash - Application tracing system for Go, based on Google's Dapper. SkyWalking - A distributed tracing system, and APM ( Application Performance Monitoring ) Zipkin - Zipkin is a distributed tracing system Contribute Contributions are most welcome, please adhere to the contribution guidelines. ⬆ back to top License This work is licensed under a Creative Commons Attribution 4.0 International License.

优秀的个人博客,低调大师

Docker技术笔记:Docker入门浅尝

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/zhaobryant/article/details/79600059 简介 本文将用Docker的方式来构建一个应用APP。 过去,如果要开发一个Python应用APP,所需做的第一件事就是在开发机上安装Python运行时环境。在这种情形下,开发机的环境必须与APP所要求的环境一致,同时还需要与生产环境相匹配。 通过使用Docker,可以将一个可移植的Python运行时环境作为一个image获取,而无需安装。然后,就可以基于Python运行时环境image,将APP代码及其依赖库合并构建,从而简化了应用APP的部署难度。 用Dockerfile定义容器Container Dockerfile是由一系列命令和参数构成的脚本,这些命令应用于基础镜像并最终创建一个新的镜像。它简化了业务部署的流程,大大提高了业务的部署速度。Dockerfile的产出为一个新的可以用于创建容器的镜像。 Dockerfile语法由两部分构成,分别是“注释”和“命令+参数”。 # Line blocks used for commenting COMMAND argument1 argument2 ... 对于Dockerfile,我们首先创建一个空目录,然后cd到该目录,并创建Dockerfile文件。 # Use an official Python runtime as a parent image FROM python:2.7-slim # Set the working directory to /app WORKDIR /app # Copy the current directory contents into the container at /app ADD . /app # Install any needed packages specified in requirements.txt RUN pip install --trusted-host pypi.python.org -r requirements.txt # Make port 80 available to the world outside this container EXPOSE 80 # Define environment variable ENV NAME World # Run app.py when the container launches CMD ["python", "app.py"] 其中,app.py和requirements.txt都位于与Dockerfile相同的目录下,具体如下: zjl@ubuntu:~/docker/pyapp$ ls app.py Dockerfile requirements.txt 对于requirements.txt,其内容如下: Flask Redis 对于app.py,其内容如下: from flask import Flask from redis import Redis, RedisError import os import socket redis = Redis(host="redis", db=0, socket_connect_timeout=2, socket_timeout=2) app = Flask(__name__) @app.route("/") def hello(): try: visits = redis.incr("counter") except RedisError: visits = "<i>cannot connect to Redis, counter disabled</i>" html = "<h3>Hello {name}!</h3>" \ "<b>Hostname:</b> {hostname}<br/>" \ "<b>Visits:</b> {visits}" return html.format(name=os.getenv("NAME", "world"), hostname=socket.gethostname(), visits=visits) if __name__ == "__main__": app.run(host="0.0.0.0", port=80) 现在,我们知道,pip install -r requirements.txt为Python安装了Flask和Redis库,同时APP会打印出环境变量NAME,同时将socket.gethostname()打印出来。 构建新的image镜像 下面,我们进行构建,具体如下: zjl@ubuntu:~/docker/pyapp$ ls app.py Dockerfile requirements.txt zjl@ubuntu:~/docker/pyapp$ sudo docker build -t fhello . Sending build context to Docker daemon 4.608kB Step 1/7 : FROM python:2.7-slim 2.7-slim: Pulling from library/python d2ca7eff5948: Pull complete cef69dd0e5b9: Pull complete 50e1d7e4f3c6: Pull complete 861e9de5333f: Pull complete Digest: sha256:e9baca9b405d3bbba71d4c3c4ce8a461e4937413b8b910cb1801dfac0a2423aa Status: Downloaded newer image for python:2.7-slim ---> 52ad41c7aea4 Step 2/7 : WORKDIR /app ... Step 3/7 : ADD . /app ... Step 4/7 : RUN pip install --trusted-host pypi.python.org -r requirements.txt ... Step 5/7 : EXPOSE 80 ... Step 6/7 : ENV NAME World ... Step 7/7 : CMD ["python", "app.py"] ... Successfully built d3fafd68e807 Successfully tagged fhello:latest zjl@ubuntu:~/docker/pyapp$ sudo docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE fhello latest 12d000cd7a1b 12 minutes ago 148MB 运行新的image镜像 $ sudo docker run -p 4000:80 fhello * Running on http://0.0.0.0:80/ (Press CTRL+C to quit) # 访问该网站 $ curl localhost:4000 <h3>Hello World!</h3><b>Hostname:</b> f4e37f061593<br/><b>Visits:</b> <i>cannot connect to Redis, counter disabled</i> $ sudo docker container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4a150bdc67cd fhello "python app.py" 10 seconds ago Up 9 seconds 0.0.0.0:4000->80/tcp upbeat_austin 关闭容器,命令如下: $ sudo docker container stop 4a150bdc67cd 4a150bdc67cd 给镜像打上标签 syntax: -->> docker tag image username/repository:tag $ sudo docker tag fhello zhjl/getstarted:alpha-1 $ sudo docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE fhello latest 12d000cd7a1b 18 minutes ago 148MB zhjl/getstarted alpha-1 12d000cd7a1b 18 minutes ago 148MB $ docker run -p 4000:80 zhjl/getstarted:alpha-1 Recap and cheat sheet # Create image using this directory's Dockerfile docker build -t fhello . # Run "friendlyname" mapping port 4000 to 80 docker run -p 4000:80 fhello # Same thing, but in detached mode docker run -d -p 4000:80 fhello # List all running containers docker container ls # List all containers, even those not running docker container ls -a # Gracefully stop the specified container docker container stop <hash> # Force shutdown of the specified container docker container kill <hash> # Remove specified container from this machine docker container rm <hash> # Remove all containers docker container rm $(docker container ls -a -q) # List all images on this machine docker image ls -a # Remove specified image from this machine docker image rm <image id> # Remove all images from this machine docker image rm $(docker image ls -a -q) # Log in this CLI session using your Docker credential docker login # Tag <image> for upload to registry docker tag <image> username/repository:tag # Upload tagged image to registry docker push username/repository:tag # Run image from a registry docker run username/repository:tag

优秀的个人博客,低调大师

技术分享连载(四十五)

资源管理 Q1:我从UWA的报告中看出我们的GC比较严重,里面有我们自己GC的地方(切换场景的时候),然后就是系统GC的。这个我们无法控制,那么有什么办法优化吗?我发现一个很尴尬的地方,如果我们不自己去GC,内存好像不会返回,但是GC又会引起卡顿。 一般来说,手动调用GC都不太必要,而切换场景时多数情况下会因为堆内存的大量分配导致系统GC的自动触发,但这里的卡顿相对可以接受(只是稍微增加了切场景的时间)。但在非场景切换的时间里,如果系统GC也频繁出现,那么就需要对脚本的堆内存分配进行严格的控制,可以通过查看堆内存分配的TOP10函数,逐个进行分析和优化,减少各个函数的累积堆内存分配量。另外,内存的下降与GC并没有十分的关联。开发者需要注意两点:首先,在Android平台上,GC无法降低Mono部分的总内存(只是把空闲内存变大);其次,Unity部分的内存是通过Resources相关的接口来释放的,GC的操作对这部分内存也没有直接影响(理论上只要引用某资源的GameObject被销毁,那么该资源就能被Resources.UnloadUnusedAssets卸载,因此和GC也没有直接关系)。 资源管理 Q2:场景切换完了后,我们发现上个场景的内容并没有卸载完毕,所以我们调用了Assetbundle.Unload(true)这个函数,紧接着我们调用了UnloadUnusedAssets+GC操作帮我们清理资源(如果不调用后续两个操作的话,我发现内存曲线并没有得到明显下降),而这个时候GC次数又高了,GC次数高了CPU降低,不去GC内存得不到完整释放。我在你们网站上看到说不要开发者频繁去使用UnloadUnusedAssets+GC这个操作,所以希望得到一些帮助。 通过LoadLevel等API来切换场景时,Unity会自动触发Resources.UnloadUnusedAssets的操作,但在切换完成后再次调用Resources.UnloadUnusedAssets来确保卸载完全的做法也是较为常见的,而我们网站上主要是建议开发者不要在其他的时间点来手动调用Resources.UnloadUnusedAssets,在切换场景时还是可以酌情调用一次的,另外对于大场景的MMO类型的游戏,因为切换场景的频率较低,也可以考虑每隔几分钟来手动触发一次Resources.UnloadUnusedAssets来降低内存。而GC的话,则不建议手动调用,即使是在切换场景时。 资源管理 Q3:我们没有使用AssestBundle的方式,使用的都是在Resources路径下的Load,请问Resources目录下的所有内容都会加载到内存里吗?如果里面东西多,是不是会导致占用内存过高? 不会,Resources.Load也是即用即加载,但就目前我们统计的结果来看,Resources文件下的资源越多,其生成的ResourceManager内存占用也越大,研发团队可在Unity Profiler中通过Take Sample来进行查看。 资源管理 Q4:请问下,Unity 5.3.3版本,Android用AssetBundle.LoadFromFile读取Application.streamingAssetsPath目录下的AssetBundle文件,用什么样的地址? 在Unity 5.3之前,安卓上加载 StreamingAssets 目录下的 AssetBundle 时,可以直接使用 Application.streamingAssetsPath 作为目录路径,而Unity 5.3之后,通过新增的LoadFromFile接口加载AssetBundle时,则需要改为 Application.dataPath+"!assets。具体也可参考雨松的博客http://www.xuanyusong.com/archives/4033。 资源管理 Q5:Font Texture 资源是如何生成的,因为我发现好像有重复的出现,如何优化呢? 这是Unity为动态字体自动产生的纹理,一般来说不用特别关注。即使重复出现,里面的内容一般也是不一样的(内容即屏幕上显示的文字)。 原文出处:侑虎科技 本文作者:admin 转载请与作者联系,同时请务必标明文章原始出处和原文链接及本声明。

优秀的个人博客,低调大师

技术分享连载(八十六)

加载 Q1:在手机上测试LoadLevelAsync和LoadLevel的加载速度,同一个场景,LoadLevelAsync要比LoadLevel多花费40%左右的时间,请问这是正常的么?LoadLevel会有卡顿,导致Loading进度条不平滑,但是LoadLevelAsync好像又会增加Loading的时间?我项目中场景动态加载的做法是,是把物件做成Prefab,然后根据主角的位置做动态加载相应的Prefab,用的是Resources.LoadAsync方法。现在加载一些比较大的物件时,在红米2等低端机上,仍然比较卡,要消耗200ms以上。请问有什么好的方式,能平滑这个加载过程么,谢谢! LoadLevelAsync一般情况下是要比LoadLevel慢的,但是否要慢40%,这个其实是根据每个场景所要加载的不同量而定的,并不是确数。 LoadLevel和LoadLevelAsync其实最本质的区别,是前者一定要在下一帧结束前完成加载操作,所以当加载场景较大时,其随后的单帧开销就会很大,而后者则没有这个限制,引擎可以根据当前的使用情况或者ThreadPriority(这个值是是否对LoadLevelAsync有影响,还没做过具体实验,但对于LoadAsync确实有影响 )来自行调控。 但还需要说明一点的是,不是说使用Async,它就一定是绝对平滑,下图红框是LoadLevel,绿框是LoadLevelAsync操作。 关于真正“平滑”异步的加载方式,Unity引擎目前还是没有的。在遇到较为复杂的Prefab(比如大纹理、多AnimationClip等等)时,其加载依然会出现卡顿。如果想要缓解该问题,建议如下: (1)先定位具体是哪个Prefab加载造成的CPU耗时较高; (2)查看该Prefab中是否含有大量加载耗时开销(比如大纹理、较多AnimationClip、较多Shader等等); (3)定位后尝试将其进行前期预加载,从而缓解该Prefab加载时的局部高开销。 该问题来自UWA问答社区,如您对该问题仍有疑问,可以转至社区进行进一步交流。 https://answer.uwa4d.com/question/5a06cc3cc4ccc1804bea31ac 内存 Q2:我使用了IL2CPP后是否还存在Mono内存呢?使用IL2CPP后,通过Profiler工具获取的managedObject(例如: int32[])是哪种内存? 可以简单地认为,Il2CPP只是替换掉了Mono的虚拟机实现,所以该分配堆内存的地方还是会一样的分配(可能会有某些细节的地方不一样)。 IL2CPP在堆内存分配方面和Mono 最大的不同主要是Reserved Total 是可以下降的,而 Mono的 Reserved Total 只会上升不会下降。 该问题来自UWA问答社区,如您对该问题仍有疑问,可以转至社区进行进一步交流。 https://answer.uwa4d.com/question/5a0271297acd3ac1606d5d32 渲染 Q3:场景自动烘焙的导航网格信息保存在 NavMesh.asset 中,打开可以可以看到数据如下:项目需要在服务器上进行寻路,希望有大牛帮忙解析这个数据,还原成 recast 用的导航网格。Unity版本是 5.5.1,Unity 4.2版本的解析代码是有的,新版本格式变了。 首先用 NavMesh.CalculateTriangulation 获取所有的顶点和索引数据,这个是完整的、构建过的NavMesh,包括了多边形信息(最大边数为6边型的凸多边形),但是这个数据中所有的多边形是不共边的(多边形的边是重复数据),所以需要首先合并重复的边,可以利用 RacastNavigation提供的代码,基于 rcBuildPolyMesh 增加一个修改版本(比如叫:rcBuildPolyMeshBySeparatePolyTriangle),目的是合并已有的多边形的重复边,这一步结束后,就可以借用 RacastNavigation 剩余的构建流程: 1)创建 NavMesh并使用以上的 PolyMesh 初始化 (dtCreateNavMeshData, dtNavMesh::Init); 2)根据已有的 NavMesh 更新包围盒; 3)根据现有的 NavMesh 将每一个 Tile 写入文件(这之前可以写入自定义的文件头信息,方便读取); 4)服务器读取存储的数据,并用来初始化生成 NavMesh,就可以 findpath 了。(也可以将这个数据导入 RacastNavigation 的 Demo 中查看可视化情况)。 以上 1-4 是RacastNavigation示例中构建流程的一部分,重点是 rcBuildPolyMeshBySeparatePolyTriangle 这一步合并多边形即可。这么做的好处是,导出的 NavMesh,跟 Unity 里构建的,多边形信息是一模一样的。想要支持 DetailMesh,估计还不行。 另外,服务器直接用可能还有点问题,如果 Unity 导出的数据很多细长的三角形,那么RacastNavigation的结果会和 Unity出现很大的差异,需要稍微修改下RacastNavigation的代码 。 我之前记录了个不太详细的描述,仅供参考:http://www.cnblogs.com/yaukey/p/recastnavigation_unity_optimize_visible_result.html 该问题来自UWA问答社区,感谢 Yaukey提供了回答,如您对该问题仍有疑问,可以转至社区进行进一步交流。 https://answer.uwa4d.com/question/59f93d2aa740da296dd03662 渲染 Q4:目前我在用的Unity版本是2017.1.0f3 for MAC,在iOS平台下Bake Environment Reflections的时候丢失HDR信息,操作步骤如下: 1、当我使用Standalone Platform时,我使用了一张带有HDR的SkyBox CubeMap贴图(默认压缩方式BC6H),通过它生成的ReflectionProbe-0.exr会给物体带来明显的一个高光效果,如图所示:2、当切换到iOS Platform后,再次用同样的SkyBox CubeMap贴图(压缩方式为PVRTC,或不压缩RGB24 bit都试过)烘焙ReflectionProbe-0.exr时,发现这个效果消失了,如图所示:3、此时,如果我把之前的那张Standalone下面生成ReflectionProbe-0.exr替换过来,可以获得跟Standalone下一样的高光效果。把该工程发布到iPhone 7P上运行,效果与“步骤一”中一致。 说明ReflectionProbe-0中的HDR信息得到了保留。但是我注意到,在iOS Platform下面,ReflectionProbe-0的默认压缩方式是PVRTC或RGB 24 bit,这两种方式都无法保留HDR信息。如图所示:4、通过对比两张ReflectionProbe-0.exr贴图,发现他们的文件大小是不同的,Standalone下面的贴图要大一些。如图所示,请问这是Unity的Bug吗?我在此上传了示例工程:exr_EnvRef_iOS.zip,默认是PC Standalone,需要手动切换一下Platform,切换后重新Generate Lighting。 用题主的工程文件尝试了一下,的确是出现同样问题,并且在Android平台也是一样。目前看来只能判断是Reflection-Probe烘焙时使用了两套不同的操作。 再提供一个线索,用renderdoc->texture viewer可以查看像素值:Standalone(PC) Android 从上图可以看到相似区域Standalone比Android的烘焙结果数值要高出许多。 该回复由UWA提供。 我在Unity 5.6下发现效果是一样的,结论和题主一样:Mobile平台下Bake出来的Probe是LDR的。他是通过放大亮度来做的,但其实有更简单的办法:看高mip就行,首先是PC模式下Bake的。 然后是Android下Bake的: 出现亮度区别的原因是PC上有亮度>1的像素(一般是太阳之类的) 然后在卷积的时候就值大一些;而Android下就相对值低 在mip=0的情况下大家都是0-1看不出(显示时候截断了) ,目前的结论就是在移动平台Bake的Probe依然是LDR的,这是有问题的…临时解决方案就是在Standalone上Bake完复制过来。感谢钱康来提供了以上解答。 题主补充:有了新的发现,基本上可以确认是Bug了。两张reflectionProbe-0.exr贴图的确不一样!除了尺寸之外,切换到iOS平台后生成的exr丢失了高光部分的HDR信息。 这两张图片初看没什么不一样的: 调整exposure,令exposure变为-5 (我用的是openexr,也可以用ps直接调整曝光度): 可以看到上图有两个非常明显的光斑,这个光斑的亮度非常高,因此会在物体上产生非常强的亮度。然而,在iOS Platform下面进行烘焙的时候,Unity把这个信息给丢失了。但不是所有的exr都会出现这样的问题。只有那些具有局部强光源的HDR图片才会出现这种情况,例如楼主用的那张,还有这张图片: http://gl.ict.usc.edu/Data/HighResProbes/probes/grace-new.exr 原文出处:侑虎科技 本文作者:admin 转载请与作者联系,同时请务必标明文章原始出处和原文链接及本声明。

优秀的个人博客,低调大师

技术分享连载(八十三)

性能 Q1:我用UWA GOT进行本地性能测试,在CPU的数据分析中发现,某些帧StackTraceUtility耗时特别高,这是什么原因导致的呢? StackTraceUtility.XXX是Unity引擎的Log输出,可能是本身Debug.Log/LogError的调用输出,也可能是使用过程中引擎端出现了Warning/Error等信息而自动输出的。 该问题来自UWA问答社区,如您对该问题仍有疑问,可以转至社区进行进一步交流。 https://answer.uwa4d.com/question/59ed93681b83b8df50804f6a 加载 Q2:我在Profiler中观察性能曲线,发现某一帧AssetBundle加载中,LockPersistentManager耗时比较大。请问这部分能否优化? 这说明当前帧或前几帧中存在较大量的资源在通过LoadAsync来进行加载,其本质是所加载的资源过大所致,对自身资源进行合理优化可降低Loading.LockPersistentManager的开销。另外,将异步加载换成同步加载,LockPersistentManager就不会出现了,但其总加载耗时是没有变化的,因为总加载量没变。 关于主要资源的加载优化,可参考如下链接: Unity加载模块深度解析(纹理篇) Unity加载模块深度解析(网格篇) Unity加载模块深度解析(Shader篇) Unity加载模块深度解析(动画片段篇) UWA 六月直播季 | 6.8 移动游戏加载性能和内存管理全解析 该问题来自UWA问答社区,如您对该问题仍有疑问,可以转至社区进行进一步交流。 https://answer.uwa4d.com/question/59eae0c4e0a5e637724a1572 制作 Q3:我在Unity 5.4.4f1下打出的AssetBundle无法加载,抛出以下错误,但是只是部分无法加载,平台是iOS ,打包参数有以下:DeterministicAssetBundle;DisableWriteTypeTree;ChunkBasedCompression;CollectDependencies;CompleteAssets。 出现这个Log是因为项目的逻辑脚本序列化信息更新了,也就是资源的Typetree变化了。这种情况下,需要重新Build相应的AssetBundle,如果使用之前的AssetBundle,那么就会出现上述Log。 该问题来自UWA问答社区,如您对该问题仍有疑问,可以转至社区进行进一步交流。 https://answer.uwa4d.com/question/59e8aa5fe02a95cc6d0c498a 性能 Q4:我在安卓真机上跑游戏,发现Profiler下的合批数据和PC或者iOS下的不一样,因此不确定Android的合批是否有效。如下图,左边的是Android的,右边的是PC的(iOS和PC基本一致),Android的Render.Mesh明显比PC和iOS的高好多。我用的设备是魅族 mote2 Android版本 - 5.1补上FrameDebug的数据,上下的Mesh数据都是一样的,也没有z-fighting的情况,因为都是间隔地渲的,同样的物体一个一个地渲出来。经过分析,Android上不能设置Graphics Jobs(在Player Settings里面),也是不断打包测试发现这个问题,想了解一下具体是什么原因呢? Graphics Job目前在Android平台上是为Vulkan而设计的,也就是只有支持Vulkan设备的才会真正起作用。按照Unity原厂的说法,该选项在不支持Vulkan的Android设备上应该是没有效果的。 另外,Graphics Job和MultiThread Rendering并不建议同时使用,而且以目前的Android设备来说,建议只开启MultiThread Rendering一项即可。 该问题来自UWA问答社区,如您对该问题仍有疑问,可以转至社区进行进一步交流。 https://answer.uwa4d.com/question/59e4510ce5b24a0754868985 制作 Q5:Unity 5.5.1版本下,Sprite Packer在iOS平台下RGBA PVRTC4打包图集失真非常严重(对单个的Sprite设置PVRTC4是正常的),参照了4.6.7版本是正常的,我想知道为什么呢? 确实可以复现。Unity 5.5 的PVRTC压缩有三档 fast/nomal/best,但都比Unity 4.7下的质量要差。找到一个官方的 issue ,说是 pvrtc 压缩工具(第三方)的Bug。 https://issuetracker.unity3d.com/issues/textures-rgba-compressed-pvrtc-2-slash-4bit-texture-with-compression-quality-set-to-to-best-slash-normal-is-less-detailed-than-fast 有人在这个帖子里提到,在 Windows 下尝试用 4.7 的 pvrtextool.exe 替换了 5.x 的,暂时解决了这个问题,建议也尝试一下。 原文出处:侑虎科技 本文作者:admin 转载请与作者联系,同时请务必标明文章原始出处和原文链接及本声明。

优秀的个人博客,低调大师

大数据技术原理与应用

推荐一个视频: http://study.163.com/course/courseMain.htm?courseId=1002887002 章节1课程介绍 课时1课程介绍06:53 章节2大数据概述 课时2大数据概述第1部分26:56 课时3大数据概述第2部分31:11 课时4大数据概述第3部分13:04 章节3大数据处理架构Hadoop 课时5大数据处理架构Hadoop第1部分26:52 课时6大数据处理架构Hadoop第2部分20:50 课时7大数据处理架构Hadoop第3部分15:14 课时8大数据处理架构Hadoop第4部分18:43 章节4分布式文件系统HDFS 课时9分布式文件系统HDFS第1部分24:43 课时10分布式文件系统HDFS第2部分20:15 课时11分布式文件系统HDFS第3部分20:19 课时12分布式文件系统HDFS第4部分21:33 章节5分布式数据库HBase 课时13分布式数据库HBase第1部分24:49 课时14分布式数据库HBase第2部分26:17 课时15分布式数据库HBase第3部分30:37 课时16分布式数据库HBase第4部分25:26 章节6NoSQL数据库 课时17NoSQL数据库第1部分23:45 课时18NoSQL数据库第2部分26:44 课时19NoSQL数据库第3部分23:09 课时20NoSQL数据库第4部分16:24 章节7云数据库 课时21云数据库第1部分21:59 课时22云数据库第2部分28:40 课时23云数据库第3部分31:04 课时24云数据库第4部分21:28 章节8MapReduce 课时25MapReduce第1部分25:11 课时26MapReduce第2部分28:17 课时27MapReduce第3部分26:35 课时28MapReduce第4部分19:40 章节9基于Hadoop的数据仓库Hive 课时29基于Hadoop的数据仓库Hive第1部分29:10 课时30基于Hadoop的数据仓库Hive第2部分25:56 课时31基于Hadoop的数据仓库Hive第3部分23:30 课时32基于Hadoop的数据仓库Hive第4部分25:17 章节10Hadoop架构再探讨 课时33Hadoop架构再探讨第1部分28:32 课时34Hadoop架构再探讨第2部分25:58 课时35Hadoop架构再探讨第3部分21:39 课时36Hadoop架构再探讨第4部分35:39 章节11Spark 课时37Spark第1部分30:15 课时38Spark第2部分31:38 课时39Spark第3部分35:53 课时40Spark第4部分26:05 章节12流计算 课时41流计算第1部分30:17 课时42流计算第2部分35:36 课时43流计算第3部分33:41 课时44流计算第4部分26:43 章节13图计算 课时45图计算第1部分28:51 课时46图计算第2部分35:38 课时47图计算第3部分34:47 课时48图计算第4部分30:30 章节14大数据在不同领域的应用 课时49大数据在不同领域的应用第1部分22:41 课时50大数据在不同领域的应用第2部分24:09 课时51大数据在不同领域的应用第3部分25:09 章节15附录:大数据在线课程建设团队风采 课时52附录:大数据在线课程建设团队风

资源下载

更多资源
Mario

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长,特征是大鼻子、头戴帽子、身穿背带裤,还留着胡子。与他的双胞胎兄弟路易基一起,长年担任任天堂的招牌角色。

腾讯云软件源

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题,腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构,目前腾讯云软件源站支持公网访问和内网访问。

Nacos

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称,一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集,帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

WebStorm

WebStorm

WebStorm 是jetbrains公司旗下一款JavaScript 开发工具。目前已经被广大中国JS开发者誉为“Web前端开发神器”、“最强大的HTML5编辑器”、“最智能的JavaScript IDE”等。与IntelliJ IDEA同源,继承了IntelliJ IDEA强大的JS部分的功能。

用户登录
用户注册