首页 文章 精选 留言 我的

精选列表

搜索[蓝鲸平台],共10006篇文章
优秀的个人博客,低调大师

ELK日志收集平台部署

需求背景 一位朋友的公司研发最近有一些苦恼。由于他们公司的后台服务有三台,每当后台服务运行异常,需要看日志排查错误的时候,都必须开启3个ssh窗口进行查看,研发们觉得很不方便,于是便有了统一日志收集与查看的需求。 这里,我用ELK集群,通过收集三台后台服务的日志,再统一进行日志展示,实现了这一需求。 当然,当前只是进行了简单的日志采集,如果后期相对某些日志字段进行分析,则可以通过logstash以及Kibana来实现。 部署环境 系统:CentOS 7 软件: elasticsearch-6.1.1 logstash-6.1.1 kibana-6.1.1 下载地址:https://www.elastic.co/cn/products 搭建步骤 一:elasticsearch: elasticsearch是用于存储日志的数据库。 下载elasticsearch软件,解压: 1 2 #tar-zxvfelasticsearch-6.1.1.tar.gz #mvelasticsearch-6.1.1/opt/apps/elasticsearch 由于elasticsearch建议使用非root用户启动,使用root启动会报错,故需创建一个普通用户,并进行一些简单配置: 1 2 3 4 5 6 #useraddelk #vi/opt/apps/elasticsearch/config/elasticsearch.yml network.host:0.0.0.0 http.port:9200 http.cors.enabled: true http.cors.allow-origin: "*" 启动,并验证: 1 2 3 4 5 6 7 #su-elk $ nohup /opt/apps/elasticsearch/bin/elasticsearch & #netstat-ntpl|grep9200 tcp000.0.0.0:92000.0.0.0:*LISTEN6637 /java #curl'localhost:9200/_cat/health?v' epochtimestampclusterstatusnode.totalnode.datashardsprireloinitunassignpending_tasksmax_task_wait_timeactive_shards_percent 151485803309:53:53elasticsearchyellow11241241002410-50.0% 二:logstash logstash用于收集各服务器上的日志,然后把收集到的日志,存储进elasticsearch。收集日志的方式有很多种,例如结合redis或者filebeat,这里我们使用redis收集的方式。 安装logstash: 1 2 3 在所有服务器上: #tar-zxvflogstash-6.1.1.tar.gz #mvlogstash-6.1.1/opt/apps/logstash/ 配置后台服务器,收集相关的日志: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 在三台后台服务器上新建logstash文件,配置日志收集: #vi/opt/conf/logstash/logstash.conf input{ file { #指定type type => "web_stderr" #匹配多行的日志 codec=>multiline{ pattern=> "^[^0-9]" what=> "previous" } #指定本地的日志路径 path=>[ "/opt/logs/web-stderr.log" ] sincedb_path=> "/opt/logs/logstash/sincedb-access" } file { type => "web_stdout" codec=>multiline{ pattern=> "^[^0-9]" what=> "previous" } path=>[ "/opt/logs/web-stdout.log" ] sincedb_path=> "/opt/logs/logstash/sincedb-access" } #收集nginx日志 file { type => "nginx" path=>[ "/opt/logs/nginx/*.log" ] sincedb_path=> "/opt/logs/logstash/sincedb-access" } } output{ #指定输出的目标redis redis{ host=> "xx.xx.xx.xx" port=> "6379" data_type=> "list" key=> "logstash" } } 配置elk日志服务器上的logstash,从redis队列中读取日志,并存储到elasticsearch中: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 #vi/opt/conf/logstash/logstash-server.conf #配置从redis队列中读取收集的日志 input{ redis{ host=> "xx.xx.xx.xx" port=> "6379" type => "redis-input" data_type=> "list" key=> "logstash" threads=>10 } } #把日志输出到elasticsearch中 output{ elasticsearch{ hosts=> "localhost:9200" index=> "logstash-%{type}.%{+YYYY.MM.dd}" } #这里把日志收集到本地文件 file { path=> "/opt/logs/logstash/%{type}.%{+yyyy-MM-dd}" codec=>line{ format => "%{message}" } } } 启动logstash进程: 1 2 3 4 后台服务器: #nohup/opt/apps/logstash/bin/logstash-f/opt/conf/logstash/logstash.conf--path.data=/opt/data/logstash/logstash& elk日志服务器: #nohup/opt/apps/logstash/bin/logstash-f/opt/conf/logstash/logstash-server.conf--path.data=/opt/data/logstash/logstash-server& 三:kibana kibana用于日志的前端展示。 安装、配置kibana: 1 2 3 4 5 6 7 8 #tar-zxvfkibana-6.1.1-linux-x86_64.tar.gz #mvkibana-6.1.1-linux-x86_64/opt/apps/kibana 配置elasticsearch链接: #vi/opt/apps/kibana/config/kibana.yml server.port:5601 server.host: "0.0.0.0" #配置elasticsearch链接: elasticsearch.url: "http://localhost:9200" 启动kibana: 1 nohup /opt/apps/kibana/bin/kibana & 访问kibana: http://localhost:5601 可以根据我们在logstash中配置的type,创建索引: 可以根据我们创建的索引,进行查看(这里查看nginx日志): 后记: 当然了,结合logstash和kibana不单单仅能实现收集日志的功能,通过对字段的匹配、筛选以及结合kibana的图标功能,能对我们想要的字段进行分析,实现相应的数据报表等。 ELK的功能十分强大,以后有机会,继续和大家探讨!! 本文转自 icenycmh 51CTO博客,原文链接:http://blog.51cto.com/icenycmh/2056540,如需转载请自行联系原作者

优秀的个人博客,低调大师

libyuv编译(各平台)【转】

转自:http://blog.csdn.net/wszawsz33/article/details/51669719 版权声明:本文为博主原创文章,未经博主允许不得转载。 目录(?)[-] Getting Started Pre-requisites Getting the Code Android Building the Library and Unittests Windows Building with clangcl OSX iOS Android Building with GN Building Offical with GN Linux CentOS Windows Shared Library Build targets Building the Library with make Linux Building the Library with cmake Windows 8 Phone Windows Shared Library 64 bit Windows ARM Linux Running Unittests Windows OSX Linux CPU Emulator tools Intel SDE Software Development Emulator Memory tools Running Dr Memory memcheck for Windows Running UBSan Running Valgrind memcheck Running Thread Sanitizer TSan Running Address Sanitizer ASan Benchmarking Windows Linux and Mac Making a change 最近用到视频格式转码,发现谷歌的开源库 libyuv 很好用,所以记录下编译过程 直接贴 官网的编译流程 https://chromium.googlesource.com/libyuv/libyuv/+/master/docs/getting_started.md 不过我得吐槽一下,我特么就想编译个libyuv 你特么非要让我把chrome项目check 下来是几个意思。。。。如果没猜错 应该是好几个g Getting Started How to get and build the libyuv code. Pre-requisites You'll need to have depot tools installed:https://www.chromium.org/developers/how-tos/install-depot-toolsRefer to chromium instructions for each platform for other prerequisites. 说明:依赖工具,谷歌的这套东西还是比较好用的,首先要安装depottools 这点不多说,命令行就行,最重要的是在下载完成之后要有 export PATH=`pwd`/depot_tools:"$PATH" 命令使能这个工具 Getting the Code Create a working directory, enter it, and run: 说明:设置与同步代码,里面自带脚本,关于代码如何同步的就无需关心了。 Then you'll get a .gclient file like: solutions = [ { "name" : "libyuv", "url" : "https://chromium.googlesource.com/libyuv/libyuv", "deps_file" : "DEPS", "managed" : True, "custom_deps" : { }, "safesync_url": "", }, ]; For iOS addios'];to your OSX .gclient and run Browse the Git reprository:https://chromium.googlesource.com/libyuv/libyuv/+/master Android For Android addandroid'];to your Linux .gclient solutions = [ { "name" : "libyuv", "url" : "https://chromium.googlesource.com/libyuv/libyuv", "deps_file" : "DEPS", "managed" : True, "custom_deps" : { }, "safesync_url": "", }, ]; target_os = ["android", "unix"]; Then run: Caveat: Theres an error with Google Play services updates. If you get the error “Your version of the Google Play services library is not up to date”, run the following: cd chromium/src ./build/android/play_services/update.py download cd ../.. For Windows the gclient sync must be done from an Administrator command prompt. The sync will generate native build files for your environment using gyp (Windows: Visual Studio, OSX: XCode, linux: make). This generation can also be forced manually: To get just the source (not buildable): git clonehttps://chromium.googlesource.com/libyuv/libyuv Building the Library and Unittests Windows Building with clangcl OSX Clang 64 bit shown. Removeclang=1for GCC and change x64 to ia32 for 32 bit. iOS http://www.chromium.org/developers/how-tos/build-instructions-ios Add to .gclient last line:target_os=['ios']; armv7 arm64 both armv7 and arm64 (fat) simulator Android https://code.google.com/p/chromium/wiki/AndroidBuildInstructions Add to .gclient last line:target_os=['android']; armv7 arm64 ia32 mipsel arm32 disassembly: arm64 disassembly: Running tests: Running test as benchmark: Running test with C code: Building with GN Building Offical with GN Linux CentOS On CentOS 32 bit the following work around allows a sync: Windows Shared Library Modify libyuv.gyp from ‘static_library’ to ‘shared_library’, and add ‘LIBYUV_BUILDING_SHARED_LIBRARY’ to ‘defines’. After this command follow the building the library instructions above. If you get a compile error for atlthunk.lib on Windows, readhttp://www.chromium.org/developers/how-tos/build-instructions-windows Build targets Building the Library with make Linux Building the Library with cmake Install cmake:http://www.cmake.org/ Default debug build: Release build/install Windows 8 Phone Pre-requisite: Install Visual Studio 2012 and Arm to your environment. Then: or with Visual Studio 2013: Windows Shared Library Modify libyuv.gyp from ‘static_library’ to ‘shared_library’, and add ‘LIBYUV_BUILDING_SHARED_LIBRARY’ to ‘defines’. Then run this. After this command follow the building the library instructions above. If you get a compile error for atlthunk.lib on Windows, readhttp://www.chromium.org/developers/how-tos/build-instructions-windows 64 bit Windows ARM Linux export GYP_DEFINES="target_arch=arm" export CROSSTOOL=`<path>`/arm-none-linux-gnueabi export CXX=$CROSSTOOL-g++ export CC=$CROSSTOOL-gcc export AR=$CROSSTOOL-ar export AS=$CROSSTOOL-as export RANLIB=$CROSSTOOL-ranlib gclient runhooks Running Unittests Windows OSX Linux Replace --gtest_filter=“*” with specific unittest to run. May include wildcards. e.g. CPU Emulator tools Intel SDE (Software Development Emulator) Pre-requisite: Install IntelSDE for Windows:http://software.intel.com/en-us/articles/intel-software-development-emulator Then run: Memory tools Running Dr Memory memcheck for Windows Pre-requisite: Install Dr Memory for Windows and add it to your path:http://www.drmemory.org/docs/page_install_windows.html Running UBSan See Chromium instructions for sanitizers:https://www.chromium.org/developers/testing/undefinedbehaviorsanitizer Sanitizers available: TSan, MSan, ASan, UBSan, LSan Running Valgrind memcheck Memory errors and race conditions can be found by running tests under special memory tools. [Valgrind]1is an instrumentation framework for building dynamic analysis tools. Various tests and profilers are built upon it to find memory handling errors and memory leaks, for instance. Then run: For more information, seehttp://www.chromium.org/developers/how-tos/using-valgrind Running Thread Sanitizer (TSan) For more info, seehttp://www.chromium.org/developers/how-tos/using-valgrind/threadsanitizer Running Address Sanitizer (ASan) For more info, seehttp://dev.chromium.org/developers/testing/addresssanitizer Benchmarking The unittests can be used to benchmark. Windows Linux and Mac Indicates 0.547 ms/frame for 1280 x 720. Making a change gclient sync git checkout -b mycl -t origin/master git pull <edit files> git add -u git commit -m "my change" git cl lint git cl try git cl upload -r a-reviewer@chomium.org -s <once approved..> git cl land 【作者】 张昺华 【出处】 http://www.cnblogs.com/sky-heaven/ 【博客园】 http://www.cnblogs.com/sky-heaven/ 【新浪博客】 http://blog.sina.com.cn/u/2049150530 【知乎】 http://www.zhihu.com/people/zhang-bing-hua 【我的作品---旋转倒立摆】 http://v.youku.com/v_show/id_XODM5NDAzNjQw.html?spm=a2hzp.8253869.0.0&from=y1.7-2 【我的作品---自平衡自动循迹车】 http://v.youku.com/v_show/id_XODM5MzYyNTIw.html?spm=a2hzp.8253869.0.0&from=y1.7-2 【新浪微博】 张昺华--sky 【twitter】 @sky2030_ 【facebook】 张昺华 zhangbinghua 本文版权归作者和博客园共有,欢迎转载,但未经作者同意必须保留此段声明,且在文章页面明显位置给出原文连接,否则保留追究法律责任的权利.

资源下载

更多资源
Nacos

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称,一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集,帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

Spring

Spring

Spring框架(Spring Framework)是由Rod Johnson于2002年提出的开源Java企业级应用框架,旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念,提供核心容器、应用上下文、数据访问集成等模块,支持整合Hibernate、Struts等第三方框架,其适用范围不仅限于服务器端开发,绝大多数Java应用均可从中受益。

Rocky Linux

Rocky Linux

Rocky Linux(中文名:洛基)是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版,作为CentOS稳定版停止维护后与RHEL(Red Hat Enterprise Linux)完全兼容的开源替代方案,由社区拥有并管理,支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性,采用模块化包装和SELinux安全架构,默认包含GNOME桌面环境及XFS文件系统,支持十年生命周期更新。

Sublime Text

Sublime Text

Sublime Text具有漂亮的用户界面和强大的功能,例如代码缩略图,Python的插件,代码段等。还可自定义键绑定,菜单和工具栏。Sublime Text 的主要功能包括:拼写检查,书签,完整的 Python API , Goto 功能,即时项目切换,多选择,多窗口等等。Sublime Text 是一个跨平台的编辑器,同时支持Windows、Linux、Mac OS X等操作系统。

用户登录
用户注册