LazyLLM教程 | 第4讲：RAG项目工程化入门：从脚本走向模块化与可维护性

>>> mkdir my-project && ​cd​ my-project >>> echo "# test" >> README.md >>> git init  # 首次使用需要设置用户名和邮箱 >>> git config --global user.name "你的用户名" >>> git config --global user.email "你的邮箱"  >>> git add README.md >>> git commit -m "test" >>> git branch -M main >>> git remote add origin <your-repo-url> >>> git push -u origin main 

>>> git push -u origin main Username for 'https://github.com': lwj-st Password for 'https://lwj-st@github.com':  Enumerating objects: 3, done. Counting objects: 100% (3/3), done. Writing objects: 100% (3/3), 223 bytes | 223.00 KiB/s, done. Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 To https://github.com/lwj-st/my-project.git  * [new branch] main -> main Branch 'main' set up to track remote branch 'main' from 'origin'. 

# my-project  ## 介绍 这是一个示例 Python 项目，支持自动化测试、Docker 部署，并符合 PEP 8 代码规范。  ## 安装 pip install my-project 

 __pycache__  *.pyc  test/  dist/  tmp/  .vscode  build  *.lock  *.db 

 lazyllm 

from lazyllm import Retriever, Document  def create_retriever(path: str, query: str):  """  创建并执行检索   Args:  path (str): 文档的绝对路径  query (str): 查询语句   Returns:  list: 检索结果  """  doc = Document(path)  retriever = Retriever(doc, group_name="CoarseChunk", similarity="bm25_chinese", topk=3)  return retriever(query) 

from .retriever import create_retriever  __version__ = '0.1.0' __all__ = ['create_retriever'] 

# 创建并切换到开发分支 >>> git checkout -b dev  # 在dev分支上新创建功能分支 >>> git checkout -b feature/new-feature  ...修改代码...  >>> git add . >>> git commit -m "Add new feature" >>> git push origin feature/new-feature  # 开发完成后合并回 dev >>> git checkout dev  >>> git merge feature/new-feature >>> git push origin dev 

To https://github.com/lwj-st/my-project.git  ! [rejected] dev -> dev (fetch first) error: failed to push some refs to 'https://github.com/lwj-st/my-project.git' hint: Updates were rejected because the remote contains work that you do hint: not have locally. This is usually caused by another repository pushing hint: to the same ref. You may want to first integrate the remote changes hint: (e.g., 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details. 

CONFLICT (content): Merge conflict in requirements.txt Automatic merge failed; fix conflicts and then commit the result. 

>>> git status On branch dev Your branch and 'origin/dev' have diverged, and have 1 and 1 different commits each, respectively.  (use "git pull" to merge the remote branch into yours)  You have unmerged paths.  (fix conflicts and run "git commit")  (use "git merge --abort" to abort the merge)  Unmerged paths:  (use "git add <file>..." to mark resolution)  both modified: requirements.txt  no changes added to commit (use "git add" and/or "git commit -a") 

>>> git add requirements.txt >>> git commit -m "解决冲突" >>> git push

<<<<<<< dev xxxx  ======= xxxx  >>>>>>> main 

>>> pip install black 

>>> black . reformatted /root/my_project/__init__.py reformatted /root/my_project/retriever.py  All done! ✨ 🍰 ✨ 2 files reformatted. 

>>> pip install flake8 

>>> flake8 . 

>>> pip install pre-commit 

>>> pre-commit install 

repos:  - repo: https://github.com/psf/black  rev: 23.1.0  hooks:  - id: black   - repo: https://github.com/pycqa/flake8  rev: 6.0.0  hooks:  - id: flake8 

>>> pip install pytest 

import pytest from my_project import create_retriever  # 测试文档路径 TEST_PATH = "./data_kb"  def test_retriever_contains_keyword():  test_query = "为我介绍一下2008年北京奥运会"  expected_keyword = "奥运比赛"   results = create_retriever(TEST_PATH, test_query)  top_content = results[0].get_content() if results else ""   assert expected_keyword in top_content, f"检索结果中未找到关键词 '{expected_keyword}'"  def test_retriever_empty_query():  results = create_retriever(TEST_PATH, "")  assert isinstance(results, list), "结果应该是列表类型" 

>>> export PYTHONPATH=${PWD}:$PYTHONPATH >>> pytest --disable-warnings tests/test_retriever.py  ==================================== test session starts ==================================== platform linux -- Python 3.10.9, pytest-8.3.3, pluggy-1.5.0 rootdir: /root plugins: anyio-4.4.0, hydra-core-1.3.2 collected 2 items   tests/test_retriever.py .. [100%]  ===================================== 2 passed in 0.01s ===================================== 

# content of test_server.py import pytest  @pytest.mark.webtest def test_send_http():  pass # perform some webtest test for your app  @pytest.mark.device(serial="123") def test_something_quick():  pass  @pytest.mark.device(serial="abc") def test_another():  pass  class TestClass:  def test_method(self):  pass 

>>> pytest -v -m webtest ==================================== test session starts ==================================== platform linux -- Python 3.10.9, pytest-8.3.3, pluggy-1.5.0 -- /opt/miniconda3/envs/lazyllm/bin/python cachedir: .pytest_cache rootdir: /root/my-project plugins: anyio-4.4.0, hydra-core-1.3.2 collected 4 items / 3 deselected / 1 selected  tests/test_server.py::test_send_http PASSED [100%] ======================== 1 passed, 3 deselected, 3 warnings in 0.01s ======================== 

>>> pytest -v -m "device(serial='123')" ==================================== test session starts ==================================== platform linux -- Python 3.10.9, pytest-8.3.3, pluggy-1.5.0 -- /opt/miniconda3/envs/lazyllm/bin/python cachedir: .pytest_cache rootdir: /root/my-project plugins: anyio-4.4.0, hydra-core-1.3.2 collected 4 items / 3 deselected / 1 selected  tests/test_server.py::test_something_quick PASSED [100%] ======================== 1 passed, 3 deselected, 3 warnings in 0.03s ======================== 

>>> export PYTHONPATH=${PWD}:$PYTHONPATH >>> pytest --cov=my_project --cov-append --cov-report=html 

name: Run Tests on: [push, pull_request] jobs:  test:  runs-on: ubuntu-latest  steps:  - uses: actions/checkout@v3  - name: Set up Python  uses: actions/setup-python@v3  with:  python-version: "3.10"  - name: Install dependencies  run: pip install -r requirements.txt  - name: Run tests  run: |  export PYTHONPATH=${PWD}:$PYTHONPATH   pytest 

>>> pip install mkdocs 

>>> mkdocs new my-project >>> cd my-project 

my-project/ ├── docs │ └── index.md └── mkdocs.yml 

 $ mkdocs serve  INFO - Building documentation...  INFO - Cleaning site directory  INFO - Documentation built in 0.22 seconds  INFO - [15:50:43] Watching paths for changes: 'docs', 'mkdocs.yml'  INFO - [15:50:43] Serving on http://127.0.0.1:8000/ 

# .readthedocs.yaml # Read the Docs 配置文件  # 必需的版本字段 version: 2  # 设置构建环境 build:  os: ubuntu-24.04  tools:  python: "3.10" # MkDocs 需要 Python  # 配置 MkDocs mkdocs:  configuration: mkdocs.yml # 默认的 MkDocs 配置文件  # 可选：指定 Python 依赖（如果有） #python: # install: # - requirements: requirements.txt # 如果你有依赖文件 

from setuptools import setup, find_packages  setup(  name="my_project",  version="0.1.0",  packages=find_packages(),  install_requires=[  "lazyllm",  ],  author="Your Name",  author_email="your@email.com",  description="A simple RAG retriever package",  long_description=open("README.md").read(),  long_description_content_type="text/markdown",  python_requires=">=3.6", ) 

>>> python setup.py bdist_wheel running bdist_wheel running build running build_py creating build creating build/lib creating build/lib/my_project  ....  adding 'my_project-0.1.0.dist-info/WHEEL' adding 'my_project-0.1.0.dist-info/top_level.txt' adding 'my_project-0.1.0.dist-info/RECORD' removing build/bdist.linux-x86_64/wheel >>> ls dist/ my_project-0.1.0-py3-none-any.whl 

[pypi]  username = __token__  password = pypi-xxxxxxxxxxxxxxxxxxxYjNjNS0xMDExNWMwMzhlNDMiXQAABiDpxiNjoqIT3SJDNrQPP-BJl_AhO7pHErgKvOnS4jzNrQ 

>>> pip install twine >>> twine upload dist/* 

name: Publish to PyPI  on:  push:  tags:  - "v*" # 仅在创建 tag（如 v1.0.0）时触发  jobs:  deploy:  runs-on: ubuntu-latest  steps:  - name: Checkout repository  uses: actions/checkout@v3   - name: Setup Python  uses: actions/setup-python@v4  with:  python-version: "3.x"   - name: Install dependencies  run: |  python -m pip install --upgrade pip  pip install build twine   - name: Build package  run: python -m build   - name: Upload to PyPI  env:  INDEX_PYPI_TOKEN: ${{ secrets.INDEX_PYPI_TOKEN }}  run: |  twine upload --username __token__ --password $INDEX_PYPI_TOKEN dist/* 

>>> pip install my-project 

>>> pip install "my-project==0.1.0" >>> pip install "my-project>=0.1.0" 

>>> docker login 

# 1. 选择基础镜像 FROM python:3.10  # 2. 设置环境变量 ENV PYTHONUNBUFFERED=1  # 3. 创建应用文件夹 RUN mkdir /app  # 4. 拷贝源码 COPY my_project /app/my_project COPY requirements.txt /tmp/requirements.txt ENV PYTHONPATH="/app:${PYTHONPATH}"  # 5. 安装依赖 RUN pip install -r /tmp/requirements.txt \  && rm -rf /tmp/requirements.txt  # 6. 创建非 root 用户，提升安全性 RUN useradd -m myuser USER myuser  # 7. 设置默认启动命令  CMD ["/bin/bash"] 

>>> docker build -t username/my-project:0.1.0 . 

>>> docker push username/my-project:0.1.0 

>>> docker pull username/my-project:0.1.0 

>>> docker run -it --name my-project username/my-project:0.1.0