download wikitext bug fix

.github/dependabot.yml

@@ -0,0 +1,11 @@
+# To get started with Dependabot version updates, you'll need to specify which
+# package ecosystems to update and where the package manifests are located.
+# Please see the documentation for all configuration options:
+# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
+
+version: 2
+updates:
+  - package-ecosystem: "pip" # See documentation for possible values
+    directory: "/" # Location of package manifests
+    schedule:
+      interval: "weekly"

.github/workflows/dev.yml

@@ -0,0 +1,38 @@
+name: Olah GitHub Actions for Development
+run-name: Olah GitHub Actions for Development
+on:
+  push:
+    branches: [ "dev" ]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.9", "3.10", "3.11", "3.12"]
+
+    steps:
+      - name: Check out repository code
+        uses: actions/checkout@v4
+      - name: Set up Apache Arrow
+        run: |
+          sudo apt update
+          sudo apt install -y -V ca-certificates lsb-release wget
+          wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
+          sudo apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
+          sudo apt update
+          sudo apt install -y -V libarrow-dev libarrow-glib-dev libparquet-dev libparquet-glib-dev
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install Olah
+        run: |
+          cd ${{ github.workspace }}
+          pip install --upgrade pip
+          pip install -e .
+
+      - name: Test Olah
+        run: |
+          cd ${{ github.workspace }}
+          python -m unittest discover olah/tests

.github/workflows/release.yml

@@ -0,0 +1,54 @@
+name: Olah GitHub Actions to release
+run-name: Olah GitHub Actions to release
+on:
+  push:
+    tags:
+      - "[0-9]+.[0-9]+.[0-9]+"
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.12"]
+
+    steps:
+      - name: Check out repository code
+        uses: actions/checkout@v4
+      - name: Set up Apache Arrow
+        run: |
+          sudo apt update
+          sudo apt install -y -V ca-certificates lsb-release wget
+          wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
+          sudo apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
+          sudo apt update
+          sudo apt install -y -V libarrow-dev libarrow-glib-dev libparquet-dev libparquet-glib-dev
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install Olah
+        run: |
+          cd ${{ github.workspace }}
+          pip install --upgrade pip
+          pip install -e .
+
+      - name: Test Olah
+        run: |
+          cd ${{ github.workspace }}
+          python -m unittest discover olah/tests
+
+      - name: Build Olah
+        run: |
+          cd ${{ github.workspace }}
+          pip install build
+          python -m build
+
+      - name: Release
+        uses: "marvinpinto/action-automatic-releases@latest"
+        with:
+          repo_token: "${{ secrets.GITHUB_TOKEN }}"
+          prerelease: true
+          files: |
+            dist/*.tar.gz
+            dist/*.whl

README.md

@@ -3,6 +3,7 @@ Olah is self-hosted lightweight huggingface mirror service. `Olah` means `hello`
 
 Other languages: [中文](README_zh.md)
 ## Features
+* Huggingface Data Cache
 * Models mirror
 * Datasets mirror
 * Spaces mirror
@@ -42,21 +43,42 @@ python -m olah.server
 ```
 
 Then set the Environment Variable `HF_ENDPOINT` to the mirror site (Here is http://localhost:8090).
+
+Linux: 
 ```bash
 export HF_ENDPOINT=http://localhost:8090
 ```
 
+Windows Powershell:
+```bash
+$env:HF_ENDPOINT = "http://localhost:8090"
+```
+
 Starting from now on, all download operations in the HuggingFace library will be proxied through this mirror site.
 ```python
 from huggingface_hub import snapshot_download
 
 snapshot_download(repo_id='Qwen/Qwen-7B', repo_type='model',
                   local_dir='./model_dir', resume_download=True,
                   max_workers=8)
+```
 
+Or you can download models and datasets by using huggingface cli.
+```bash
+pip install -U huggingface_hub
 ```
 
-You can check the path `./repos` which stores all cached datasets and models.
+Download GPT2:
+```bash
+huggingface-cli download --resume-download openai-community/gpt2 --local-dir gpt2
+```
+
+Download WikiText:
+```bash
+huggingface-cli download --repo-type dataset --resume-download Salesforce/wikitext --local-dir wikitext
+```
+
+You can check the path `./repos`, in which olah stores all cached datasets and models.
 
 ## Start the server
 Run the command in the console: 
@@ -75,6 +97,8 @@ The default mirror cache path is `./repos`, you can change it by `--repos-path`
 python -m olah.server --host localhost --port 8090 --repos-path ./hf_mirrors
 ```
 
+**Note that the cached data between different versions cannot be migrated. Please delete the cache folder before upgrading to the latest version of Olah.**
+
 ## Future Work
 
 * Authentication

README_zh.md

@@ -1,8 +1,10 @@
 Olah是一种自托管的轻量级HuggingFace镜像服务。`Olah`在丘丘人语中意味着`你好`。
 
 ## 特性
+* 数据缓存，减少下载流量
 * 模型镜像
 * 数据集镜像
+* 空间镜像
 
 ## 安装
 
@@ -39,10 +41,16 @@ python -m olah.server
 ```
 
 然后将环境变量`HF_ENDPOINT`设置为镜像站点（这里是http://localhost:8090）。
+Linux: 
 ```bash
 export HF_ENDPOINT=http://localhost:8090
 ```
 
+Windows Powershell:
+```bash
+$env:HF_ENDPOINT = "http://localhost:8090"
+```
+
 从现在开始，HuggingFace库中的所有下载操作都将通过此镜像站点代理进行。
 ```python
 from huggingface_hub import snapshot_download
@@ -53,7 +61,22 @@ snapshot_download(repo_id='Qwen/Qwen-7B', repo_type='model',
 
 ```
 
-您可以检查存储所有缓存的数据集和模型的路径`./repos`。
+或者你也可以使用huggingface cli直接下载模型和数据集.
+```bash
+pip install -U huggingface_hub
+```
+
+下载GPT2:
+```bash
+huggingface-cli download --resume-download openai-community/gpt2 --local-dir gpt2
+```
+
+下载WikiText:
+```bash
+huggingface-cli download --repo-type dataset --resume-download Salesforce/wikitext --local-dir wikitext
+```
+
+您可以查看路径`./repos`，其中存储了所有数据集和模型的缓存。
 
 ## 启动服务器
 在控制台运行以下命令：
@@ -72,6 +95,8 @@ python -m olah.server --host localhost --port 8090
 python -m olah.server --host localhost --port 8090 --repos-path ./hf_mirrors
 ```
 
+**注意，不同版本之间的缓存数据不能迁移，请删除缓存文件夹后再进行olah的升级**
+
 ## 许可证
 
 olah采用MIT许可证发布。

olah/configs.py

@@ -6,6 +6,11 @@
 import fnmatch
 
 DEFAULT_PROXY_RULES = [
+    {
+        "repo": "*",
+        "allow": True,
+        "use_re": False
+    },
     {
         "repo": "*/*",
         "allow": True,
@@ -14,6 +19,11 @@
 ]
 
 DEFAULT_CACHE_RULES = [
+    {
+        "repo": "*",
+        "allow": True,
+        "use_re": False
+    },
     {
         "repo": "*/*",
         "allow": True,
@@ -87,7 +97,7 @@ def __init__(self, path: Optional[str] = None) -> None:
         self.mirror_lfs_url = "http://localhost:8090"
 
         # accessibility
-        self.offline = True
+        self.offline = False
         self.proxy = OlahRuleList.from_list(DEFAULT_PROXY_RULES)
         self.cache = OlahRuleList.from_list(DEFAULT_CACHE_RULES)
 
@@ -100,7 +110,7 @@ def empty_str(self, s: str) -> Optional[str]:
         else:
             return s
 
-    def read_toml(self, path: str):
+    def read_toml(self, path: str) -> None:
         config = toml.load(path)
 
         if "basic" in config:

olah/constants.py

@@ -1,5 +1,4 @@
 
-
 WORKER_API_TIMEOUT = 15
 CHUNK_SIZE = 4096
-LFS_FILE_BLOCK = 64 * 1024 * 1024
+LFS_FILE_BLOCK = 64 * 1024 * 1024