# FAQ 文档

> 脱敏说明：文档中的用户名、工号、主机名、IP、PID、内部路径和内部链接均已替换为占位符。示例中的 <PORT_START>-<PORT_END>、<MASTER_PORT> 等参数必须替换为现场可用值。

### 端口与地址约定

分布式启动和 HCCL 通信均应使用部署允许的空闲端口区间，不固化某一个端口。以下以 20000-29999 为示例区间，实际范围以集群网络策略和 CANN 版本为准：

~~~bash
export MASTER_PORT=<FREE_PORT_IN_20000_29999>
torchrun --master_port=<MASTER_PORT> --nproc-per-node=<WORLD_SIZE> <SCRIPT> <ARGS>
~~~

如果主机存在多张网卡，应选择所有 rank 均可达的网卡或 IP；不能直接复制日志中的地址。

## 目录

- [一、安装构建篇](#一安装构建篇)
  - [1.1 clang++ 缺失](#11-clangxx-缺失)
  - [1.2 torch.distributed.DistNetworkError 端口占用](#12-torchdistributeddistnetworkerror-端口占用)
  - [1.3 No module named 'triton.backends.amd'](#13-no-module-named-tritonbackendsamd)
  - [1.4 FileNotFoundError: bishengir-opt](#14-filenotfounderror-bishengiropt)
  - [1.5 Cannot find hivmc under $PATH](#15-cannot-find-hivmc-under-path)
  - [1.6 Failed to enable listening for the NPU network adapter socket](#16-failed-to-enable-listening-for-the-npu-network-adapter-socket)
  - [1.7 error: 'hivm.hir.custom' op operand count mismatch](#17-error-hivmhircustom-op-operand-count-mismatch)
  - [1.8 unsupport operand type(s) for](#18-unsupport-operand-types-for)
  - [1.9 The following imported targets are referenced, but are missing: LLVMCGData](#19-the-following-imported-targets-are-referenced-but-are-missing-llvmcgdata)
  - [1.10 构建 triton_dist 时报错](#110-构建-triton_dist-时报错)
  - [1.11 创建 conda 环境失败](#111-创建-conda-环境失败)
- [二、编译篇](#二编译篇)
  - [2.1 undefined symbol](#21-undefined-symbol)
- [三、运行时篇](#三运行时篇)
  - [3.1 npu function error: device error type 0, error code is 4](#31-npu-function-error-device-error-type-0-error-code-is-4)
  - [3.2 端口占用](#32-端口占用)
  - [3.3 掉卡](#33-掉卡)
  - [3.4 AclOpKernelInit failed](#34-aclopkernelinit-failed)
- [四、精度问题篇](#四精度问题篇)
- [五、性能问题篇](#五性能问题篇)
- [六、附录：常用命令速查](#六附录常用命令速查)

---

## 一、安装构建篇

### 1.1 clang++ 缺失

**问题描述：**
编译过程中提示找不到 `clang++` 编译器。

**报错信息：**

```
CMake Error at CMakeLists.txt:45 (project):
The CMAKE_C_COMPILER:
  clang
is not a full path and was not found in the PATH.

CMake Error at CMakeLists.txt:45 (project):
The CMAKE_CXX_COMPILER:
  clang++
is not a full path and was not found in the PATH.

CMake Error at CMakeLists.txt:45 (project):
No CMAKE_ASM_COMPILER could be found.
```

**解决方案:**  从 LLVM 官网下载对应版本的 clang++ 并配置环境变量：

1. 下载地址：https://github.com/llvm/llvm-project/releases
2. 下载对应版本后解压
3. 配置环境变量：

```bash
export PATH=<LLVM_INSTALL_ROOT>/bin:$PATH
```

> 建议将上述命令添加到 `~/.bashrc` 或 `~/.zshrc` 中永久生效。

---

### 1.2 torch.distributed.DistNetworkError 端口占用

**问题描述：**
启动分布式训练时出现以下错误：

**报错信息：**

```
File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/agent/server/api.py", line <LINE>, in _rendezvous
    rdzv_info = spec.rdzv_handler.next_rendezvous()
File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/rendezvous/static_tcp_rendezvous.py", line <LINE>, in next_rendezvous
    self._store = TCPStore(
torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: <PORT>, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
[ERROR] <TIMESTAMP> (PID:<PID>, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
```

**解决方案：**
在 `torchrun` 命令中指定其他端口号：

```bash
torchrun --master_port=<FREE_PORT_IN_20000_29999> ...
```

或者从部署允许的空闲端口区间（例如 20000-29999）选择任意未被占用的端口。

---

### 1.3 No module named 'triton.backends.amd'

**问题描述：**
导入 triton 模块时报错：

**报错信息：**

```
File "<PYTHON_SITE_PACKAGES>/triton/backends/__init__.py", line <LINE>, in _discover_backends
    compiler = importlib.import_module(f"{value}.compiler")
File "<PYTHON_STDLIB>/importlib/__init__.py", line <LINE>, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'triton.backends.amd'
[ERROR] <TIMESTAMP> (PID:<PID>, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
```

**解决方案：**
重新执行 triton-dist 的 pip install 安装：

```bash
pip install -e .
```

> 原因：首次安装时可能存在文件夹拷贝不完整的情况。

---

### 1.4 FileNotFoundError: bishengir-opt

**问题描述：**
执行命令时提示找不到 `bishengir-opt` 工具。

**报错信息：**

```
File "<PYTHON_PREFIX>/lib/python<PYTHON_VERSION>/subprocess.py", line <LINE>, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'bishengir-opt'
```

**解决方案：**
配置 `bishengir-opt` 所在路径到环境变量：

```bash
export PATH=<ASCEND_NPU_IR_ROOT>/build/bin:$PATH
```

> 请根据实际安装路径调整。

---

### 1.5 Cannot find hivmc under $PATH

**问题描述：**
系统提示找不到 `hivmc` 工具。

**报错信息：**

```
[ERROR][Triton][BEG]-
[ConvertLinalgIRToBinary] encounters error:
[ERROR] Cannot find hivmc under $PATH
[ERROR] Failed to run 'hivmc --version'
[WARNING] Failed to detect hivmc version for backward compatibility
[ERROR] Cannot find hivmc under $PATH
loc("<TMP_DIR>/kernel.mlir":<LINE>:<COLUMN>): error: External hivmc run fails, returning module before running external compiler
[ERROR] Failed to run BishengIR pipeline
```

**解决方案:** CANN 包版本不对，需从官网下载正确的 CANN 包：

1. 下载地址：https://www.hiascend.com/developer/download
2. 下载并安装 CANN 9.1.0 正式版本
3. 确保正确 source 环境脚本：

```bash
source <CANN_ROOT>/set_env.sh
```

---

### 1.6 Failed to enable listening for the NPU network adapter socket

**问题描述：**
启动时提示 NPU network adapter socket 无法监听。

**报错信息：**

```
[rank0]: RuntimeError: create_config:build/cMakeFiles/torch_npu.dir/compiler_depend.ts:149 HccL function error: hcclCommInitRootInfoConfig(numRanks, &rootInfo, rank, config, &(comm->hcclcomm_)), error code is 7
[rank0]: [ERROR] <TIMESTAMP> (PID:<PID>, Device:<DEVICE_ID>, RankID:0) ERR02200 DIST call hccl api failed
[PID:<PID>]<TIMESTAMP> Communication_Error_Bind_IP_Port (EI0020):
Failed to enable listening for the NPU network adapter socket. Reason: The IP address <IP_ADDRESS> and port <PORT> have already been bound.
[rank0]: Solution: Check whether the single-card multi-process scenario is used. If yes, configure the port number using the environment variable HCCL_NPU_SOCKET_PORT_RANGE.
```

**原因：**
NPU socket 端口被其他进程占用。

**解决方案 1：**
查看占用情况：

```bash
npu-smi info
```

找到占用进程并释放资源。

**解决方案 2（推荐）：**
设置环境变量自动分配端口：

```bash
export HCCL_NPU_SOCKET_PORT_RANGE="auto"
```

也可以指定部署允许的端口区间：

```bash
export HCCL_NPU_SOCKET_PORT_RANGE="<PORT_START>-<PORT_END>"
```

> 实际端口区间以集群网络策略和 CANN 版本为准。所有 rank 应使用一致的端口策略；如果仍然失败，请同时检查监听网卡和残留进程。

> 建议将此命令添加到启动脚本中。

---

### 1.7 error: 'hivm.hir.custom' op operand count mismatch

**问题描述：**
编译时报错 operand count 不匹配。

**报错信息：**

```
[rank0]: subprocess.CalledProcessError: Command ['<CANN_ROOT>/bin/bishengir-opt', '<TMP_DIR>/kernel.mlirbc', '-o', '<TMP_DIR>/kernel.mlir'] returned non-zero exit status 1.
[rank0]: triton.compiler.errors.MLIRcompilationError:
[ERROR][Triton][BEG]--
[BytecodeToLinalgIRByBishengiropt] encounters error:
<SOURCE_ROOT>/tutorials/ascend/01-ascend-allgather-gemm.py:<LINE>:<COLUMN>: error: 'hivm.hir.custom' op operand count (2) does not match with the total size (0) specified in attribute 'operandSegmentSizes'
```

**解决方案：**
AscendNPU-IR 路径配置不正确，重新 source 环境变量：

```bash
source <ASCEND_NPU_IR_ROOT>/build/setup.sh
```

---

### 1.8 unsupport operand type(s) for

**问题描述：**
运行时出现运算符不支持的错误。

**报错信息：**

```
File "<PYTHON_SITE_PACKAGES>/triton_dist/language/extra/language_extra.py", line <LINE>, in <module>
    from triton_dist.utils import is_ascend, is_cuda, is_hip, is_maca
def rand_tensor(shape, dtype: torch.dtype, device: torch.device = "cuda" if type ...
[ERROR] <TIMESTAMP> (PID:<PID>, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
```

**解决方案：**
Python 版本过低，建议使用 Python 3.11 或更高版本：

```bash
python3.11 -m venv venv
source venv/bin/activate
```

---

### 1.9 The following imported targets are referenced, but are missing: LLVMCGData

**问题描述：**
编译 triton-distributed-ascend 时提示缺少 LLVMCGData 目标。

**报错信息：**

```
CMake Error at 3rdparty/triton-ascend/third_party/amd/CMakeLists.txt:6 (find_package):
Found package configuration file:
<LLVM_INSTALL_ROOT>/lib/cmake/lld/LLDConfig.cmake
but it set LLD_FOUND to FALSE so package "LLD" is considered to be NOT FOUND.
Reason given by package:
The following imported targets are referenced, but are missing: LLVMCGData
Configuring incomplete, errors occurred!
```

**解决方案：**
构建残留导致，清理后重新构建：

```bash
rm -rf build/
mkdir build && cd build
cmake ..
make -j$(nproc)
```

---

### 1.10 构建 triton_dist 时报错

**问题描述：**
构建 triton_dist 时链接失败。

**报错信息 1（cmake_args 配置）：**

```python
if check_env_flag("TRITON_BUILD_WITH_CLANG_LLD"):
    cmake_args += [
        "-DCMAKE_C_COMPILER=clang",
        "-DCMAKE_CXX_COMPILER=clang++",
        "-DCMAKE_LINKER=lld",
        "-DCMAKE_EXE_LINKER_FLAGS=-fuse-ld=lld",
        "-DCMAKE_MODULE_LINKER_FLAGS=-fuse-ld=lld",
        "-DCMAKE_EXE_LINKER_FLAGS=-L<SYSTEM_LIBRARY_DIR>",
        "-DCMAKE_SHARED_LINKER_FLAGS=-L<SYSTEM_LIBRARY_DIR>"
    ]
```

**报错信息 2（链接失败）：**

```
FAILED: 3rdparty/triton-ascend/test/lib/Proton/libTritonTestProton.a
<SHELL>: <LLVM_INSTALL_ROOT>/bin/llvm-ar: not found
ninja: build stopped: subcommand failed.
```

**解决方案：**

1. 删除 build 目录
2. 在 `setup.py` 的 `cmake_args` 中添加以下链接参数：

```python
cmake_args = [
    "-DCMAKE_EXE_LINKER_FLAGS=-L<SYSTEM_LIBRARY_DIR>",
    "-DCMAKE_SHARED_LINKER_FLAGS=-L<SYSTEM_LIBRARY_DIR>"
]
```

3. 重新构建

---

### 1.11 创建 conda 环境失败

**问题描述：**
创建 conda 环境时报错，提示 channel 损坏。

**报错信息：**

```
Channels:
defaults
Platform: linux-64
Collecting package metadata (repodata.json): failed
UnavailableInvalidChannel: HTTP 404 Not Found for channel anaconda/pkgs/r <https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r>
The channel is not accessible or is invalid.
You will need to adjust your conda configuration to proceed.
Use `conda config --show channels` to view your configuration's current state,
and use `conda config --show-sources` to view config file locations.
```

**解决方案：**

1. 移除损坏的 channel：

```bash
conda config --remove channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
conda config --remove default_channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
```

2. 配置 pip 镜像源：

```bash
mkdir -p ~/.pip
cat > ~/.pip/pip.conf << 'EOF'
[global]
index-url = http://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
extra-index-url = http://mirror.nju.edu.cn/pytorch/whl/cpu
trusted-host =
    mirrors.tuna.tsinghua.edu.cn
    mirror.nju.edu.cn
EOF
```

3. 重新创建 conda 环境

---

## 二、编译篇

### 2.1 undefined symbol

**问题描述：**
`test/ascend/test_*.py` 测试通过，但 `tutorials/ascend/01-*.py` 运行时报错，提示找不到符号。

**报错信息 1（链接错误）：**

```
[rank1]: ld.lld: error: undefined symbol:
  mlir_ciface_aclshmem_barrier_all.cube
[rank1]: >>> referenced by LLVMDialectModule
[rank1]: >>> <TMP_DIR>/kernel.o:(kernel_allgather_gemm_mix_aic)
[rank1]: ld.lld: error: undefined symbol:
  mlir_ciface_aclshmem_ptr_half.cube
[rank1]: >>> referenced by LLVMDialectModule
[rank1]: >>> <TMP_DIR>/kernel.o:(aclshmem_ptr_half.cube)
[rank1]: [ERROR] Executing: <CANN_ROOT>/tools/ccec_compiler/bin/ld.lld -maicorelinux -Ttext ...
[rank1]: loc("<TMP_DIR>/module.hivm.opt.mlir":2:1): error: Failed to compile BishengIR for device
[rank1]: [ERROR] Failed to run HIVMC pipeline
```

**报错信息 2（版本兼容性问题）：**

```
Failed: Distributed worker error:
[rank 0]
[ERROR][Triton][BEG]---
[ConvertTritonIRToLinalgIR] encounters error:
add_convert_triton_distributed_to_hivm(): incompatible function arguments. The following argument types are supported:
1. (arg0: triton._C.libtriton.ir.pass_manager, arg1: str) -> None
Invoked with: <triton._C.libtriton.ir.pass_manager object at 0xfffcf8f67eb0>
```

**背景：**
自己构建的 `bishengir-compile` 存在兼容性问题。

**解决方案：**

**方案 1（推荐）：**
使用官方 daily 包，避免自行编译。

**方案 2：**
如果必须自行编译，确保满足以下条件：

a. 构建时添加 `--build-shmem-template` 参数：

```bash
./build-tools/build.sh \
  --build-shmem-template \
  -o ./build \
  -t \
  --build-type Release \
  --apply-patches \
  --bisheng-compiler=<CANN_ROOT>/bin
```

b. 确保使用了自己编译的 `bishengir-compile`：

```bash
export TRITON_DEBUG=1
```

运行 `.py` 脚本前执行上述命令，可查看实际使用的 `bishengir-compile` 路径。

c. 确保自行编译的 `bishengir-compile` 与 `hivmc` 版本兼容。

---

## 三、运行时篇

### 3.1 npu function error: device error type 0, error code is 4

**问题描述：**
运行时出现 NPU 设备错误。

**报错信息：**

```
[rank1]: File "<PYTHON_SITE_PACKAGES>/torch/distributed/distributed_c10d.py", line <LINE>, in barrier
[rank1]: work = group.barrier(opts=opts)
[rank1]: RuntimeError: create_config: <TORCH_NPU_ROOT>/csrc/distributed/HcCLUtils.cpp:<LINE> NPU function error: device error type 0, error code is 4
```

**解决方案：**
更新到 CANN 9.1.0 正式版本：

参考：对应 CANN 版本的公开发行说明和项目支持矩阵。

---

### 3.2 端口占用

**问题描述：**
torch-run 启动时端口被占用。

**报错信息：**

```
File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/metrics/api.py", line <LINE>, in wrapper
    result = f(*args, **kwargs)
File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/agent/server/api.py", line <LINE>, in _rendezvous
    rdzv_info = spec.rdzv_handler.next_rendezvous()
File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/rendezvous/static_tcp_rendezvous.py", line <LINE>, in next_rendezvous
    self._store = TCPStore(
torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: <PORT>, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
[ERROR] <TIMESTAMP> (PID:<PID>, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
```

**解决方案：**
在 `torch-run` 命令后添加 `--master_port` 参数指定其他端口：

```bash
torch-run --master_port=<FREE_PORT_IN_20000_29999> ...
```

> 注意：`--master_port` 参数需紧挨着 `torch-run` 命令添加。端口应从部署允许的空闲区间（例如 20000-29999）中选择。

---

### 3.3 掉卡

**问题描述：**
运行时出现掉卡现象，NPU 卡无法正常工作。

**报错信息：**

```
[rank0]: RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is HcclAllreduce.
[rank0]: Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[rank0]: [ERROR] <TIMESTAMP> (PID:<PID>, Device:0, RankID:0) ERR00100 PTA call acl api failed.
[PID:<PID>]<TIMESTAMP> Communication_Error_Bind_IP_Port(EI0020): Failed to enable listening for the NPU network adapter socket. Reason: The configured IP address and port <PORT> have already been bound.
[rank0]: Solution: Check whether the single-card multi-process scenario is used. If yes, configure the port number using the environment variable HCCL_NPU_SOCKET_PORT_RANGE.
```

**解决方案 1：**
设置环境变量自动分配端口：

```bash
export HCCL_NPU_SOCKET_PORT_RANGE="auto"
```

也可以显式指定部署允许的端口区间：

```bash
export HCCL_NPU_SOCKET_PORT_RANGE="<PORT_START>-<PORT_END>"
```

> 端口区间应符合当前集群网络策略和 CANN 版本要求，并确保未被其他进程占用。

**解决方案 2：**
重置 NPU 卡：

```bash
npu-smi set -t reset -i ${CARD_ID} -m 1
```

> 将 `${CARD_ID}` 替换为实际的卡号。

---

### 3.4 AclOpKernelInit failed

**问题描述：**
运行时提示 AclOpKernelInit 失败。

**报错信息：**

```
<USER>@<HOST> $ HCCL_NPU_SOCKET_PORT_RANGE="auto" torchrun --nproc-per-node=2 <SCRIPT>
...
[rank0]: RuntimeError: ones: <OP_PLUGIN_ROOT>/ops/opapi/OnesKernelNpuOpApi.cpp:<LINE> NPU function error: device error type 0, error code is 561103
[rank0]: [ERROR] <TIMESTAMP> (PID:<PID>, Device:0, RankID:0) ERR00100 PTA call acl api failed.
[rank0]: EZ9999: Inner Error!
[rank0]: EZ9999 [PID:<PID>] <TIMESTAMP> AclNN_Inner_Error(EZ9999): Parse dynamic kernel config fail.
[rank0]: TraceBack (most recent call last):
[rank0]: Failed to ParseDynamicKernels.
[rank0]: AclOpKernelInit failed opType. Maybe caused by the following reasons:
[rank0]: 1. Please check whether the CANN enviroment variables are sourced.
[rank0]: 2. Please check if the kernel package is installed.
[rank0]: 3. Please confirm whether the current chip or version already supports the faulty operator.
[rank0]: OnesLike ADD_TO_LAUNCHER_LIST_AICORE failed.
```

**可能原因：**

1. CANN 未正确安装
2. 未 source `set_env.sh`
3. CANN 版本与硬件不匹配（如在 A5 机器上安装了 A3/A2 的 CANN 包）

**解决方案：**

1. 检查 CANN 版本是否与硬件匹配
2. 确保正确 source 环境脚本：

```bash
source <CANN_ROOT>/set_env.sh
```

3. 重新安装正确的 CANN 版本

---

## 四、精度问题篇

---

## 五、性能问题篇

---

## 六、附录：常用命令速查

### 环境变量配置

```bash
# clang++ 路径
export PATH=<LLVM_INSTALL_ROOT>/bin:$PATH

# AscendNPU-IR 路径
export PATH=<ASCEND_NPU_IR_ROOT>/build/bin:$PATH

# NPU socket 端口自动分配
export HCCL_NPU_SOCKET_PORT_RANGE="auto"

# 或指定部署允许的端口区间
export HCCL_NPU_SOCKET_PORT_RANGE="<PORT_START>-<PORT_END>"

# 调试模式
export TRITON_DEBUG=1
```

### 常用工具命令

```bash
# 查看 NPU 状态
npu-smi info

# 重置指定 NPU 卡
npu-smi set -t reset -i ${CARD_ID} -m 1

# 启动分布式任务（从部署允许的区间选择空闲端口）
torchrun --master_port=<FREE_PORT_IN_20000_29999> ...

# 重新安装 triton-dist
pip install -e .
```

### 清理重建

```bash
# 清理构建目录
rm -rf build/

# 创建并进入构建目录
mkdir build && cd build

# 配置 CMake（根据实际系统库目录调整）
cmake -DCMAKE_EXE_LINKER_FLAGS="-L<SYSTEM_LIBRARY_DIR>" ..

# 编译
make -j$(nproc)
```

### 问题反馈

提交问题时请提供脱敏后的关键报错、软件版本、芯片型号、rank 数、shape、dtype、复现命令和已尝试方案。

提交前请删除用户名、工号、主机名、IP/MAC、PID、内部路径、访问令牌和业务数据。