# FAQ

This document collects common issues in using and developing Triton-distributed-ascend, organized by category for quick look-up.

> **Note**: Sensitive information in this document has been replaced with placeholders. All parameters marked with `<>` must be replaced with actual values available on site.

## Port and Address Conventions

Both distributed launch and HCCL communication should use available free port ranges allowed by deployment, without hardcoding a specific port. The following uses 20000-29999 as an example range; the actual range depends on cluster network policy and CANN version:

```bash
export MASTER_PORT=<FREE_PORT_IN_20000_29999>
torchrun --master_port=<MASTER_PORT> --nproc-per-node=<WORLD_SIZE> <SCRIPT> <ARGS>
```

If the host has multiple network interfaces, select a network interface or IP that is reachable by all ranks; do not directly copy addresses from logs.

## Installation and Build

**Q: `clang++` compiler cannot be found**

The error message is as follows:

```text
CMake Error at CMakeLists.txt:45 (project):
The CMAKE_C_COMPILER:
  clang
is not a full path and was not found in the PATH.

CMake Error at CMakeLists.txt:45 (project):
The CMAKE_CXX_COMPILER:
  clang++
is not a full path and was not found in the PATH.

CMake Error at CMakeLists.txt:45 (project):
No CMAKE_ASM_COMPILER could be found.
```
**Solution**:

Download the corresponding version of clang++ from the LLVM official website and configure environment variables:

1. Download URL: <https://github.com/llvm/llvm-project/releases>
2. Extract after downloading the corresponding version
3. Configure environment variables. It is recommended to add the following command to `~/.bashrc` or `~/.zshrc` to make it permanent:

    ```bash
    export PATH=<LLVM_INSTALL_ROOT>/bin:$PATH
    ```

**Q: `torch.distributed.DistNetworkError` port occupied when starting distributed training**

The error message is as follows:

```text
File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/agent/server/api.py", line <LINE>, in _rendezvous
    rdzv_info = spec.rdzv_handler.next_rendezvous()
File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/rendezvous/static_tcp_rendezvous.py", line <LINE>, in next_rendezvous
    self._store = TCPStore(
torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: <PORT>, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
[ERROR] <TIMESTAMP> (PID:<PID>, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
```

**Solution**:

Specify a different port number in the `torchrun` command:

```bash
torchrun --master_port=<FREE_PORT_IN_20000_29999> ...
```

Or select any available port from the deployment-allowed free port range (e.g., 20000-29999).

**Q: `triton.backends.amd` module not found when importing triton**

The error message is as follows:

```text
File "<PYTHON_SITE_PACKAGES>/triton/backends/__init__.py", line <LINE>, in _discover_backends
    compiler = importlib.import_module(f"{value}.compiler")
File "<PYTHON_STDLIB>/importlib/__init__.py", line <LINE>, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'triton.backends.amd'
[ERROR] <TIMESTAMP> (PID:<PID>, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
```
**Reason**: The initial installation may have incomplete folder copying.

**Solution**:

Re-run pip install for triton-dist:

```bash
pip install -e .
```

**Q: `bishengir-opt` tool cannot be found**

The error message is as follows:

```text
File "<PYTHON_PREFIX>/lib/python<PYTHON_VERSION>/subprocess.py", line <LINE>, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'bishengir-opt'
```

**Solution**:

Add the directory of `bishengir-opt` to the environment variable. Replace the placeholder with the actual installation path. An example is as follows:

```bash
export PATH=<ASCEND_NPU_IR_ROOT>/build/bin:$PATH
```

**Q: `hivmc` tool cannot be found**

The error message is as follows:

```text
[ERROR][Triton][BEG]-
[ConvertLinalgIRToBinary] encounters error:
[ERROR] Cannot find hivmc under $PATH
[ERROR] Failed to run 'hivmc --version'
[WARNING] Failed to detect hivmc version for backward compatibility
[ERROR] Cannot find hivmc under $PATH
loc("<TMP_DIR>/kernel.mlir":<LINE>:<COLUMN>): error: External hivmc run fails, returning module before running external compiler
[ERROR] Failed to run BishengIR pipeline
```

**Solution**:

The CANN package version is incorrect. Download the correct CANN package from the official website:

1. Download URL: <https://www.hiascend.com/developer/download>
2. Download and install CANN 9.1.0 official version
3. Ensure correct sourcing of the environment script:

```bash
source <CANN_ROOT>/set_env.sh
```

**Q: Failed to enable listening for the NPU network adapter socket at startup**

The error message is as follows:

```text
[rank0]: RuntimeError: create_config:build/cMakeFiles/torch_npu.dir/compiler_depend.ts:149 HccL function error: hcclCommInitRootInfoConfig(numRanks, &rootInfo, rank, config, &(comm->hcclcomm_)), error code is 7
[rank0]: [ERROR] <TIMESTAMP> (PID:<PID>, Device:<DEVICE_ID>, RankID:0) ERR02200 DIST call hccl api failed
[PID:<PID>]<TIMESTAMP> Communication_Error_Bind_IP_Port (EI0020):
Failed to enable listening for the NPU network adapter socket. Reason: The IP address <IP_ADDRESS> and port <PORT> have already been bound.
[rank0]: Solution: Check whether the single-card multi-process scenario is used. If yes, configure the port number using the environment variable HCCL_NPU_SOCKET_PORT_RANGE.
```

**Reason**: The NPU socket port is occupied by another process.

**Solution**:

- **Option 1 (Recommended):** Set the environment variable for automatic port allocation:

  ```bash
  export HCCL_NPU_SOCKET_PORT_RANGE="auto"
  ```

  You can also specify the deployment-allowed port range:

  ```bash
  export HCCL_NPU_SOCKET_PORT_RANGE="<PORT_START>-<PORT_END>"
  ```

  The actual port range depends on cluster network policy and CANN version. All ranks should use a consistent port policy; if it still fails, also check the listening network interface and residual processes. It is recommended to add this command to the startup script.

- **Option 2:** Check device usage status, use `npu-smi info` to locate the occupying process, and then release the resources.

**Q: Compilation error: `'hivm.hir.custom' op operand count mismatch`**

The error message is as follows:

```text
[rank0]: subprocess.CalledProcessError: Command ['<CANN_ROOT>/bin/bishengir-opt', '<TMP_DIR>/kernel.mlirbc', '-o', '<TMP_DIR>/kernel.mlir'] returned non-zero exit status 1.
[rank0]: triton.compiler.errors.MLIRcompilationError:
[ERROR][Triton][BEG]--
[BytecodeToLinalgIRByBishengiropt] encounters error:
<SOURCE_ROOT>/tutorials/ascend/allgather-gemm.py:<LINE>:<COLUMN>: error: 'hivm.hir.custom' op operand count (2) does not match with the total size (0) specified in attribute 'operandSegmentSizes'
```

**Solution**:

The AscendNPU-IR path is incorrectly configured. Re-source the environment variables:

```bash
source <ASCEND_NPU_IR_ROOT>/build/setup.sh
```

**Q: `unsupport operand type(s) for` error at runtime**

The error message is as follows:

```text
File "<PYTHON_SITE_PACKAGES>/triton_dist/language/extra/language_extra.py", line <LINE>, in <module>
    from triton_dist.utils import is_ascend, is_cuda, is_hip, is_maca
def rand_tensor(shape, dtype: torch.dtype, device: torch.device = "cuda" if type ...
[ERROR] <TIMESTAMP> (PID:<PID>, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
```

**Solution**:

The Python version is too low. It is recommended to use Python 3.11 or higher:

```bash
python3.11 -m venv venv
source venv/bin/activate
```

**Q: `missing: LLVMCGData` when compiling triton-distributed-ascend**

The error message is as follows:

```text
CMake Error at 3rdparty/triton-ascend/third_party/amd/CMakeLists.txt:6 (find_package):
Found package configuration file:
<LLVM_INSTALL_ROOT>/lib/cmake/lld/LLDConfig.cmake
but it set LLD_FOUND to FALSE so package "LLD" is considered to be NOT FOUND.
Reason given by package:
The following imported targets are referenced, but are missing: LLVMCGData
Configuring incomplete, errors occurred!
```

**Solution**:

This is caused by build residue. Clean and rebuild:

```bash
rm -rf build/
mkdir build && cd build
cmake ..
make -j$(nproc)
```

**Q: Linking fails when building triton_dist**

**Error message 1 (cmake_args configuration):**

```python
if check_env_flag("TRITON_BUILD_WITH_CLANG_LLD"):
    cmake_args += [
        "-DCMAKE_C_COMPILER=clang",
        "-DCMAKE_CXX_COMPILER=clang++",
        "-DCMAKE_LINKER=lld",
        "-DCMAKE_EXE_LINKER_FLAGS=-fuse-ld=lld",
        "-DCMAKE_MODULE_LINKER_FLAGS=-fuse-ld=lld",
        "-DCMAKE_EXE_LINKER_FLAGS=-L<SYSTEM_LIBRARY_DIR>",
        "-DCMAKE_SHARED_LINKER_FLAGS=-L<SYSTEM_LIBRARY_DIR>"
    ]
```

**Error message 2 (link failure log):**

```text
FAILED: 3rdparty/triton-ascend/test/lib/Proton/libTritonTestProton.a
<SHELL>: <LLVM_INSTALL_ROOT>/bin/llvm-ar: not found
ninja: build stopped: subcommand failed.
```

**Solution**:

1. Delete the build directory
2. Add the following link parameters to `cmake_args` in `setup.py`:

    ```python
    cmake_args = [
        "-DCMAKE_EXE_LINKER_FLAGS=-L<SYSTEM_LIBRARY_DIR>",
        "-DCMAKE_SHARED_LINKER_FLAGS=-L<SYSTEM_LIBRARY_DIR>"
    ]
    ```

3. Rebuild

**Q: Error when creating conda environment, reporting channel corruption**

The error message is as follows:

```text
Channels:
defaults
Platform: linux-64
Collecting package metadata (repodata.json): failed
UnavailableInvalidChannel: HTTP 404 Not Found for channel anaconda/pkgs/r <https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r>
The channel is not accessible or is invalid.
You will need to adjust your conda configuration to proceed.
Use `conda config --show channels` to view your configuration's current state,
and use `conda config --show-sources` to view config file locations.
```

**Solution**:

1. Remove the corrupted channel:

    ```bash
    conda config --remove channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
    conda config --remove default_channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
    ```

2. Configure the pip mirror source:

    ```bash
    mkdir -p ~/.pip
    cat > ~/.pip/pip.conf << 'EOF'
    [global]
    index-url = http://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
    extra-index-url = http://mirror.nju.edu.cn/pytorch/whl/cpu
    trusted-host =
        mirrors.tuna.tsinghua.edu.cn
        mirror.nju.edu.cn
    EOF
    ```

3. Recreate the conda environment

## Compilation

**Q: `test/ascend/test_*.py` tests pass, but `tutorials/ascend/01-*.py` runtime error reports `undefined symbol`**

**Problem description:** The `test/ascend/test_*.py` tests pass, but running `tutorials/ascend/01-*.py` reports an error indicating that the symbol cannot be found.

**Error message 1 (link error):**

```text
[rank1]: ld.lld: error: undefined symbol:
  mlir_ciface_aclshmem_barrier_all.cube
[rank1]: >>> referenced by LLVMDialectModule
[rank1]: >>> <TMP_DIR>/kernel.o:(kernel_allgather_gemm_mix_aic)
[rank1]: ld.lld: error: undefined symbol:
  mlir_ciface_aclshmem_ptr_half.cube
[rank1]: >>> referenced by LLVMDialectModule
[rank1]: >>> <TMP_DIR>/kernel.o:(aclshmem_ptr_half.cube)
[rank1]: [ERROR] Executing: <CANN_ROOT>/tools/ccec_compiler/bin/ld.lld -maicorelinux -Ttext ...
[rank1]: loc("<TMP_DIR>/module.hivm.opt.mlir":2:1): error: Failed to compile BishengIR for device
[rank1]: [ERROR] Failed to run HIVMC pipeline
```

**Error message 2 (version compatibility issue):**

```text
Failed: Distributed worker error:
[rank 0]
[ERROR][Triton][BEG]---
[ConvertTritonIRToLinalgIR] encounters error:
add_convert_triton_distributed_to_hivm(): incompatible function arguments. The following argument types are supported:
1. (arg0: triton._C.libtriton.ir.pass_manager, arg1: str) -> None
Invoked with: <triton._C.libtriton.ir.pass_manager object at 0xfffcf8f67eb0>
```

**Reason**: The built `bishengir-compile` has compatibility issues.

**Solution**:

- **Option 1 (Recommended):** Use the official daily packages to avoid self-compilation.
- **Option 2:** If you must self-compile, ensure the following conditions are met:
    - Add the `--build-shmem-template` parameter when building:

      ```bash
      ./build-tools/build.sh \
        --build-shmem-template \
        -o ./build \
        -t \
        --build-type Release \
        --apply-patches \
        --bisheng-compiler=<CANN_ROOT>/bin
      ```

    - Ensure you are using your self-compiled `bishengir-compile`:

      ```bash
      export TRITON_DEBUG=1
      ```

      Execute the above command before running `.py` scripts to view the actual `bishengir-compile` path being used.

    - Ensure your self-compiled `bishengir-compile` is compatible with the `hivmc` version.

## Runtime

**Q: `npu function error: device error type 0, error code is 4` at runtime**

The error message is as follows:

```text
[rank1]: File "<PYTHON_SITE_PACKAGES>/torch/distributed/distributed_c10d.py", line <LINE>, in barrier
[rank1]: work = group.barrier(opts=opts)
[rank1]: RuntimeError: create_config: <TORCH_NPU_ROOT>/csrc/distributed/HcCLUtils.cpp:<LINE> NPU function error: device error type 0, error code is 4
```

**Solution**:

Update to the CANN 9.1.0 official version. Reference: the corresponding CANN version's public release notes and project support matrix.

**Q: Port occupied when starting torchrun**

The error message is as follows:

```text
File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/metrics/api.py", line <LINE>, in wrapper
    result = f(*args, **kwargs)
File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/agent/server/api.py", line <LINE>, in _rendezvous
    rdzv_info = spec.rdzv_handler.next_rendezvous()
File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/rendezvous/static_tcp_rendezvous.py", line <LINE>, in next_rendezvous
    self._store = TCPStore(
torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: <PORT>, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
[ERROR] <TIMESTAMP> (PID:<PID>, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
```

**Solution**:

Add the `--master_port` parameter after the `torchrun` command to specify a free port. The `--master_port` parameter must immediately follow the `torchrun` command; the port must be chosen from a free range allowed by the deployment environment, such as 20000-29999.

```bash
torchrun --master_port=<FREE_PORT_IN_20000_29999> ...
```

**Q: Card failure at runtime, NPU card cannot work normally**

The error message is as follows:

```text
[rank0]: RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is HcclAllreduce.
[rank0]: Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[rank0]: [ERROR] <TIMESTAMP> (PID:<PID>, Device:0, RankID:0) ERR00100 PTA call acl api failed.
[PID:<PID>]<TIMESTAMP> Communication_Error_Bind_IP_Port(EI0020): Failed to enable listening for the NPU network adapter socket. Reason: The configured IP address and port <PORT> have already been bound.
[rank0]: Solution: Check whether the single-card multi-process scenario is used. If yes, configure the port number using the environment variable HCCL_NPU_SOCKET_PORT_RANGE.
```

**Solution**:

- **Option 1:** Set the environment variable for automatic port allocation:

  ```bash
  export HCCL_NPU_SOCKET_PORT_RANGE="auto"
  ```

  You can also explicitly specify the deployment-allowed port range. The port range should comply with the current cluster network policy and CANN version requirements, and ensure it is not occupied by other processes.

  ```bash
  export HCCL_NPU_SOCKET_PORT_RANGE="<PORT_START>-<PORT_END>"
  ```

- **Option 2:** Reset the NPU card, replacing `${CARD_ID}` with the actual card number.

  ```bash
  npu-smi set -t reset -i ${CARD_ID} -m 1
  ```

**Q: Runtime reports `AclOpKernelInit failed`**

The error message is as follows:

```text
<USER>@<HOST> $ HCCL_NPU_SOCKET_PORT_RANGE="auto" torchrun --nproc-per-node=2 <SCRIPT>
...
[rank0]: RuntimeError: ones: <OP_PLUGIN_ROOT>/ops/opapi/OnesKernelNpuOpApi.cpp:<LINE> NPU function error: device error type 0, error code is 561103
[rank0]: [ERROR] <TIMESTAMP> (PID:<PID>, Device:0, RankID:0) ERR00100 PTA call acl api failed.
[rank0]: EZ9999: Inner Error!
[rank0]: EZ9999 [PID:<PID>] <TIMESTAMP> AclNN_Inner_Error(EZ9999): Parse dynamic kernel config fail.
[rank0]: TraceBack (most recent call last):
[rank0]: Failed to ParseDynamicKernels.
[rank0]: AclOpKernelInit failed opType. Maybe caused by the following reasons:
[rank0]: 1. Please check whether the CANN enviroment variables are sourced.
[rank0]: 2. Please check if the kernel package is installed.
[rank0]: 3. Please confirm whether the current chip or version already supports the faulty operator.
[rank0]: OnesLike ADD_TO_LAUNCHER_LIST_AICORE failed.
```

**Possible reasons**:

1. CANN is not correctly installed
2. `source set_env.sh` was not executed to configure the environment.
3. The CANN version does not match the hardware. For example, a CANN installation package for Atlas A3 products and Atlas A2 products was installed on Ascend 950PR&950DT products.

**Solution**:

1. Check whether the CANN version matches the hardware.
2. Ensure the environment is configured correctly:

    ```bash
    source <CANN_ROOT>/set_env.sh
    ```

3. Reinstall the CANN version that matches the hardware.

## Issue Feedback

**Q: How to report an issue?**

When submitting an issue, please provide the key error messages, software versions, chip model, number of ranks, shape, dtype, reproduction commands, and solutions already attempted.

> **Note**: Before submitting, remove sensitive information (for example, usernames, employee IDs, hostnames, IP/MAC addresses, PIDs, internal paths, access tokens, and business data).

## Quick Reference for Common Commands

### Environment Variable Configuration Commands

```bash
# clang++ path
export PATH=<LLVM_INSTALL_ROOT>/bin:$PATH

# AscendNPU-IR path
export PATH=<ASCEND_NPU_IR_ROOT>/build/bin:$PATH

# NPU socket port auto allocation
export HCCL_NPU_SOCKET_PORT_RANGE="auto"

# Or specify deployment-allowed port range
export HCCL_NPU_SOCKET_PORT_RANGE="<PORT_START>-<PORT_END>"

# Debug mode
export TRITON_DEBUG=1
```

### Common Tool Commands

```bash
# View NPU status
npu-smi info

# Reset specified NPU card
npu-smi set -t reset -i ${CARD_ID} -m 1

# Start distributed task (select free port from deployment-allowed range)
torchrun --master_port=<FREE_PORT_IN_20000_29999> ...

# Reinstall triton-dist
pip install -e .
```

### Clean and Rebuild Commands

```bash
# Clean build directory
rm -rf build/

# Create and enter build directory
mkdir build && cd build

# Configure CMake (adjust according to actual system library directory)
cmake -DCMAKE_EXE_LINKER_FLAGS="-L<SYSTEM_LIBRARY_DIR>" ..

# Compile
make -j$(nproc)
```