FAQ Documentation

Privacy Notice: Usernames, employee IDs, hostnames, IPs, PIDs, internal paths, and internal links in this document have been replaced with placeholders. Parameters such as <PORT_START>-<PORT_END>, <MASTER_PORT> in examples must be replaced with actual values available on site.

Port and Address Conventions

Both distributed launch and HCCL communication should use available free port ranges allowed by deployment, without hardcoding a specific port. The following uses 20000-29999 as an example range; the actual range depends on cluster network policy and CANN version:

export MASTER_PORT=<FREE_PORT_IN_20000_29999>
torchrun --master_port=<MASTER_PORT> --nproc-per-node=<WORLD_SIZE> <SCRIPT> <ARGS>

If the host has multiple network interfaces, select a network interface or IP that is reachable by all ranks; do not directly copy addresses from logs.

Table of Contents

• I. Installation and Build

  • 1.1 Missing clang++

  • 1.2 torch.distributed.DistNetworkError Port Conflict

  • 1.3 No module named ‘triton.backends.amd’

  • 1.4 FileNotFoundError: bishengir-opt

  • 1.5 Cannot find hivmc under $PATH

  • 1.6 Failed to enable listening for the NPU network adapter socket

  • 1.7 error: ‘hivm.hir.custom’ op operand count mismatch

  • 1.8 unsupport operand type(s) for

  • 1.9 The following imported targets are referenced, but are missing: LLVMCGData

  • 1.10 Error building triton_dist

  • 1.11 Failed to create conda environment

• II. Compilation

  • 2.1 undefined symbol

• III. Runtime

  • 3.1 npu function error: device error type 0, error code is 4

  • 3.2 Port conflict

  • 3.3 Card failure

  • 3.4 AclOpKernelInit failed

• IV. Accuracy Issues

• V. Performance Issues

• VI. Appendix: Quick Reference for Common Commands

---

I. Installation and Build

1.1 Missing clang++

Problem Description: The build process reports that the clang++ compiler cannot be found.

Error Message:

CMake Error at CMakeLists.txt:45 (project):
The CMAKE_C_COMPILER:
  clang
is not a full path and was not found in the PATH.

CMake Error at CMakeLists.txt:45 (project):
The CMAKE_CXX_COMPILER:
  clang++
is not a full path and was not found in the PATH.

CMake Error at CMakeLists.txt:45 (project):
No CMAKE_ASM_COMPILER could be found.

Solution: Download the corresponding version of clang++ from the LLVM official website and configure environment variables:

1. Download URL: https://github.com/llvm/llvm-project/releases

2. Extract after downloading the corresponding version

3. Configure environment variables:

export PATH=<LLVM_INSTALL_ROOT>/bin:$PATH

It is recommended to add the above command to ~/.bashrc or ~/.zshrc to make it permanent.

---

1.2 torch.distributed.DistNetworkError Port Conflict

Problem Description: The following error occurs when starting distributed training:

Error Message:

File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/agent/server/api.py", line <LINE>, in _rendezvous
    rdzv_info = spec.rdzv_handler.next_rendezvous()
File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/rendezvous/static_tcp_rendezvous.py", line <LINE>, in next_rendezvous
    self._store = TCPStore(
torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: <PORT>, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
[ERROR] <TIMESTAMP> (PID:<PID>, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception

Solution: Specify a different port number in the torchrun command:

torchrun --master_port=<FREE_PORT_IN_20000_29999> ...

Or select any available port from the deployment-allowed free port range (e.g., 20000-29999).

---

1.3 No module named ‘triton.backends.amd’

Problem Description: Error when importing the triton module:

Error Message:

File "<PYTHON_SITE_PACKAGES>/triton/backends/__init__.py", line <LINE>, in _discover_backends
    compiler = importlib.import_module(f"{value}.compiler")
File "<PYTHON_STDLIB>/importlib/__init__.py", line <LINE>, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'triton.backends.amd'
[ERROR] <TIMESTAMP> (PID:<PID>, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception

Solution: Re-run pip install for triton-dist:

pip install -e .

Reason: The initial installation may have incomplete folder copying.

---

1.4 FileNotFoundError: bishengir-opt

Problem Description: Command execution reports that the bishengir-opt tool cannot be found.

Error Message:

File "<PYTHON_PREFIX>/lib/python<PYTHON_VERSION>/subprocess.py", line <LINE>, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'bishengir-opt'

Solution: Configure the path where bishengir-opt is located to the environment variable:

export PATH=<ASCEND_NPU_IR_ROOT>/build/bin:$PATH

Please adjust according to the actual installation path.

---

1.5 Cannot find hivmc under $PATH

Problem Description: The system reports that the hivmc tool cannot be found.

Error Message:

[ERROR][Triton][BEG]-
[ConvertLinalgIRToBinary] encounters error:
[ERROR] Cannot find hivmc under $PATH
[ERROR] Failed to run 'hivmc --version'
[WARNING] Failed to detect hivmc version for backward compatibility
[ERROR] Cannot find hivmc under $PATH
loc("<TMP_DIR>/kernel.mlir":<LINE>:<COLUMN>): error: External hivmc run fails, returning module before running external compiler
[ERROR] Failed to run BishengIR pipeline

Solution: The CANN package version is incorrect. Download the correct CANN package from the official website:

1. Download URL: https://www.hiascend.com/developer/download

2. Download and install CANN 9.1.0 official version

3. Ensure correct sourcing of the environment script:

source <CANN_ROOT>/set_env.sh

---

1.6 Failed to enable listening for the NPU network adapter socket

Problem Description: Startup reports that the NPU network adapter socket cannot listen.

Error Message:

[rank0]: RuntimeError: create_config:build/cMakeFiles/torch_npu.dir/compiler_depend.ts:149 HccL function error: hcclCommInitRootInfoConfig(numRanks, &rootInfo, rank, config, &(comm->hcclcomm_)), error code is 7
[rank0]: [ERROR] <TIMESTAMP> (PID:<PID>, Device:<DEVICE_ID>, RankID:0) ERR02200 DIST call hccl api failed
[PID:<PID>]<TIMESTAMP> Communication_Error_Bind_IP_Port (EI0020):
Failed to enable listening for the NPU network adapter socket. Reason: The IP address <IP_ADDRESS> and port <PORT> have already been bound.
[rank0]: Solution: Check whether the single-card multi-process scenario is used. If yes, configure the port number using the environment variable HCCL_NPU_SOCKET_PORT_RANGE.

Reason: NPU socket port is occupied by another process.

Solution 1: Check usage status:

npu-smi info

Find the occupying process and release resources.

Solution 2 (Recommended): Set environment variable for automatic port allocation:

export HCCL_NPU_SOCKET_PORT_RANGE="auto"

You can also specify the deployment-allowed port range:

export HCCL_NPU_SOCKET_PORT_RANGE="<PORT_START>-<PORT_END>"

The actual port range depends on cluster network policy and CANN version. All ranks should use a consistent port policy; if it still fails, also check the listening network interface and residual processes.

It is recommended to add this command to the startup script.

---

1.7 error: ‘hivm.hir.custom’ op operand count mismatch

Problem Description: Compilation error reporting operand count mismatch.

Error Message:

[rank0]: subprocess.CalledProcessError: Command ['<CANN_ROOT>/bin/bishengir-opt', '<TMP_DIR>/kernel.mlirbc', '-o', '<TMP_DIR>/kernel.mlir'] returned non-zero exit status 1.
[rank0]: triton.compiler.errors.MLIRcompilationError:
[ERROR][Triton][BEG]--
[BytecodeToLinalgIRByBishengiropt] encounters error:
<SOURCE_ROOT>/tutorials/ascend/01-ascend-allgather-gemm.py:<LINE>:<COLUMN>: error: 'hivm.hir.custom' op operand count (2) does not match with the total size (0) specified in attribute 'operandSegmentSizes'

Solution: AscendNPU-IR path is incorrectly configured. Re-source the environment variables:

source <ASCEND_NPU_IR_ROOT>/build/setup.sh

---

1.8 unsupport operand type(s) for

Problem Description: Runtime error with unsupported operator.

Error Message:

File "<PYTHON_SITE_PACKAGES>/triton_dist/language/extra/language_extra.py", line <LINE>, in <module>
    from triton_dist.utils import is_ascend, is_cuda, is_hip, is_maca
def rand_tensor(shape, dtype: torch.dtype, device: torch.device = "cuda" if type ...
[ERROR] <TIMESTAMP> (PID:<PID>, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception

Solution: Python version is too low. It is recommended to use Python 3.11 or higher:

python3.11 -m venv venv
source venv/bin/activate

---

1.9 The following imported targets are referenced, but are missing: LLVMCGData

Problem Description: Compiling triton-distributed-ascend reports missing LLVMCGData target.

Error Message:

CMake Error at 3rdparty/triton-ascend/third_party/amd/CMakeLists.txt:6 (find_package):
Found package configuration file:
<LLVM_INSTALL_ROOT>/lib/cmake/lld/LLDConfig.cmake
but it set LLD_FOUND to FALSE so package "LLD" is considered to be NOT FOUND.
Reason given by package:
The following imported targets are referenced, but are missing: LLVMCGData
Configuring incomplete, errors occurred!

Solution: Caused by build residue. Clean and rebuild:

rm -rf build/
mkdir build && cd build
cmake ..
make -j$(nproc)

---

1.10 Error building triton_dist

Problem Description: Linking fails when building triton_dist.

Error Message 1 (cmake_args configuration):

if check_env_flag("TRITON_BUILD_WITH_CLANG_LLD"):
    cmake_args += [
        "-DCMAKE_C_COMPILER=clang",
        "-DCMAKE_CXX_COMPILER=clang++",
        "-DCMAKE_LINKER=lld",
        "-DCMAKE_EXE_LINKER_FLAGS=-fuse-ld=lld",
        "-DCMAKE_MODULE_LINKER_FLAGS=-fuse-ld=lld",
        "-DCMAKE_EXE_LINKER_FLAGS=-L<SYSTEM_LIBRARY_DIR>",
        "-DCMAKE_SHARED_LINKER_FLAGS=-L<SYSTEM_LIBRARY_DIR>"
    ]

Error Message 2 (link failure):

FAILED: 3rdparty/triton-ascend/test/lib/Proton/libTritonTestProton.a
<SHELL>: <LLVM_INSTALL_ROOT>/bin/llvm-ar: not found
ninja: build stopped: subcommand failed.

Solution:

1. Delete the build directory

2. Add the following link parameters to cmake_args in setup.py:

cmake_args = [
    "-DCMAKE_EXE_LINKER_FLAGS=-L<SYSTEM_LIBRARY_DIR>",
    "-DCMAKE_SHARED_LINKER_FLAGS=-L<SYSTEM_LIBRARY_DIR>"
]

3. Rebuild

---

1.11 Failed to create conda environment

Problem Description: Error when creating conda environment, reporting channel corruption.

Error Message:

Channels:
defaults
Platform: linux-64
Collecting package metadata (repodata.json): failed
UnavailableInvalidChannel: HTTP 404 Not Found for channel anaconda/pkgs/r <https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r>
The channel is not accessible or is invalid.
You will need to adjust your conda configuration to proceed.
Use `conda config --show channels` to view your configuration's current state,
and use `conda config --show-sources` to view config file locations.

Solution:

1. Remove the corrupted channel:

conda config --remove channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
conda config --remove default_channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r

2. Configure pip mirror source:

mkdir -p ~/.pip
cat > ~/.pip/pip.conf << 'EOF'
[global]
index-url = http://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
extra-index-url = http://mirror.nju.edu.cn/pytorch/whl/cpu
trusted-host =
    mirrors.tuna.tsinghua.edu.cn
    mirror.nju.edu.cn
EOF

3. Recreate conda environment

---

II. Compilation

2.1 undefined symbol

Problem Description: test/ascend/test_*.py tests pass, but tutorials/ascend/01-*.py runtime error reporting cannot find symbol.

Error Message 1 (link error):

[rank1]: ld.lld: error: undefined symbol:
  mlir_ciface_aclshmem_barrier_all.cube
[rank1]: >>> referenced by LLVMDialectModule
[rank1]: >>> <TMP_DIR>/kernel.o:(kernel_allgather_gemm_mix_aic)
[rank1]: ld.lld: error: undefined symbol:
  mlir_ciface_aclshmem_ptr_half.cube
[rank1]: >>> referenced by LLVMDialectModule
[rank1]: >>> <TMP_DIR>/kernel.o:(aclshmem_ptr_half.cube)
[rank1]: [ERROR] Executing: <CANN_ROOT>/tools/ccec_compiler/bin/ld.lld -maicorelinux -Ttext ...
[rank1]: loc("<TMP_DIR>/module.hivm.opt.mlir":2:1): error: Failed to compile BishengIR for device
[rank1]: [ERROR] Failed to run HIVMC pipeline

Error Message 2 (version compatibility issue):

Failed: Distributed worker error:
[rank 0]
[ERROR][Triton][BEG]---
[ConvertTritonIRToLinalgIR] encounters error:
add_convert_triton_distributed_to_hivm(): incompatible function arguments. The following argument types are supported:
1. (arg0: triton._C.libtriton.ir.pass_manager, arg1: str) -> None
Invoked with: <triton._C.libtriton.ir.pass_manager object at 0xfffcf8f67eb0>

Background: Self-built bishengir-compile has compatibility issues.

Solution:

Method 1 (Recommended): Use official daily packages to avoid self-compilation.

Method 2: If you must self-compile, ensure the following conditions are met:

a. Add --build-shmem-template parameter when building:

./build-tools/build.sh \
  --build-shmem-template \
  -o ./build \
  -t \
  --build-type Release \
  --apply-patches \
  --bisheng-compiler=<CANN_ROOT>/bin

b. Ensure you are using your self-compiled bishengir-compile:

export TRITON_DEBUG=1

Execute the above command before running .py scripts to view the actual bishengir-compile path being used.

c. Ensure your self-compiled bishengir-compile is compatible with the hivmc version.

---

III. Runtime

3.1 npu function error: device error type 0, error code is 4

Problem Description: NPU device error occurs at runtime.

Error Message:

[rank1]: File "<PYTHON_SITE_PACKAGES>/torch/distributed/distributed_c10d.py", line <LINE>, in barrier
[rank1]: work = group.barrier(opts=opts)
[rank1]: RuntimeError: create_config: <TORCH_NPU_ROOT>/csrc/distributed/HcCLUtils.cpp:<LINE> NPU function error: device error type 0, error code is 4

Solution: Update to CANN 9.1.0 official version:

Reference: The corresponding CANN version’s public release notes and project support matrix.

---

3.2 Port conflict

Problem Description: Port is occupied when starting torch-run.

Error Message:

File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/metrics/api.py", line <LINE>, in wrapper
    result = f(*args, **kwargs)
File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/agent/server/api.py", line <LINE>, in _rendezvous
    rdzv_info = spec.rdzv_handler.next_rendezvous()
File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/rendezvous/static_tcp_rendezvous.py", line <LINE>, in next_rendezvous
    self._store = TCPStore(
torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: <PORT>, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
[ERROR] <TIMESTAMP> (PID:<PID>, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception

Solution: Add the --master_port parameter after the torch-run command to specify a different port:

torch-run --master_port=<FREE_PORT_IN_20000_29999> ...

Note: The --master_port parameter needs to be added immediately after the torch-run command. Ports should be selected from the deployment-allowed free range (e.g., 20000-29999).

---

3.3 Card failure

Problem Description: Card failure occurs at runtime, NPU card cannot work normally.

Error Message:

[rank0]: RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is HcclAllreduce.
[rank0]: Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[rank0]: [ERROR] <TIMESTAMP> (PID:<PID>, Device:0, RankID:0) ERR00100 PTA call acl api failed.
[PID:<PID>]<TIMESTAMP> Communication_Error_Bind_IP_Port(EI0020): Failed to enable listening for the NPU network adapter socket. Reason: The configured IP address and port <PORT> have already been bound.
[rank0]: Solution: Check whether the single-card multi-process scenario is used. If yes, configure the port number using the environment variable HCCL_NPU_SOCKET_PORT_RANGE.

Solution 1: Set environment variable for automatic port allocation:

export HCCL_NPU_SOCKET_PORT_RANGE="auto"

You can also explicitly specify the deployment-allowed port range:

export HCCL_NPU_SOCKET_PORT_RANGE="<PORT_START>-<PORT_END>"

The port range should comply with the current cluster network policy and CANN version requirements, and ensure it is not occupied by other processes.

Solution 2: Reset the NPU card:

npu-smi set -t reset -i ${CARD_ID} -m 1

Replace ${CARD_ID} with the actual card number.

---

3.4 AclOpKernelInit failed

Problem Description: Runtime reports AclOpKernelInit failure.

Error Message:

<USER>@<HOST> $ HCCL_NPU_SOCKET_PORT_RANGE="auto" torchrun --nproc-per-node=2 <SCRIPT>
...
[rank0]: RuntimeError: ones: <OP_PLUGIN_ROOT>/ops/opapi/OnesKernelNpuOpApi.cpp:<LINE> NPU function error: device error type 0, error code is 561103
[rank0]: [ERROR] <TIMESTAMP> (PID:<PID>, Device:0, RankID:0) ERR00100 PTA call acl api failed.
[rank0]: EZ9999: Inner Error!
[rank0]: EZ9999 [PID:<PID>] <TIMESTAMP> AclNN_Inner_Error(EZ9999): Parse dynamic kernel config fail.
[rank0]: TraceBack (most recent call last):
[rank0]: Failed to ParseDynamicKernels.
[rank0]: AclOpKernelInit failed opType. Maybe caused by the following reasons:
[rank0]: 1. Please check whether the CANN enviroment variables are sourced.
[rank0]: 2. Please check if the kernel package is installed.
[rank0]: 3. Please confirm whether the current chip or version already supports the faulty operator.
[rank0]: OnesLike ADD_TO_LAUNCHER_LIST_AICORE failed.

Possible Reasons:

1. CANN is not correctly installed

2. set_env.sh was not sourced

3. CANN version does not match hardware (e.g., CANN package for A3/A2 installed on an A5 machine)

Solution:

1. Check if CANN version matches the hardware

2. Ensure correct sourcing of the environment script:

source <CANN_ROOT>/set_env.sh

3. Reinstall the correct CANN version

---

IV. Accuracy Issues

---

V. Performance Issues

---

VI. Appendix: Quick Reference for Common Commands

Environment Variable Configuration

# clang++ path
export PATH=<LLVM_INSTALL_ROOT>/bin:$PATH

# AscendNPU-IR path
export PATH=<ASCEND_NPU_IR_ROOT>/build/bin:$PATH

# NPU socket port auto allocation
export HCCL_NPU_SOCKET_PORT_RANGE="auto"

# Or specify deployment-allowed port range
export HCCL_NPU_SOCKET_PORT_RANGE="<PORT_START>-<PORT_END>"

# Debug mode
export TRITON_DEBUG=1

Common Tool Commands

# View NPU status
npu-smi info

# Reset specified NPU card
npu-smi set -t reset -i ${CARD_ID} -m 1

# Start distributed task (select free port from deployment-allowed range)
torchrun --master_port=<FREE_PORT_IN_20000_29999> ...

# Reinstall triton-dist
pip install -e .

Clean and Rebuild

# Clean build directory
rm -rf build/

# Create and enter build directory
mkdir build && cd build

# Configure CMake (adjust according to actual system library directory)
cmake -DCMAKE_EXE_LINKER_FLAGS="-L<SYSTEM_LIBRARY_DIR>" ..

# Compile
make -j$(nproc)

Issue Reporting

When submitting issues, please provide redacted key error messages, software versions, chip model, number of ranks, shape, dtype, reproduction commands, and solutions already attempted.

Before submitting, please remove usernames, employee IDs, hostnames, IP/MAC addresses, PIDs, internal paths, access tokens, and business data.