FAQ

This document collects common issues in using and developing Triton-distributed-ascend, organized by category for quick look-up.

Note: Sensitive information in this document has been replaced with placeholders. All parameters marked with <> must be replaced with actual values available on site.

Port and Address Conventions

Both distributed launch and HCCL communication should use available free port ranges allowed by deployment, without hardcoding a specific port. The following uses 20000-29999 as an example range; the actual range depends on cluster network policy and CANN version:

export MASTER_PORT=<FREE_PORT_IN_20000_29999>
torchrun --master_port=<MASTER_PORT> --nproc-per-node=<WORLD_SIZE> <SCRIPT> <ARGS>

If the host has multiple network interfaces, select a network interface or IP that is reachable by all ranks; do not directly copy addresses from logs.

Installation and Build

Q: clang++ compiler cannot be found

The error message is as follows:

CMake Error at CMakeLists.txt:45 (project):
The CMAKE_C_COMPILER:
  clang
is not a full path and was not found in the PATH.

CMake Error at CMakeLists.txt:45 (project):
The CMAKE_CXX_COMPILER:
  clang++
is not a full path and was not found in the PATH.

CMake Error at CMakeLists.txt:45 (project):
No CMAKE_ASM_COMPILER could be found.

Solution:

Download the corresponding version of clang++ from the LLVM official website and configure environment variables:

  1. Download URL: https://github.com/llvm/llvm-project/releases

  2. Extract after downloading the corresponding version

  3. Configure environment variables. It is recommended to add the following command to ~/.bashrc or ~/.zshrc to make it permanent:

    export PATH=<LLVM_INSTALL_ROOT>/bin:$PATH

Q: torch.distributed.DistNetworkError port occupied when starting distributed training

The error message is as follows:

File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/agent/server/api.py", line <LINE>, in _rendezvous
    rdzv_info = spec.rdzv_handler.next_rendezvous()
File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/rendezvous/static_tcp_rendezvous.py", line <LINE>, in next_rendezvous
    self._store = TCPStore(
torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: <PORT>, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
[ERROR] <TIMESTAMP> (PID:<PID>, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception

Solution:

Specify a different port number in the torchrun command:

torchrun --master_port=<FREE_PORT_IN_20000_29999> ...

Or select any available port from the deployment-allowed free port range (e.g., 20000-29999).

Q: triton.backends.amd module not found when importing triton

The error message is as follows:

File "<PYTHON_SITE_PACKAGES>/triton/backends/__init__.py", line <LINE>, in _discover_backends
    compiler = importlib.import_module(f"{value}.compiler")
File "<PYTHON_STDLIB>/importlib/__init__.py", line <LINE>, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'triton.backends.amd'
[ERROR] <TIMESTAMP> (PID:<PID>, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception

Reason: The initial installation may have incomplete folder copying.

Solution:

Re-run pip install for triton-dist:

pip install -e .

Q: bishengir-opt tool cannot be found

The error message is as follows:

File "<PYTHON_PREFIX>/lib/python<PYTHON_VERSION>/subprocess.py", line <LINE>, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'bishengir-opt'

Solution:

Add the directory of bishengir-opt to the environment variable. Replace the placeholder with the actual installation path. An example is as follows:

export PATH=<ASCEND_NPU_IR_ROOT>/build/bin:$PATH

Q: hivmc tool cannot be found

The error message is as follows:

[ERROR][Triton][BEG]-
[ConvertLinalgIRToBinary] encounters error:
[ERROR] Cannot find hivmc under $PATH
[ERROR] Failed to run 'hivmc --version'
[WARNING] Failed to detect hivmc version for backward compatibility
[ERROR] Cannot find hivmc under $PATH
loc("<TMP_DIR>/kernel.mlir":<LINE>:<COLUMN>): error: External hivmc run fails, returning module before running external compiler
[ERROR] Failed to run BishengIR pipeline

Solution:

The CANN package version is incorrect. Download the correct CANN package from the official website:

  1. Download URL: https://www.hiascend.com/developer/download

  2. Download and install CANN 9.1.0 official version

  3. Ensure correct sourcing of the environment script:

source <CANN_ROOT>/set_env.sh

Q: Failed to enable listening for the NPU network adapter socket at startup

The error message is as follows:

[rank0]: RuntimeError: create_config:build/cMakeFiles/torch_npu.dir/compiler_depend.ts:149 HccL function error: hcclCommInitRootInfoConfig(numRanks, &rootInfo, rank, config, &(comm->hcclcomm_)), error code is 7
[rank0]: [ERROR] <TIMESTAMP> (PID:<PID>, Device:<DEVICE_ID>, RankID:0) ERR02200 DIST call hccl api failed
[PID:<PID>]<TIMESTAMP> Communication_Error_Bind_IP_Port (EI0020):
Failed to enable listening for the NPU network adapter socket. Reason: The IP address <IP_ADDRESS> and port <PORT> have already been bound.
[rank0]: Solution: Check whether the single-card multi-process scenario is used. If yes, configure the port number using the environment variable HCCL_NPU_SOCKET_PORT_RANGE.

Reason: The NPU socket port is occupied by another process.

Solution:

  • Option 1 (Recommended): Set the environment variable for automatic port allocation:

    export HCCL_NPU_SOCKET_PORT_RANGE="auto"

    You can also specify the deployment-allowed port range:

    export HCCL_NPU_SOCKET_PORT_RANGE="<PORT_START>-<PORT_END>"

    The actual port range depends on cluster network policy and CANN version. All ranks should use a consistent port policy; if it still fails, also check the listening network interface and residual processes. It is recommended to add this command to the startup script.

  • Option 2: Check device usage status, use npu-smi info to locate the occupying process, and then release the resources.

Q: Compilation error: 'hivm.hir.custom' op operand count mismatch

The error message is as follows:

[rank0]: subprocess.CalledProcessError: Command ['<CANN_ROOT>/bin/bishengir-opt', '<TMP_DIR>/kernel.mlirbc', '-o', '<TMP_DIR>/kernel.mlir'] returned non-zero exit status 1.
[rank0]: triton.compiler.errors.MLIRcompilationError:
[ERROR][Triton][BEG]--
[BytecodeToLinalgIRByBishengiropt] encounters error:
<SOURCE_ROOT>/tutorials/ascend/allgather-gemm.py:<LINE>:<COLUMN>: error: 'hivm.hir.custom' op operand count (2) does not match with the total size (0) specified in attribute 'operandSegmentSizes'

Solution:

The AscendNPU-IR path is incorrectly configured. Re-source the environment variables:

source <ASCEND_NPU_IR_ROOT>/build/setup.sh

Q: unsupport operand type(s) for error at runtime

The error message is as follows:

File "<PYTHON_SITE_PACKAGES>/triton_dist/language/extra/language_extra.py", line <LINE>, in <module>
    from triton_dist.utils import is_ascend, is_cuda, is_hip, is_maca
def rand_tensor(shape, dtype: torch.dtype, device: torch.device = "cuda" if type ...
[ERROR] <TIMESTAMP> (PID:<PID>, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception

Solution:

The Python version is too low. It is recommended to use Python 3.11 or higher:

python3.11 -m venv venv
source venv/bin/activate

Q: missing: LLVMCGData when compiling triton-distributed-ascend

The error message is as follows:

CMake Error at 3rdparty/triton-ascend/third_party/amd/CMakeLists.txt:6 (find_package):
Found package configuration file:
<LLVM_INSTALL_ROOT>/lib/cmake/lld/LLDConfig.cmake
but it set LLD_FOUND to FALSE so package "LLD" is considered to be NOT FOUND.
Reason given by package:
The following imported targets are referenced, but are missing: LLVMCGData
Configuring incomplete, errors occurred!

Solution:

This is caused by build residue. Clean and rebuild:

rm -rf build/
mkdir build && cd build
cmake ..
make -j$(nproc)

Q: Linking fails when building triton_dist

Error message 1 (cmake_args configuration):

if check_env_flag("TRITON_BUILD_WITH_CLANG_LLD"):
    cmake_args += [
        "-DCMAKE_C_COMPILER=clang",
        "-DCMAKE_CXX_COMPILER=clang++",
        "-DCMAKE_LINKER=lld",
        "-DCMAKE_EXE_LINKER_FLAGS=-fuse-ld=lld",
        "-DCMAKE_MODULE_LINKER_FLAGS=-fuse-ld=lld",
        "-DCMAKE_EXE_LINKER_FLAGS=-L<SYSTEM_LIBRARY_DIR>",
        "-DCMAKE_SHARED_LINKER_FLAGS=-L<SYSTEM_LIBRARY_DIR>"
    ]

Error message 2 (link failure log):

FAILED: 3rdparty/triton-ascend/test/lib/Proton/libTritonTestProton.a
<SHELL>: <LLVM_INSTALL_ROOT>/bin/llvm-ar: not found
ninja: build stopped: subcommand failed.

Solution:

  1. Delete the build directory

  2. Add the following link parameters to cmake_args in setup.py:

    cmake_args = [
    "-DCMAKE_EXE_LINKER_FLAGS=-L<SYSTEM_LIBRARY_DIR>",
    "-DCMAKE_SHARED_LINKER_FLAGS=-L<SYSTEM_LIBRARY_DIR>"
]

  3. Rebuild

Q: Error when creating conda environment, reporting channel corruption

The error message is as follows:

Channels:
defaults
Platform: linux-64
Collecting package metadata (repodata.json): failed
UnavailableInvalidChannel: HTTP 404 Not Found for channel anaconda/pkgs/r <https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r>
The channel is not accessible or is invalid.
You will need to adjust your conda configuration to proceed.
Use `conda config --show channels` to view your configuration's current state,
and use `conda config --show-sources` to view config file locations.

Solution:

  1. Remove the corrupted channel:

    conda config --remove channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
conda config --remove default_channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r

  2. Configure the pip mirror source:

    mkdir -p ~/.pip
cat > ~/.pip/pip.conf << 'EOF'
[global]
index-url = http://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
extra-index-url = http://mirror.nju.edu.cn/pytorch/whl/cpu
trusted-host =
    mirrors.tuna.tsinghua.edu.cn
    mirror.nju.edu.cn
EOF

  3. Recreate the conda environment

Compilation

Q: test/ascend/test_*.py tests pass, but tutorials/ascend/01-*.py runtime error reports undefined symbol

Problem description: The test/ascend/test_*.py tests pass, but running tutorials/ascend/01-*.py reports an error indicating that the symbol cannot be found.

Error message 1 (link error):

[rank1]: ld.lld: error: undefined symbol:
  mlir_ciface_aclshmem_barrier_all.cube
[rank1]: >>> referenced by LLVMDialectModule
[rank1]: >>> <TMP_DIR>/kernel.o:(kernel_allgather_gemm_mix_aic)
[rank1]: ld.lld: error: undefined symbol:
  mlir_ciface_aclshmem_ptr_half.cube
[rank1]: >>> referenced by LLVMDialectModule
[rank1]: >>> <TMP_DIR>/kernel.o:(aclshmem_ptr_half.cube)
[rank1]: [ERROR] Executing: <CANN_ROOT>/tools/ccec_compiler/bin/ld.lld -maicorelinux -Ttext ...
[rank1]: loc("<TMP_DIR>/module.hivm.opt.mlir":2:1): error: Failed to compile BishengIR for device
[rank1]: [ERROR] Failed to run HIVMC pipeline

Error message 2 (version compatibility issue):

Failed: Distributed worker error:
[rank 0]
[ERROR][Triton][BEG]---
[ConvertTritonIRToLinalgIR] encounters error:
add_convert_triton_distributed_to_hivm(): incompatible function arguments. The following argument types are supported:
1. (arg0: triton._C.libtriton.ir.pass_manager, arg1: str) -> None
Invoked with: <triton._C.libtriton.ir.pass_manager object at 0xfffcf8f67eb0>

Reason: The built bishengir-compile has compatibility issues.

Solution:

  • Option 1 (Recommended): Use the official daily packages to avoid self-compilation.

  • Option 2: If you must self-compile, ensure the following conditions are met:

    • Add the --build-shmem-template parameter when building:

      ./build-tools/build.sh \
  --build-shmem-template \
  -o ./build \
  -t \
  --build-type Release \
  --apply-patches \
  --bisheng-compiler=<CANN_ROOT>/bin

    • Ensure you are using your self-compiled bishengir-compile:

      export TRITON_DEBUG=1

      Execute the above command before running .py scripts to view the actual bishengir-compile path being used.

    • Ensure your self-compiled bishengir-compile is compatible with the hivmc version.

Runtime

Q: npu function error: device error type 0, error code is 4 at runtime

The error message is as follows:

[rank1]: File "<PYTHON_SITE_PACKAGES>/torch/distributed/distributed_c10d.py", line <LINE>, in barrier
[rank1]: work = group.barrier(opts=opts)
[rank1]: RuntimeError: create_config: <TORCH_NPU_ROOT>/csrc/distributed/HcCLUtils.cpp:<LINE> NPU function error: device error type 0, error code is 4

Solution:

Update to the CANN 9.1.0 official version. Reference: the corresponding CANN version’s public release notes and project support matrix.

Q: Port occupied when starting torchrun

The error message is as follows:

File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/metrics/api.py", line <LINE>, in wrapper
    result = f(*args, **kwargs)
File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/agent/server/api.py", line <LINE>, in _rendezvous
    rdzv_info = spec.rdzv_handler.next_rendezvous()
File "<PYTHON_SITE_PACKAGES>/torch/distributed/elastic/rendezvous/static_tcp_rendezvous.py", line <LINE>, in next_rendezvous
    self._store = TCPStore(
torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: <PORT>, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
[ERROR] <TIMESTAMP> (PID:<PID>, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception

Solution:

Add the --master_port parameter after the torchrun command to specify a free port. The --master_port parameter must immediately follow the torchrun command; the port must be chosen from a free range allowed by the deployment environment, such as 20000-29999.

torchrun --master_port=<FREE_PORT_IN_20000_29999> ...

Q: Card failure at runtime, NPU card cannot work normally

The error message is as follows:

[rank0]: RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is HcclAllreduce.
[rank0]: Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[rank0]: [ERROR] <TIMESTAMP> (PID:<PID>, Device:0, RankID:0) ERR00100 PTA call acl api failed.
[PID:<PID>]<TIMESTAMP> Communication_Error_Bind_IP_Port(EI0020): Failed to enable listening for the NPU network adapter socket. Reason: The configured IP address and port <PORT> have already been bound.
[rank0]: Solution: Check whether the single-card multi-process scenario is used. If yes, configure the port number using the environment variable HCCL_NPU_SOCKET_PORT_RANGE.

Solution:

  • Option 1: Set the environment variable for automatic port allocation:

    export HCCL_NPU_SOCKET_PORT_RANGE="auto"

    You can also explicitly specify the deployment-allowed port range. The port range should comply with the current cluster network policy and CANN version requirements, and ensure it is not occupied by other processes.

    export HCCL_NPU_SOCKET_PORT_RANGE="<PORT_START>-<PORT_END>"

  • Option 2: Reset the NPU card, replacing ${CARD_ID} with the actual card number.

    npu-smi set -t reset -i ${CARD_ID} -m 1

Q: Runtime reports AclOpKernelInit failed

The error message is as follows:

<USER>@<HOST> $ HCCL_NPU_SOCKET_PORT_RANGE="auto" torchrun --nproc-per-node=2 <SCRIPT>
...
[rank0]: RuntimeError: ones: <OP_PLUGIN_ROOT>/ops/opapi/OnesKernelNpuOpApi.cpp:<LINE> NPU function error: device error type 0, error code is 561103
[rank0]: [ERROR] <TIMESTAMP> (PID:<PID>, Device:0, RankID:0) ERR00100 PTA call acl api failed.
[rank0]: EZ9999: Inner Error!
[rank0]: EZ9999 [PID:<PID>] <TIMESTAMP> AclNN_Inner_Error(EZ9999): Parse dynamic kernel config fail.
[rank0]: TraceBack (most recent call last):
[rank0]: Failed to ParseDynamicKernels.
[rank0]: AclOpKernelInit failed opType. Maybe caused by the following reasons:
[rank0]: 1. Please check whether the CANN enviroment variables are sourced.
[rank0]: 2. Please check if the kernel package is installed.
[rank0]: 3. Please confirm whether the current chip or version already supports the faulty operator.
[rank0]: OnesLike ADD_TO_LAUNCHER_LIST_AICORE failed.

Possible reasons:

  1. CANN is not correctly installed

  2. source set_env.sh was not executed to configure the environment.

  3. The CANN version does not match the hardware. For example, a CANN installation package for Atlas A3 products and Atlas A2 products was installed on Ascend 950PR&950DT products.

Solution:

  1. Check whether the CANN version matches the hardware.

  2. Ensure the environment is configured correctly:

    source <CANN_ROOT>/set_env.sh

  3. Reinstall the CANN version that matches the hardware.

Issue Feedback

Q: How to report an issue?

When submitting an issue, please provide the key error messages, software versions, chip model, number of ranks, shape, dtype, reproduction commands, and solutions already attempted.

Note: Before submitting, remove sensitive information (for example, usernames, employee IDs, hostnames, IP/MAC addresses, PIDs, internal paths, access tokens, and business data).

Quick Reference for Common Commands

Environment Variable Configuration Commands

# clang++ path
export PATH=<LLVM_INSTALL_ROOT>/bin:$PATH

# AscendNPU-IR path
export PATH=<ASCEND_NPU_IR_ROOT>/build/bin:$PATH

# NPU socket port auto allocation
export HCCL_NPU_SOCKET_PORT_RANGE="auto"

# Or specify deployment-allowed port range
export HCCL_NPU_SOCKET_PORT_RANGE="<PORT_START>-<PORT_END>"

# Debug mode
export TRITON_DEBUG=1

Common Tool Commands

# View NPU status
npu-smi info

# Reset specified NPU card
npu-smi set -t reset -i ${CARD_ID} -m 1

# Start distributed task (select free port from deployment-allowed range)
torchrun --master_port=<FREE_PORT_IN_20000_29999> ...

# Reinstall triton-dist
pip install -e .

Clean and Rebuild Commands

# Clean build directory
rm -rf build/

# Create and enter build directory
mkdir build && cd build

# Configure CMake (adjust according to actual system library directory)
cmake -DCMAKE_EXE_LINKER_FLAGS="-L<SYSTEM_LIBRARY_DIR>" ..

# Compile
make -j$(nproc)