# Lamellar as an RLST Array Backend ## Objective Investigate whether [Lamellar](https://github.com/pnnl/lamellar), an asynchronous PGAS runtime for HPC, can provide distributed storage and communication for RLST arrays without making RLST’s existing array APIs unsound or misleading. The intended outcome is an evidence-based decision, not a commitment to replacing RLST’s existing MPI support. ## Starting point RLST represents dense arrays as: ```rust Array<ArrayImpl, const NDIM: usize> ``` The wrapper is deliberately trait-oriented. Different APIs require different capabilities from `ArrayImpl`: | Capability | Relevant RLST traits | Consequence for a backend | | ------------------------ | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | Shape | `Shape<NDIM>` | The backend must report an unambiguous logical shape. | | Value access | `UnsafeRandom1DAccessByValue`, `UnsafeRandomAccessByValue` | A backend can support read-by-value even if it cannot expose Rust references. | | Reference access | `UnsafeRandom1DAccessByRef`, `UnsafeRandomAccessByRef` | Requires a real, stable `&Item`. | | Mutable reference access | `UnsafeRandom1DAccessMut`, `UnsafeRandomAccessMut` | Requires a real, stable `&mut Item` with ordinary Rust aliasing semantics. | | Contiguous data | `RawAccess`, `RawAccessMut` | Requires a genuine whole-array `&[Item]` or `&mut [Item]`; callers use this for BLAS/LAPACK. | | Layout | `Stride<NDIM>` | RLST’s logical one-dimensional ordering is column-major. | This distinction is central. An implementation should only claim the subset of traits it can uphold truthfully. ## Why a global Lamellar array is not automatically an RLST `Array` Lamellar’s purpose is distributed memory and asynchronous remote operations. In contrast, much of RLST’s ordinary array interface is synchronous and reference-based: - `Index` and `iter_ref` ultimately require `&Item`. - `IndexMut` and `iter_mut` require `&mut Item`. - `RawAccess` and `RawAccessMut` expose whole Rust slices. - Dense GEMM and related BLAS/LAPACK paths obtain such slices and currently call `data().unwrap()`. A remote element, a runtime-managed element, a proxy, a future, or a lock/guard is not a substitute for `&Item` or `&mut Item`. Similarly, a globally distributed allocation is not a single contiguous Rust slice. Attempting to force those semantics through unsafe code would create incorrect lifetime, synchronization, aliasing, and visibility guarantees. Therefore, a global Lamellar handle should **not** be made to masquerade as an ordinary fully-featured RLST `Array` unless the runtime provides the exact synchronous reference and slice guarantees required by the traits. ## Recommended architecture: local RLST partitions plus explicit distributed coordination The leading strategy is to distinguish a local array backend from a distributed array abstraction. ```text +-------------------------------------------------------------+ | Distributed Lamellar-facing type | | - global logical shape and distribution metadata | | - Lamellar world/team and distributed array handle | | - explicit async get/put, collectives, fences, barriers | | - no fabricated Rust references to remote elements | +----------------------------+--------------------------------+ | | obtains/exports a local partition v +-------------------------------------------------------------+ | RLST local-partition adapter | | - one PE's contiguous owned partition | | - local shape, stride and column-major logical order | | - only implements the RLST traits proven safe | | - can be copied/materialized as a `DynArray` when needed | +----------------------------+--------------------------------+ | v +-------------------------------------------------------------+ | Existing RLST local dense/sparse algorithms | | - local element-wise operations | | - BLAS/LAPACK only when local contiguous storage is exposed | | - CSR kernels on local rows | +-------------------------------------------------------------+ ``` This is intentionally similar in spirit to RLST’s current MPI-only `DistributedArray`: the distributed wrapper holds a conventional local RLST array and distributes the first axis. Lamellar could provide the communication/runtime layer while local RLST kernels continue to work with local data. ## Initial adapter contract The first disposable adapter should represent a **local Lamellar partition only**. It should not initially promise every `DynArray` operation. ### Required initial behavior 1. Store a logical local shape `[usize; NDIM]`. 2. Translate RLST multi-indices using column-major order: ```text linear_index = i0 + shape[0] * (i1 + shape[1] * (...)) ``` 3. Support value-based reads only if Lamellar has a documented, completed read operation. 4. Provide explicit bulk import/export between local Lamellar storage and `DynArray`. 5. Make completion visible in the public local/distributed operation that triggered remote or asynchronous work. ### Traits to implement only after proof | Trait family | Initial default | Condition for adding it | |---|---|---| | `BaseItem`, `ContainerType`, `Shape` | Expected | Straightforward local metadata and element constraints. | | Value-access traits | Expected candidate | A documented local/remote value read can be completed synchronously at this API boundary. | | `Stride` | Expected for a contiguous local partition | The local physical layout can be described accurately; do not infer global contiguity. | | `UnsafeRandomAccessByRef` | Omit initially | Lamellar exposes stable local references with valid lifetimes. | | `UnsafeRandomAccessMut` | Omit initially | Lamellar exposes stable mutable local references without violating PGAS synchronization. | | `RawAccess` / `RawAccessMut` | Omit for global handles | The represented object is exactly a valid contiguous local slice for its entire logical shape. | | `ResizeInPlace` | Omit initially | The runtime’s allocation and distribution semantics support it without changing ownership unexpectedly. | ## BLAS/LAPACK boundary RLST’s BLAS/LAPACK calls are local-memory operations. The safe rule is: > A BLAS/LAPACK algorithm may operate only on a local contiguous partition or on an explicit materialized temporary; it must never receive a fictitious global slice. Possible paths: 1. **Local operation:** distribute matrices/vectors such that each PE owns a local dense block, expose only that block to BLAS/LAPACK, then exchange/aggregate results explicitly. 2. **Materialize-and-compute:** export needed data to `DynArray`, perform existing RLST dense operation, and import/copy results back. This is acceptable for a feasibility spike, but must be measured before becoming a production default. 3. **Distributed algorithm:** implement a dedicated distributed method with Lamellar communication. This is a later design effort; it is not achieved merely by supplying a storage backend. ## Relationship to RLST MPI support Lamellar should initially **complement**, not replace, `feature = "mpi"`. RLST currently has: - `DistributedArray`, which distributes the first axis; - `IndexLayout` metadata for ownership; - MPI collectives used for gather/scatter; - distributed CSR and vector-space/operator interfaces. The Lamellar investigation must establish whether equivalent ownership metadata and collectives map naturally to Lamellar. Until then: - preserve the existing MPI API and tests; - put Lamellar behind a separate optional Cargo feature; - avoid changing default features or default build dependencies; - make the runtime choice explicit in public types and documentation. ## Evidence-driven spike sequence ### 1. Lamellar semantic inventory Pin the investigation to Lamellar 0.8.1 and document exact source/API evidence for: - construction and distribution policies; - local partition and slice/iterator exposure; - global reads and writes; - whether access returns values, references, guards, proxies, or futures; - completion/fence/barrier semantics; - collectives and copy-in/copy-out; - runtime launcher requirements and minimal Cargo feature set. Produce a matrix mapping each RLST trait contract to one of: supported, supported for local partitions only, requires an explicit conversion, or cannot be implemented safely. ### 2. Single-PE local adapter Build a standalone, throwaway Cargo spike that depends on RLST by path and Lamellar 0.8.1. Test a non-square `[2, 3]` local array: - exact shape; - column-major one-dimensional reads; - correct multi-index mapping; - out-of-bounds safe lookup; - local write visibility after the documented completion operation; - bulk export matching a `DynArray`. The adapter starts with value-access traits. It must not add reference or raw-slice traits merely for API symmetry. ### 3. Multi-PE semantics Run a small program at one PE, then at two or more PEs. Each PE should read a local and remote entry, perform a remote mutation, execute the required completion/fence/barrier, and verify the owner observes the expected value exactly once. The required output is an explicit answer to: - Can local access supply stable `&Item`? - Can local access supply stable `&mut Item`? - Can a global object supply a truthful whole `&[Item]` / `&mut [Item]`? - Where must asynchronous completion occur? Any negative answer is a design boundary, not a failure of the investigation. ### 4. Representative RLST operation Use a tiny deterministic distributed vector inner product or distributed CSR matrix-vector application. It should have a hand-computable expected result and report local partitions plus all required communication/materialization steps. The objective is to establish data flow, not to claim performance. Compare the result to the existing MPI `DistributedArray` approach: - ownership/distribution model; - synchronization points; - compatibility with `IndexLayout`; - local BLAS/LAPACK restrictions; - additional dependency and runtime cost; - copies required at algorithm boundaries. ## Go/no-go criteria ### Go: local-partition backend Proceed toward a feature-gated production implementation only if all conditions hold: - column-major value access is correct; - local partition ownership and shape are unambiguous; - asynchronous completion is explicit and correct under multiple PEs; - local data can be safely exported or exposed for the selected RLST kernels; - one representative sparse/operator workflow has verified data flow; - Lamellar’s build/runtime requirements can remain optional and isolated from RLST defaults. ### No-go: global ordinary `Array` backend Do not implement a global Lamellar allocation as a full ordinary RLST `Array` if it cannot satisfy synchronous reference or contiguous-slice traits. The preferred alternative is a separate distributed type with explicit PGAS operations and local RLST partitions. ## Non-goals for the first implementation - Replacing RLST MPI support. - Making every existing dense RLST algorithm distributed automatically. - Advertising global Lamellar storage as BLAS/LAPACK-compatible. - Adding unsafe lifetime extensions or hidden blocking communication to emulate Rust references. - Adding Lamellar to default Cargo features. - Making performance claims before dedicated benchmarks separate runtime startup, communication, synchronization, conversion, and local kernel time. ## Current status - Branch: `lamellar`, created from RLST `main`. - A feasibility plan exists at `rlst/.hermes/plans/2026-08-23_172246-lamellar-array-backend-exploration.md`. - Initial exploration and both executed spikes use Lamellar 0.8.1. - The two spike crates are intentionally untracked under `rlst/spikes/`; no production RLST API, Cargo feature, default dependency, or MPI implementation has been changed. ## Executed observations (2026-08-23) ### Lamellar/RLST value-access adapter The first verified implementation is a disposable local adapter in: - `rlst/spikes/002-lamellar-local-adapter/src/lib.rs` - `rlst/spikes/002-lamellar-local-adapter/tests/local_adapter.rs` `LamellarValueArray<T>` wraps a `LocalLockArray<T>` and a two-dimensional logical shape. It implements only the RLST capabilities that can be represented honestly: - `BaseItem`; - `ContainerType` with `Unknown`; - `Shape<2>`; - `UnsafeRandom1DAccessByValue`; - `UnsafeRandomAccessByValue<2>`. Value reads use Lamellar's completed, owned-value operation `blocking_get`. Multi-index lookup maps RLST's column-major logical indices as: ```text index([row, column]) = row + number_of_rows * column ``` The test populated a local `f64` array with `[10, 20, 30, 40, 50, 60]`, gave it shape `[2, 3]`, and verified: | Lookup | Result | |---|---:| | `[0, 0]` | `10` | | `[1, 0]` | `20` | | `[0, 1]` | `30` | | `[1, 2]` | `60` | | `[2, 0]` | out of bounds (`None`) | RLST value iteration returned the expected column-major sequence `[10, 20, 30, 40, 50, 60]`. The adapter also offers an explicit `copy_to_vec()` conversion using `blocking_get_buffer`. After `storage.put(5, 99.0).block()`, the exported buffer was `[0, 0, 0, 0, 0, 99]`. This proves completed mutation visibility and materialization into an owned contiguous buffer. The adapter intentionally does **not** implement reference access, mutable-reference access, `RawAccess`, or `RawAccessMut`. Lamellar's `read_local_data()` and `write_local_data()` produce owned lock guards that dereference to local slices; those guards cannot be truthfully returned as stable references or slices from a long-lived global array handle. `copy_to_vec()` is therefore a deliberate algorithm boundary, not a substitute for a raw-slice trait. Focused validation passed: ```text cargo fmt --manifest-path spikes/002-lamellar-local-adapter/Cargo.toml -- --check cargo clippy --manifest-path spikes/002-lamellar-local-adapter/Cargo.toml -- -D warnings cargo test --manifest-path spikes/002-lamellar-local-adapter/Cargo.toml ``` The test `local_lock_array_reads_values_in_rlst_column_major_order` passed. ### Two-PE global PGAS semantics The second disposable executable is in: - `rlst/spikes/003-lamellar-remote-semantics/src/main.rs` - `rlst/spikes/003-lamellar-remote-semantics/run.sh` It allocates a block-distributed `GlobalLockArray<u64>` of length `2 * number_of_PEs + 1`, initializes every entry through `dist_iter_mut().enumerate().for_each(...).block()`, and then synchronizes with `barrier()`. With two PEs, each PE targets a value owned by the other partition: - PE 0 writes the final entry with `put(last, 100).block()`. - PE 1 writes entry zero with `put(0, 101).block()`. - A collective `barrier()` separates the mutation phase from observation. - Both PEs use `blocking_get` and assert the two global endpoint values. The verified run was: ```text $ spikes/003-lamellar-remote-semantics/run.sh 2 remote-semantics: pe=0/2; target=4; before=4; after=100; PASS remote-semantics: pe=1/2; target=0; before=0; after=101; PASS ``` This establishes that Lamellar's blocking one-sided reads/writes and an explicit collective barrier are sufficient for this small remote-write visibility protocol. It does **not** establish ordinary shared-memory reference semantics, a whole global slice, or direct BLAS/LAPACK compatibility. ### Local macOS runtime notes The two-PE runner builds Lamellar's vendored PRRTE support and launches the release executable with the shared-memory backend: ```text --pes 2 --lamellae shmem ``` On this macOS system, Lamellar's transitive `libevent-sys` CMake configuration needed `-DCMAKE_POLICY_VERSION_MINIMUM=3.5` because the installed CMake no longer permits the package's older compatibility setting. `run.sh` creates a temporary CMake wrapper that adds this option only to configuration invocations, leaving `cmake --build` untouched. It also discovers Homebrew `hwloc` when installed and defaults `LAMELLAR_THREADS` to `1`. These requirements reinforce that Lamellar must remain optional and separately feature-gated; they must not affect RLST's default build. ## Current decision **Go for a narrow local/explicit-PGAS experiment; no-go for a global ordinary RLST dense-array backend.** The evidence supports a local value-access adapter and explicit owned-buffer conversions. It does not support representing a globally distributed Lamellar allocation as a full RLST `Array` with normal reference, mutable-reference, or raw-slice capabilities. The next implementation should be a distinct Lamellar distributed vector type with explicit completion semantics, beginning with a local-block collective operation such as `y = alpha * x + beta * y`. This validates ownership and synchronization without inheriting the existing MPI distributed CSR and operator contracts prematurely. A representative distributed sparse/operator workflow remains the final uncompleted go/no-go criterion for a production feature. ## Related notes - [[rlst]]