feat: Add NETopKV function. #1251

morgolock · 2026-01-15T17:38:11Z

The Neon(TM) implementation of TopKV reduces execution time from 447.8 ms (scalar CPP) to 11.65 ms for the same workload (F32, C=1000, N=32000, k=3, 6 threads), achieving an approximate 38× speedup. This gain comes from SIMD vectorization, removal of per-element branches, and a more efficient inner loop.
Resolves ARMCL-1227

Change-Id: Ifdf161ce4254dc5ecd57aff9ae22410facd31705

gunes-arm · 2026-01-15T20:33:58Z

arm_compute/runtime/NEON/functions/NETopKV.h

+ * @publicapi
+ */
+
+#include "arm_compute/core/Types.h"


Which types are needed from this file?

Status

If I remove it I get

In file included from src/runtime/NEON/functions/NETopKV.cpp:24:0: ./arm_compute/runtime/NEON/functions/NETopKV.h:79:12: error: ‘Status’ does not name a type; did you mean ‘static’? static Status ^~~~~~

That class is defined under arm_compute/core/Error.h

Replaced Types.h with Error.h in next patchset

gunes-arm · 2026-01-15T20:34:41Z

arm_compute/runtime/NEON/functions/NETopKV.h

+     */
+    void configure(const ITensor *predictions, const ITensor *targets, ITensor *output, const unsigned int k);
+
+    /** Static function to check if given info will lead to a valid configuration of @ref CPPTopKVKernel


CpuTopKVKernel

Done in next patchset

gunes-arm · 2026-01-15T20:40:35Z

src/cpu/kernels/CpuTopKVKernel.cpp

+#include "src/core/helpers/WindowHelpers.h"
+#include "src/cpu/kernels/topkv/list.h"
+
+#include <array>


vector? also we need to add cstdint

Done in next patchset

gunes-arm · 2026-01-15T20:48:43Z

src/cpu/operators/CpuTopKV.h

+    void
+    configure(const ITensorInfo *predictions, const ITensorInfo *targets, ITensorInfo *output, const unsigned int k);
+
+    /** Static function to check if given info will lead to a valid configuration of @ref CPPTopKVKernel


can we just say "similar to @ref CpuTopKV::configure()" to reduce duplication? Have a look at arm_compute/runtime/experimental/operators/CpuActivation.h

Same comment is valid for every header

Done in next patchset

gunes-arm · 2026-01-15T20:51:09Z

src/cpu/operators/CpuTopKV.cpp

+    ARM_COMPUTE_TRACE_EVENT(ARM_COMPUTE_PROF_CAT_CPU, ARM_COMPUTE_PROF_LVL_CPU, "CpuTopKV::run");
+    ARM_COMPUTE_ERROR_ON_MSG(tensors.empty(), "No inputs provided");
+
+    NEScheduler::get().schedule_op(_kernel.get(), 0, _kernel->window(), tensors);


instead of 0, we use Window::DimX

gunes-arm · 2026-01-15T21:23:38Z

tests/validation/reference/TopKV.cpp

+                *reinterpret_cast<const T *>(predictions(Coordinates{static_cast<int>(c), static_cast<int>(i)}));
+            const float v = static_cast<float>(vt);
+
+            // Mirror CPPTopKVKernel: (a - b > epsilon)


What does this mean?

The same way it's implemented in CPPTopKVKernel

gunes-arm · 2026-01-15T21:26:38Z

tests/validation/reference/TopKV.cpp

+{
+
+template <typename T>
+SimpleTensor<uint8_t> topkv(SimpleTensor<T> &predictions, SimpleTensor<uint32_t> &targets, uint32_t k)


I think we need to include

SimpleTensor.h

TensorShape.h

cstdint (because of int types)

CoreTypes.h for DataType

Coordinates.h
Anything else? We don't need the big Types.h file

Done in next patchset

gunes-arm · 2026-01-15T22:53:43Z

tests/validation/fixtures/TopKVLayerFixture.h

+#include "tests/IAccessor.h"
+#include "tests/validation/Helpers.h"
+#include "tests/validation/reference/TopKV.h"
+


I think we need to include cstdint, type_traits, random and algorithm STL headers for the functions used here.

Done in next patchset

gunes-arm · 2026-01-15T22:54:53Z

tests/validation/fixtures/TopKVLayerFixture.h

+#include "tests/AssetsLibrary.h"
+#include "tests/framework/Asserts.h"
+#include "tests/framework/Fixture.h"
+#include "tests/framework/ParametersLibrary.h"


Do we need

ParametersLibrary.h

IAccessor.h

tests/validation/Helpers.h

Removed in next patch

gunes-arm · 2026-01-15T22:57:08Z

tests/validation/fixtures/TopKVLayerFixture.h

+#define ACL_TESTS_VALIDATION_FIXTURES_TOPKVLAYERFIXTURE_H
+
+#include "arm_compute/core/TensorShape.h"
+#include "arm_compute/core/Types.h"


Do we need anything particular from this file?

Removed them in next patch

gunes-arm · 2026-01-18T13:39:20Z

src/cpu/kernels/CpuTopKVKernel.h

+
+#include "src/core/common/Macros.h"
+#include "src/cpu/ICpuKernel.h"
+


Looking at more carefully, I think we need to include the STL vector, string and cstdint headers in this file; not inside the corresponding cpp. Also, type_traits for std::add_pointer.

Same goes for ITensorInfo.h, ITensor.h.

We also need to include

arm_compute/core/Window.h for Window

arm_compute/core/CPP/CPPTypes.h for CPUInfo

src/cpu/kernels/CpuKernelSelectionTypes.h for DataTypeSelector.. stuff

arm_compute/core/Error.h for Status

In a nutshell, this file should be self-sufficient and we don't need to repeat the same includes in the corresponding .cpp file for brevity.

Can you please review this file and CpuTopKVKernek.cpp accordingly?

Updated in next patch

gunes-arm · 2026-01-18T13:39:49Z

src/cpu/kernels/CpuTopKVKernel.h

+     * @param[in]  src0   First input tensor info. Data types supported: QASYMM8/QASYMM8_SIGNED/F16/F32
+     * @param[in]  src1   Second input tensor info. Data types supported: U32
+     * @param[out] dst    The dst tensor info. Data types supported: U8
+     * @param[in]  policy Overflow policy.


Last argument is wrong

Fixed in next patch

gunes-arm · 2026-01-18T13:57:20Z

tests/validation/reference/TopKV.cpp

+    for (unsigned int i = 0; i < N; ++i)
+    {
+        // targets[i] (U32)
+        const uint32_t target_class = *reinterpret_cast<const uint32_t *>(targets(Coordinates{static_cast<int>(i)}));


I find this part particularly difficult to read. Why not just targets[i]?

gunes-arm · 2026-01-18T14:32:47Z

src/cpu/kernels/topkv/generic/neon/fp16.cpp

+{
+namespace cpu
+{
+void topkv_fp16_neon(const ITensor *in1, const ITensor *in2, ITensor *out, uint32_t k, const Window &win)


I feel there should be room to use templates in these implementations (especiall fps and quantized ones are very similar). The differences look like they can be handled using wrapper::.. type calls as in other kernels and type-enabled templates (e.g. for v7 variants, numeric_limits etc.). I think we could reduce the lines of code quite a bit.

Addressed in next patch. Following the usual library pattern, common code will live in src/cpu/kernels/topkv/generic/neon/impl.h

gunes-arm · 2026-01-18T15:11:07Z

src/cpu/operators/CpuTopKV.h

+{
+namespace cpu
+{
+/** Basic function to run @ref kernels::CpuActivationKernel */


CpuActivationKernel --> CpuTopKVKernel

Solved in next patch

gunes-arm · 2026-01-18T15:32:23Z

tests/validation/fixtures/TopKVLayerFixture.h

+
+            for (unsigned int n = 0; n < N; ++n)
+            {
+                const uint32_t target = *reinterpret_cast<const uint32_t *>(targets(Coordinates{static_cast<int>(n)}));


targets[i]??

gunes-arm · 2026-01-18T15:34:39Z

tests/validation/reference/TopKV.cpp

+        for (unsigned int c = 0; c < C; ++c)
+        {
+            const T vt =
+                *reinterpret_cast<const T *>(predictions(Coordinates{static_cast<int>(c), static_cast<int>(i)}));


If we use int for all these data types, we'd reduce the number of static casts quite a lot and improve readability of this part.

gunes-arm · 2026-01-18T15:35:09Z

tests/validation/reference/TopKV.h

+
+#include "tests/SimpleTensor.h"
+#include "tests/validation/Helpers.h"
+


#include <cstdint>

gunes-arm · 2026-01-18T15:35:40Z

tests/validation/reference/TopKV.h

+#define ACL_TESTS_VALIDATION_REFERENCE_TOPKV_H
+
+#include "tests/SimpleTensor.h"
+#include "tests/validation/Helpers.h"


Is this needed in this file?

gunes-arm · 2026-01-18T15:38:01Z

arm_compute/runtime/NEON/functions/NETopKV.h

@@ -0,0 +1,87 @@
+/*


We should add this operator to docs/user_guide/operator_list.dox

Done in next patch

* The Neon(TM) implementation of TopKV reduces execution time from 447.8 ms (CPP) to 11.65 ms for the same workload (F32, C=1000, N=32000, k=3, 6 threads), achieving an approximate 38× speedup. This gain comes from SIMD vectorization, removal of per-element branches, and a more efficient inner loop * Resolves ARMCL-1227. Change-Id: Ifdf161ce4254dc5ecd57aff9ae22410facd31705 Signed-off-by: Pablo Marquez Tello <[email protected]>

morgolock force-pushed the pr/cpu_topkv branch from cb2af0c to 1f0778c Compare January 16, 2026 09:28

gunes-arm requested changes Jan 16, 2026

View reviewed changes

morgolock force-pushed the pr/cpu_topkv branch 4 times, most recently from 89cd2b8 to cd41038 Compare January 16, 2026 16:25

gunes-arm requested changes Jan 19, 2026

View reviewed changes

morgolock mentioned this pull request Jan 19, 2026

NEON implementation for TopK/TopKV #1227

Open

morgolock force-pushed the pr/cpu_topkv branch 3 times, most recently from 541f913 to e0090dc Compare January 20, 2026 17:43

morgolock force-pushed the pr/cpu_topkv branch from e0090dc to 11238b6 Compare January 20, 2026 19:06


		#include "src/core/common/Macros.h"
		#include "src/cpu/ICpuKernel.h"


		#include "tests/SimpleTensor.h"
		#include "tests/validation/Helpers.h"

feat: Add NETopKV function. #1251

Are you sure you want to change the base?

feat: Add NETopKV function. #1251

Conversation

morgolock commented Jan 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gunes-arm Jan 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gunes-arm Jan 18, 2026 •

edited

Loading