this post was submitted on 21 Aug 2026
10 points (100.0% liked)

Rust

8239 readers
31 users here now

Welcome to the Rust community! This is a place to discuss about the Rust programming language.

Wormhole

!performance@programming.dev

Credits

  • The icon is a modified version of the official rust logo (changing the colors to a gradient and black background)

founded 3 years ago
MODERATORS
 

I've been trying to contribute to statrs recently, and one of the issues I've been running into is that I'm trying to have an iterator that returns items in sorted order without cloning the underlying data. I'm working this PR if you wanted to take a look at my code so far. I looked up this problem and found this StackOverflow post about this topic from years ago but, frankly, I don't believe it. Assuming you're iterating from a vector, mutating the underlying vector is akin to keeping state on the order of the vector. Why cant that happen in a separate data structure? Is there a more efficient way to represent ordinality other than a vector? Creating an iterator is simply tracking traversal through that structure, which I think could be done using a bloom filter to track which indices have not been traversed yet. That just leaves the traversal algorithm itself. What information would an iterator need to know to make the best decision? Could I adapt a sorting algorithm to be an iterator?

I'm asking a lot of questions because I'm a statistics guys, not an algorithms guy. Any starting point or input is much appreciated!

top 10 comments
sorted by: hot top controversial new old
[–] sukhmel@programming.dev 4 points 16 hours ago

I haven't looked at the code, so my two cents may be irrelevant, but it sounds like you could define a structure that borrows the vector, keeps a separate vector of indices into borrowed vector, and has a current element index.

struct SortFacadeIterator<'a, T: Ord> {
    data: &'a Vec<T>,
    indices: Vec<usize>,
    position: usize,
}

Then in new you would need to sort the original vector but instead of mutating it you would store indices of a resulting sorted elements from the original, i.e. when passed ["b", "a", "c"] you would create index storage of [1, 0, 2]. After that you can iterate both ways, returning an element by index:

impl<'a, T: Ord> SortFacadeIterator<'a, T> {
    fn current(&self) -> &T {
        self.data[self.indices[self.position]]
    }
}

I think, maybe sorting a vector could be done by enumerating original vector and sort_by a value, also I'm not sure you need a full ordering, but I don't remember what a sort expects

[–] Gobbel2000@programming.dev 4 points 2 days ago (1 children)

Maybe a heap data structure is what you're looking for. Depending on the usecase you might have to first turn an unsorted Vec into a heap (runtime in O(n), which is still faster than fully sorting it), at that point you can both insert new elements and remove the minimum element within a time in O(log n).

[–] AshrafIbrahim03@programming.dev 1 points 1 day ago (1 children)

Wouldn't that essentially clone all the data in the vector in the first place? I'm trying to minimize memory by having an iterator implementation rather than another presorted data structure that clones the underlying data.

[–] Gobbel2000@programming.dev 1 points 1 day ago* (last edited 1 day ago)

You would have to either mutate the original vector or clone it, yes. If you really want to iterate through an immutable Vec with only constant memory usage, then you probably can't get around O(n^2) runtime, but maybe there is some time-memory-tradeoff where a small memory usage can get some time reduction.

[–] TehPers@beehaw.org 3 points 2 days ago* (last edited 2 days ago) (1 children)

First, let's start by looking at the Iterator trait. It has one method (well, two, but one relevant one):

fn next(&mut self) -> Option<Self::Item>

If the last item the iterator returns is your minimum value, you need to run the iterator all the way to the end to find and return that value.

If you do this, in order to return any other value the iterator returned, you needed to buffer it somewhere. You cannot traverse an iterator in reverse. This means that in order to properly sort an iterator, you need to collect the entire iterator into a buffer to first check whether the last value it returned was the minimum and thus should be the first value in the sorted iterator.

If you have access to a backing buffer or can iterate in reverse, other options become available. You don't need to buffer again because you can access previous elements which are stored somewhere else.

Iterators in the traditional sense represent streams of data that may or may not have a backing buffer, though. For example, repeat(1) is an infinite length iterator. Sorting an infinite length iterator would be computationally impossible, of course.

[–] AshrafIbrahim03@programming.dev 1 points 2 days ago (1 children)

One of the things I mentioned was that the iterator is defined from a Vector, making it a fixed size iterator. I'm wondering what constraints I might need to figure out implementing a sortediterator, if that means I need to enforce reverse iteration on it, then I'm OK with that. Im trying to minimize memory in this implementation, but if the tradeoff means higher compute time, that's fine with me. What other options are you referring to in your fourth paragraph?

[–] TehPers@beehaw.org 2 points 2 days ago* (last edited 2 days ago)

If you have a backing storage and random access, you can implement any sorting function and simply yield/return the values in order rather than commit them into a collection. For example, you can track the minimum value after one full iteration, then iterate in reverse tracking both the current minimum and the next minimum and yielding each current minimum, then repeat going the other direction, and so on until you're returning only the maximums. In theory, this shouldn't require any internal buffer at all, but does require a total ordering constraint on the type (and is O(n^2) and in practice slow af)

[–] BB_C@programming.dev 1 points 2 days ago (1 children)

Not sure what you're asking. If it's about sorting without moving values of big structs around, maybe something like this?

[–] AshrafIbrahim03@programming.dev 1 points 1 day ago (1 children)

That's essentially where the code started, where the vector was presorted, then moved to a function to be used for a calculation. I want to have the caller be able to pass a borrow of a collection, for the purposes of this problem right now a fixed size vector, and not have to move the vector into the function. The function needs the sorted data, but doesn't need to clone any of the data, just needs to run a calculation over it. Ideally the final implementation can a) take a borrow to a fixed size vector and b) not clone any of the underlying data in order to run its own calculations. The solution I'm toying with now is a sorted iterator, which shouldn't clone the underlying vector's data and should traverse the fixed size vector in a sorted order. Having an iterator whose next returns sorted items from the fixed size vector would be the perfect solution to this issue.