What are the conditions for T to be accessed? You could potentially wrap each instance of T in an enum that has two states, Allow and Deny, where if you encounter a Deny you just don't access that one. If you have a slice of this enum where each enum owns T, you can always just swap out a Deny for an Allow or vice versa depending on how you want to do it. If you go this route, you might want to make a struct that owns the vector mutably, can return an immutable borrow to the slice, and change permissions on the slice. Now that I'm thinking about it, you can also implement a gets function on the struct that would return a Result depending on whether it was an allow or a deny. I don't think you'd need unsafe with this.
The solution mostly depends on the context surrounding how you're managing what can access T and at what time. I can see the solution I proposed not working if you need more specific permissions than allow or deny, or if the permissions are based on the caller.
Edit: I'm going to come back in a few hours and write code about what I mean in the first paragraph.