I'm doing the same. Only once in a while I can't point myself at a specific interactive item but they built two ways to fix that (use left or right on d-pad to cycle through nearby objects or click left stick to switch to pointer-like controls). I'm super impressed so far with the controller support.
Long press the A button and it "searches" the area and gives you a list of interactable items in the area. Unfortunately, this system needs some polish because selecting items results in your character moving around to interact with items and this changes which other items are in range, and reorders others, but even with this I prefer this method.
I was going to suggest this too. Divinity 2 did searches with holding down the button too. And it will select everything in a radius around you that you can interact with and you just choose what you want and how to interact with it.
I normally will select the book and then read. I may or may not pick it up. I only like valuable books quest related books.
There does not appear to be any height limit to the search. I was at the mouth of a cave and selected some plant and my character ran all the way around the mountain to end up on top of the cave to pick the plant and I couldn't find a way to cancel it.
So, needs some work but it's still mostly useful. Haha