Have visited a number of islands – Where to next?

My wife and I have traveled extensively throughout the world, including a fair number of spots in the Caribbean, and are trying to figure out where to visit for our 25th anniversary. We only have 4 or 5 nights so we plan on visiting somewhere in the Caribbean since we’re based in the US. Here’s the challenge. We’ve been to a LOT of great islands/luxury resorts but can’t figure out where to go next. Places we have visited include:

• BVI – Peter Island
• Grenadines – Palm Island
• St. Lucia – Ladera
• St. Martin
• St. Kitts & Nevis – Nisbet
• Little Cayman
• Barbados
• Jamaica
• Dominican Republic

And honestly, I’m struggling with where to go next. We enjoy luxury but small luxury. We don’t want a 200 room resort but we do want something unique and with friendly staff but also somewhere we can enjoy a local beach BBQ along with a higher end restaurant. We want the local vibe but also some luxury.

Right now I’m leaning towards Dominica and staying at Secret Bay Hotel but I welcome suggestions. We’re an active couple (enjoy hiking, etc.) and while we enjoy the beach, if it’s not white sand, that’s not the end of the world. We just don’t want to be in a spot where cruise ships are arriving 3 at a time each day.

TL;DR – where should I take my wife for our 25th wedding anniversary?