Heat oil in a large nonstick skillet over medium-high heat.
Season fish with salt and pepper and cook, skin side down, until skin is golden brown and crisp, about 5 minutes.
With a fish spatula (if you don't have a fish spatula, use a very thin spatula) to carefully turn fish over.
Add garlic, olives, and rosemary to skillet. Cook, stirring garlic, olives, and rosemary occasionally, until fish is opaque throughout, about 3 minutes.
Add orange juice to pan and swirl to combine.
Divide radicchio and fish among plates and spoon warm sauce over the top.