However your implementation based on std::function might allocate, which is then non-zero cost compared to normal exit statements. Better directly store the lambda inside the class, which requires that to be generic (Finally<LambdaType>).
That can be hidden by using the class via a templated function which uses types interference:
auto cleanup_socket = make_scope_guard([&]{ close_socket(s); });
Searching for scope_guard yields lots of alternative implementations.