Chemical and mechanical instabilities in warm and dense nuclear matter