Using a yellow green filter on blue green water won't filter out much of the light. So you could expect the results you got. Basically, the colors of the ocean flow through the filter easily, making them brighter, whereas the colors of the beach, which are already bright, don't get dimmed too much. So by using that filter, you decreased the amount of contrast that you would normally get, even without a filter.
Orange, on the other hand, is the compliment of blue. Red is the compliment of Cyan, or blue green. Either would have filtered out much of the water (and sky) and made the water much darker than it would appear without a filter. The sand, being a normal sand color, I'm assuming, is pretty close to white (light brown) so it would have been made slightly brighter in comparison with the orange filter. Thus the sand gets brighter, water gets darker, and you have more contrast. Much more. A yellow or magenta filter might have given you something in between, with yellow probably being the better option to allow more of the sand color and slightly less of the sky and water.
Check out a color wheel for how these colors work. Colors on the opposite end of the color of the filter on the wheel are darkened while colors near the same side appear relatively brighter (though any colored filter will darken the overall image and require an increased exposure to compensate). It's also helpful to look through the filter before applying it (or if it's an SLR, just look through the viewfinder with the filter installed), and look for the differences in contrast in the scene. It can be a little hard to process at first because the entire image is colored, and the B&W film won't render that color, but you can at least notice the differences in contrast between elements.