For 35mm, I'd recommend buying from Freestyle and getting Arista Premium films. The 100 speed film is Plus-X, which is simply an amazing film. The 400 speed is Tri-X, which is also very highly regarded.
For 120, Plus-X and Tri-X are also good choices.
In those formats, all the brands are pretty much around the same price (Ilford and Kodak, that is).
For 4x5 and larger, Kodak gets kind of expensive and I don't really feel it's worth the extra money they charge. For 4x5, either go with Ilford FP4 or HP5 (their kinda-sorta Plus-X and Tri-X "equivalents), or Arista EDU Ultra film, which is very cheap. I am not a fan at all of EDU Ultra. Others love it, but I don't like the way it behaves.
As for developer, I'd recommend D-76 or Ilford's ID-11 equivalent.
In short, you simply go wrong with Plus-X or Tri-X (depending on the speed you need, 125 or 400), in D-76.
As you grow in your craft, try Rodinal or other developers. Once you master Tri-X in Rodinal, switch it up. Like JBrunner says, the key is to change one thing at a time.