I'm trying to downsize my gear and trying to decide on either Canon or Nikon.
Don't downsize! Keep both systems. I own Canon FD/FL, Canon EF, and Nikon manual focus system (plus two medium format systems) -- that's the way to happiness!!
Seriously, there are some great FD/FL lenses, EF lenses, and Nikon lenses, and losing one of the systems limits you from using a particular lens. I love the FD 135/2.5 for portraits, the FL 19/3.5R wideangle, the EF autofocus versatility, and the Nikkor 58/1.4, 135/3.5, 200/4.0, 35/2.8PC, 20/4.0 lenses for general purpose.