I second the recommendation for Warren Smith's "Modern Optical Engineering." (I'm a user of optical systems, not a designer, so I can more or less follow a lot of the book, but I would have to do some real work to work out any of the examples in full detail myself.) Another book at a similar level is Rudolph Kingslake's "Lens Design Fundamentals."
Thanks for "seconding" Warren Smith. Regarding Kinglake's book, I personally find/found it much more difficult. I actually started out with it and pretty quickly found myself under water, so to speak. So I went on to get the two preceeding books, by Conrady (who became Kingslake's father in law). (Kingslake's book is sorta considered as a 3rd book in the series.) To me, it's a long hard slog which I wasn't willing to fully commit to. Essentially the series is about designing lenses in the day when calculations were done by hand, frequently looking up log tables. Consequently calculation methods were as elegant and economical as possible, etc., using various shortcuts and minimizing the number of rays traced. So if one wants to learn the older ways, pre-personal-computer, these are your books. As a note, I believe that Kingslake's book was later updated, after his death, so perhaps the later edition is easier to follow? I dunno.
FWIW I am really not very good on the math. Nor have I had any formal training in optics. Someone who is very fluent in this sort of thing might breeze right through this.
For me, Warren Smith's book speaks in a manner that I can sorta visualize, which works better for me. Things are explained more through words than through formulas.
Of course, optical scientists designed lenses before there was specialized software, and an interesting look into those calculations is at Pencil of Rays:
https://www.pencilofrays.com/lens-design-spreadsheet/
Thanks for this link, I got immersed in it for a half hour or so before coming back to finish this post. (It's my idea of fun reading.)
Ps, like you I've been primarily a user of lenses, but have done some relatively crude designs which we actually used in our studio chain - several thousand copies of each. So every one there thought I was an optical expert when it was really just a bit more than THEY understood.
FWIW another cute little book, Walter Welford's Useful Optics, is compact and concise. Apparently a number of students can study theory but be a bit weak when it comes to actually rigging up a working optical system. So he cuts right to the chase. It certainly helped me in clarification of certain points. But I don't think it's a good beginner book.