Playing by ear is certainly a skill that takes time to develop, and this can be different for everyone. While it sounds like you excelled early in reading, others have the opposite problem; learning to play by ear first, and the struggling with reading music later down the road.
I would absolutely recommend using the Practice Loop Library to get accustomed to playing along with (or improvising to) music. Specifically, music that doesn't require you to "compete" with another drummer, which can be challenging. Any "drumless" music is going to be easier to start with than music which already contains drums.
If you're looking to learn someone else's drum parts by ear, I would always start with subdivisions. If you can first identify the subdivision that you're hearing (16th notes, 32nds note, triplets, etc.) it's going to make the learning process much simpler. After the subdivision, you'll be on to things like patterns, dynamics, orchestrations, etc.
Identifying subdivisions by ear does take some time, but the Subdivision Masterclasses on this site will give you some solid starting points! I've linked the playlist below. Enjoy!