That can happen, but was likely a fluke. Paying off your loan reduces your credit utilization, which helps your score, but there are other factors too, like # of open accounts (there's a sweet spot) and age of accounts (longer=better). If your loan was your oldest account and you didn't have many or any other credit accounts, it could drop your score. Don't stress over 20-30pts though.
Better question to ask is what would you do with the money otherwise. If you're looking to spend it regardless, pay off the loan. If you want to save/invest it, do that and leave your loan be, it's probably like 0-3% interest right?