Did you know that celebrating birthdays, which in ancient Greece and Rome you did for a "god" or "emperor" (who wanted to be worshiped as a god) was considered a PAGAN practice by early Christians?
Did you know that in some middle eastern cultures they still don't celebrate birthdays?
Now do you understand why Jesus birthday celebration wasn't something the people living in Rome who knew Christ cared to do. Mary was alive and the most likely source of Luke's gospel which has the whole infancy narrative of Jesus. If he interviewed her, the woman who give birth to the man (even if you think he was an ordinary man) do you think Mary carried a calendar around?
We think Joseph was dead by the time Luke wrote his gospel, some 20 years after Jesus was crucified. The last mention of him was when Jesus was about 12, then there is nothing after that, whereas we hear about Jesus' cousins, Mary's sister, etc. etc.