Is it selfish for my husband and I to do this?

By all means, dont feel guilty. You have mad arrangements for your children, and you want to spend some quality time with your husband. This is a wonderful way to keep the love alive, and never feel bad about wanting to spend time with your husband. When kids grow up and are gone, you will only have each other. What a wonderful Christmas present, and after 5 kids you may want protection from having more, due to your wild christmas present..lol.