Can we go to their house?

A few days ago I took the kids to the park. Dallin had a great time playing with some boys who were a little bit older than him. As soon as we got in the car to go home he asked me if we could go to their house. I told him no, because we don't know them. Then he said, "Maybe if we got straight and then left and then right we will get to their house." I tried to explain to him that even if we did know the way to their house, we couldn't just go there, because we don't really know them. Then he told me that he told them that his name is Dallin, so they know him. He was so disappointed that I insisted that we can't go to their house because I have never met their parents.