Ancestry interview question

math related problem in hacker rank