This is a standard permutation problem. It's a permutation problem as opposed to a combination because each book is considered distinct. About as basic a permutation problem as as you can get.
A truly basic permutation problem would have been. How many ways can you arrange 5 books. Answer would have been 5 ! or 5x4x3x2x1=120
For this problem we use the permutation formula with a slant of moving the books around N number at a time.
The formula for permutation is
Total Units ! /(Total-N) !
N is the amount of books taken at a time.
Plugging in, we get
5 books !/ (5-3) !
Solving we get