The following was my thought process:

It looks like a simple AP of small integers.

I will start with some concrete info. I know that 'sum of the fourth and seventh terms of this arithmetic progression is 40'
(a+3d) + (a + 6d) = 40
2a + 9d = 40

Note here that 2a is even and sum of 2a and 9d is even. So, 9d must be even too. This means, d, the common difference, must be even.
If d = 2, 9d = 18 and 2a = 40 - 18 = 22. But a = 11 is not in the options so ignore it.
If d = 4, 9d = 36 and 2a = 40 - 36 = 4. This gives us a = 2

Now check: a = 2, d = 4
1st term = 2, 3rd term = 10, 13th term = 50.
2, 10 and 50 are in GP. Good!
The answer is (A)
_________________

We can write \(a_1*a_{13}=a_3*a_3\) and \(a_3+a_7=40.\)
Since \(a_3=a_1+2d,\) (where \(d\) is the the constant defining the arithmetic progression), \(a_{13}=a_1+12d\), \(\, a_4=a_1+3d\), \(\, a_7=a_1+6d\), from the above two equations we get
\(a_1(a_1+12d)=(a_1+2d)^2\) and \(a_1+3d+a_1+6d=40.\) Further, \(12a_1d=4a_1d+4d^2\) or \(8a_1d=4d^2\) which leads to \(2a_1=d.\) (If \(d=0\) all the terms in the arithmetic progression are equal to 20, and so are those in the geometric progression. It should have been stated that the arithmetic progression is non-constant.)
The other equation leads to \(2a_1+9d=40\), so \(2a_1+18a_1=40,\) or \(a_1=2.\)

Answer A.

IMO, not a real GMAT question.
_________________

In my opinion one can solve this question by simple calculations pick an option from choices lets say 2
first one is arith matic sequence so it will be like 2,3,4,5,6,7,8,9,10,11,12,13,14,15,16
notice that square of two is 4 and square of 4 is i6 which is the thirteenth term so one can solve it by mental math.

That's wrong. The solution is: 2, 6, 10, 14, 18, 22, 26, ..., 50.
Geometric means constant factor, so in this case: 2, 10, 50 (constant factor being 5). And 14+26=40. It would be pretty hard to guess the correct solution. A lot of trial and error. It would take too long.
In your sequence, 16 isn't even the 13th term. It's the 15th.

If the first, third and thirteenth terms of an arithmetic progression are in geometric progression, and the sum of the fourth and seventh terms of this arithmetic progression is 40, find the first term of the sequence?

if 1 ,3 and 13 in ap are gp then (a+2d)sqr = a(a+12a)
and a+3d + a+6d = 40

solve the two eq.. we get a = 2

exactly
16 will be 15th term not 13th
that itself can refute further calculation based on that approach

Actually, it isn't very hard to guess. When you use 'brute force'/'hit and trial'/'logic'/'intuition', you need to have a point to start from.
You can certainly not say, "Ok, say the first term is 2 and common difference is 1 - not working. Say, common difference is 2 - not working etc etc."
You first need to analyze whatever info you have and shrink your world of possible APs. Otherwise, using the theoretical algebraic solution might actually be faster. The good thing is that as far as GMAT is concerned, a combination of logic and brute force works very well. Often, you will get the answer using logic alone. Sometimes, a couple of hit and trials are needed too.
_________________

If the first, third and thirteenth terms of an arithmetic progression are in geometric progression, and the sum of the fourth and seventh terms of this arithmetic progression is 40, find the first term of the sequence?


Slightly different algebraic approach.........

the first, third and thirteenth terms of an arithmetic progression are in geometric progression

First term = a
Third Term = a+2d
Thirteenth Term = a+12d

They are in Geometric Progression too
So \(\frac{(a+2d)}{a} = \frac{(a+12d)}{(a+2d)}\) --------------------------> {When a, b, c, d are in GP then b/a = c/b = d/c.........}

\((a+2d)^2 = a(a+12d)\) ----------> \(a^2 + 4d^2 + 4ad = a^2 + 12ad\) -----------> d^2 = 2ad -----------> d(d-2a)=0---------> Statement 1

the sum of the fourth and seventh terms of this arithmetic progression is 40 -------> (a+3d) + (a+6d) = 40 --------> 2a + 9d = 40 --------> \(d = \frac{40 - 2a}{9}\)

Putting \(d = \frac{40 - 2a}{9}\) in the Statement 1, we get \((\frac{40 - 2a}{9})\)\((\frac{40 - 2a}{9} - 2a)\) -------------------> \((\frac{40 - 2a}{9})\)\((\frac{40 - 20a}{9})\) ----------> \(\frac{(40-2a)(40-20a)}{81} = 0\) ------------------> Either (40-2a)=0 ------> a = 20 or (40-20a)=0 -----> a = 2, which is the answer
_________________

a1+3d+a1+6d=40
2a1+9d=40

a1=20-9d/2

all answer choices are integers, so 9d/2 should be integer less than 20 and 9d is even.
Only d=4 gives 18 to get 20-18=2

A

equation1: (a+2d)/a=(a+12d)/(a+2d)➡d=2a
equation2: 2a+9d=40
substituting,
2a+18a=40
a=2

Its not that difficult as its seems

Here => a+3d+a+6d=40 => 2a+9d=40 ---> equation 1

now a+2d/a=a+12d/a+2d => a^2+d^2+4ad=a^2+12ad

hence 8ad=4d^2 => d=2a
putting d=2a in equation 1
a=2

Smash that A
_________________

Given 1st,3rd and 13th term in GP.
so \(b^{2}\)=ac (GP Mean rule)
\(t_3\)\(^{2}\)=\(t_1\)*\(t_{13}\)
\((a+2d)^{2}\)=a(a+12d)
\(a^{2}\)+4ad+4\(d^{2}\)=\(a^{2}\)+12ad
4\(d^{2}\)=8ad
4d(d-2a)=0 (d is not equal to zero)
d=2a------------- (1)

Given \(t_4\)+\(t_7\)=40
a+3d+a+6d=40
2a+9(2a)=40 from (1)
20a=40
First term a =2

To clarify, I guess you're saying that if d=2, the first term would be 11 which is not in the answer choices. Otherwise d=2 also fits the constraints.