The vector field
F: \mathbb R^2 \to \mathbb R^2,
\begin{pmatrix} y_1 \\ y_2 \end{pmatrix}\mapsto \begin{pmatrix} y_1^2 - y_2^2 \\
Ay_1 + By_1 y_2 \end{pmatrix}
defines a system y' = F(y).
Determine the coordinates of the stable fixed point
y_{\infty} = \left({\color{red}y_{\infty,1}},{\color{blue}y_{\infty,2}}\right) \neq (0,0).
{\color{red}y_{\infty,1}}
=
-A/B
{\color{blue}y_{\infty,2}}
=
-A/B
We first find the fixed points. These are the solutions of the equation F(y) =0.
The first equation is y_1^2 - y_2^2 = (y_1 - y_2) (y_1 + y_2) = 0 and therefore
y_1 = \pm y_2.
The second equation is
0= Ay_1 + By_1 y_2 = Ay_1 (1 + fractionReduce(B,A) y_2)
and with y_1 \neq 0 it follows y_2 = fractionReduce(-A,B).
Thus, we have two fixed points
\displaystyle
y_{\infty} = \left({\color{red}y_{\infty,1}},{\color{blue}y_{\infty,2}}\right)
= \left(\mp fractionReduce(A,B), fractionReduce(-A,B)\right).
With Jacobi matrix (and the theorem by Hartman-Grobman) we decide about stability.
It is D(y_1,y_2) = \begin{pmatrix} 2y_1 & - 2y_2 \\
A + By_2 & By_1 \end{pmatrix}.
Substituted with the fixed points above: D(y_{\infty,1},y_{\infty,2})
= \begin{pmatrix} \mp fractionReduce(2*A,B) & fractionReduce(2*A,B) \\
0 & \mpA \end{pmatrix}.
The real eigenvalues are on the diagonal and are both negative in case of y_1 = y_2. Thus,
y_{\infty} = \left({\color{red}y_{\infty,1}},{\color{blue}y_{\infty,2}}\right)
= \left(fractionReduce(-A,B), fractionReduce(-A,B)\right) stable.