Well this is a very looong question.
Firstly, find the centre of this curve.
5x^2+5y^2+6xy-8=0=f(x,y) (Say)
For that you solve the equations
df/dy=0
and df/dx=0 simutlaneously
(Hope you know this. To find the centre of any second degree curve, differentiate the relation partially w.r.t x and then partially w.r.t. y
and set the two derivatives to zero. solve the two equations to obtain centre coordinates.)
Here, you can see that
df/dx=10x+6y=0
and df/dy=10y+6x=0s
so centre is clearly, origin (0,0).
Now consider a variable line passing through this centre (origin) making an angle A with positive x axis
So the equation of this line is y=xtanA
Now consider a point on this line at a distance "r" from the centre(origin), and which also lies on the curve
As the point lies on this line and also on the curve,
its X and Y coordinates may be written as x=rcosA and y=rsinA(Clearly.)
Further as the point lies on this curve, the point must satisfy the equation of the curve
so putting x=rcosA and y=rsinA in the eqn of the curve,
5r2cos2A+5r2sin2A+6r2sinA.cosA-8=0
therefore, r2(5sin2A+5cos2A+6sinA.cosA)-8=0
i.e. r2(5+3sin2A)=8
r2=8/(5+3sin2A)
Now r=sqrt(8/5+3sin2A)
now, r will be maximum when the denominator is minimum
The minimum value of the denominator is for sin2A=-1
i.e. rmax=sqrt{8/[5+3(-1)]}
=sqrt(8/2)=sqrt(4)=2
minimum value of r will be when the denominator is maxmimum
that happens for sin2A naximum i.e. for sin2A=1
i.e. rmin=sqrt(8/{5+3})
=1
You can find the points by using this data.
sin2A=-1
implies A=-pi/4
so the line is y=-x
and the points for max. distance are [-sqrt(2),sqrt(2)] and [sqrt(2) , -sqrt(/2)]
Similarly for the minimum distance,
y=x is the line since A=pi/4
so, the two points are [-sqrt(1/2),-sqrt(1/2)]and [sqrt(1/2),sqrt(1/2)] nearest to the origin.
Basically the equation represents an ellipse with major axis along the line y=-x and minor axis along the line y=x
Length of major and minor axes are 4 and 2 respectively. The centre of this ellipse is the origin.