The general form of the Gegenbauer ODE is given by the following:

Gegenbauer_ode := (x^2-1)*diff(y(x),x,x)-(2*m+3)*x*diff(y(x),x)+lambda*y(x)=0;

$\mathrm{Gegenbauer\_ode}≔\left(x^2-1\right)\left(\frac{{\ⅆ}^2}{\ⅆx^2}y\left(x\right)\right)-\left(2m+3\right)x\left(\frac{\ⅆ}{\ⅆx}y\left(x\right)\right)+\mathrm{\lambda }y\left(x\right)=0$

where m is an integer. See Infeld and Hull, "The Factorization Method". The solution of this type of ODE can be expressed in terms of the LegendreQ and LegendreP functions:

$\mathrm{with}\left(\mathrm{DEtools}\,\mathrm{odeadvisor}\right)$

$\left[\mathrm{odeadvisor}\right]$

$\mathrm{odeadvisor}\left(\mathrm{Gegenbauer\_ode}\right)$

$\left[\mathrm{\_Gegenbauer}\right]$

$\mathrm{dsolve}\left(\mathrm{Gegenbauer\_ode}\right)$

$y\left(x\right)=\mathrm{c\_\_1}{\left(x^2-1\right)}^{\frac{5}{4}+\frac{m}{2}}\mathrm{LegendreP}\left(\sqrt{m^2-\mathrm{\lambda }+4m+4}-\frac{1}{2}\,\frac{5}{2}+m\,x\right)+\mathrm{c\_\_2}{\left(x^2-1\right)}^{\frac{5}{4}+\frac{m}{2}}\mathrm{LegendreQ}\left(\sqrt{m^2-\mathrm{\lambda }+4m+4}-\frac{1}{2}\,\frac{5}{2}+m\,x\right)$