Is the system time invariant?
No, the system is not time invariant because the output of an input signal shifted some value in time does not equal the output of the original signal, shifted the same value in time. In this system the coefficient or amplitude of the shifted output signal changes with time.
What input will yield an output of Y[n]=u[n-1]?
The system seems to work specifically on delta functions, so I take the approach of describing u[n] as an infinite sum of shifted deltas:
$ \ u[n] = \sum_{k=0} \delta [n-k] $
Inpunting this function into the system we get the output:
$ \ Y[n] = \sum_{k=0} (k+1)^{2} \delta [n-(k+1)] $
So now representing $ \ Y[n] = u[n-1] $ as a sum of deltas we get:
$ \ Y[n]= \sum_{k=0} \delta[n-(k+1)] $
