I was seeing the same issue.. Did couple of reboots and waited for some hours... I decided it was enough.. and did the steps below
  • Navigate to the Raspberry pi2 Windows 10 Web UI : http::/IPADDRESS:8080
  • Login with the Administrator. (Dont forget the default password is p@ssw0rd and change it using Powershell)
  • Click Apps on the left pane
  • You will see the app as DefaultApp*
  • Choose an App which says IoTCoreDefaultApp*. Click Set As Default
  • It will prompt you for reboot. Click yes
  • .. and then it hung..
  • Manually rebooted and got the Welcome screen.. with Language options etc.
  • Choose your language.
  • Choose Next and you are all set
Although I'm not sure if this completes the setup etc. Seems like everything is working for me.