Help with C# basic Averager Project

using System;

namespace Treehouse.FitnessFrog
{

  class Program 
  {
    static void Main() 
    {
      Console.WriteLine ("Enter a number or type \"done\" to see the average.");
      var totalCount = 0.0;
      var averageNumber = 0.0;
      var answer = Console.ReadLine ();


      try
      {
        var number = double.Parse(answer);

        while (true)
        {
           if (answer.ToLower().Equals("done"))
           {
             Console.WriteLine ("Average: " + averageNumber);
             break;
           }
          else
          {
            number += 1;
            continue;
          }
        }

      }
      catch (FormatException)
      {
        Console.WriteLine ("That is not a valid input.");
      }



    }
  }
}

I dont know what Im doing wrong.

First off, I dont want the catch to override the Console.WriteLine when someone responds with "done". Everytime I put done, it just says That is not a valid input and it just ends.. if though it satisfies that if { condition.

2nd, I wanted that everytime someone put an number answer, it will be added to my variable averageNumber and then it will print the one in my Console.WriteLine saying "Totals : " + averageNumber

The only thing thats happening right now is when i put a number, it adds nonestop but only shows 0 over and over again.

and when i type done, it says "That is not a valid input", instead of saying the average.

What will I need to do so i can add every number in the input and then average all those numbers? ahhhhhhhhhhhhhhhh :(