Use of 'prototype' vs. 'this' in Javascript?

This is a common misunderstanding. The two given examples are really doing entirely different things.
We can take a look at the differences, but let's make a few mental notes about javascript before diving into it:

• The prototype of an object provides access to members of that object.
• The keyword this refers to the object context within which the function is executing.
• The language is functional, i.e. everything is an Object, including functions, and functions are values.

So here are the snippets in question:

var A = function () {
    this.x = function () {
        //do something
    };
};

In this case, variable A is assigned a function value. When that function is called, the runtime system will look for a variable "x" in the current object context. Without any additional code in the example, we could assume that this would be the global object. So to sum up: in this first snippet, this refers to the object invoking A().

var A = function () { };
A.prototype.x = function () {
    //do something
};

Something very different is happening in the second snippet. In the first line, variable A is assigned a function value. In javascript, functions are objects, and all objects have a prototype member. So in the second line, the object A is assigned a property x via the prototype. As you can see, this is completely different from the effect in the prior snippet.
For clarity, let's take a look at a third snippet. It's almost exactly like the first one (and may be what you meant to ask about):

var A = new function () {
    this.x = function () {
        //do something
    };
};

In this third example, I've simply added the new keyword. This changes the meaning of the function, turning it into what is commonly called a constructor function. When called with new, the function is run in a blank object context. In other words, because of the new keyword, this will refer to a blank object which will be newly created and applied to the function invocation. Property x will be assigned to the blank object, and the function will return the new object with property x, assigning it to variable A.
By making yet another small change, we have a fourth interesting case for comparison:

var A = function () {
    this.x = function () {
        //do something
    };
}();

In this fourth case, variable A is created and assigned the value of an executed inline function. The function is declared and immediately executed, so whatever value it returns is assigned to A. Here again, this will refer to the object context of the function call, and that object would have a property x assigned to it. The inline function doesn't explicitly return anything, so A will have a value of 'undefined'.
Related questions:

• What does it mean that javascript is a prototypal language?
• What is the scope of a function in Javascript?

Sidenote: There is not a real memory savings between the snippets in question. The first variable x belongs to the function value, and the second variable x is a member on an object. If you did have an inheritance chain (though our examples do not), the second snippet would potentially make x available to its children. It's really comparing apples to oranges.
If, on the other hand, you had two "A"-style objects, both with property x, each object's x would be made available to objects in an inheritance hierarchy via the prototype, regardless of whether they were explicitly assigned to a prototype object. The prototype always provides the access in the javascript system.
Javascript isn't a low-level language. It may not be very valuable to think of prototyping as a way to explicitly change the way memory is allocated.