c - enable/disable cache on intel 64bit machine: CD bit always set?

Question

I'm trying to disable all level of cache for my machine Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz in Xen. I wrote a tool to call the following assemble code to disable/enable the cache and show the CR0 register's value.

case XENMEM_disable_cache:
    __asm__ __volatile__(
        "pushq %%rax
"
        "movq %%cr0,%%rax
"
        "orq $0x40000000,%%rax
"
        "movq %%rax,%%cr0
"
        "movq %%cr0, %0
"
        "wbinvd
"
        "popq  %%rax"
        : "=r"(cr0)
        :
        :);
    // gdprintk(XENLOG_WARNING, "gdprintk:XENMEM_disable_cache disable cache!
    // TODO IMPLEMENT
");
    printk("<1>printk: disable cache! cr0=%#018lx
", cr0);
    rc = 0;
    break;

case XENMEM_enable_cache:
    __asm__ __volatile__(
        "pushq %%rax
"
        "movq %%cr0,%%rax
"
        "andq $0xffffffffbfffffff,%%rax
" /*~0x4000000*/
        "movq %%rax,%%cr0
"
        "movq %%cr0, %0
"
        "popq  %%rax"
        : "=r"(cr0)
        :
        :);
    printk("<1>printk: enable cache; cr0=%#018lx
", cr0);
    rc = 0;
    break;

case XENMEM_show_cache:
    __asm__ __volatile__(
        "pushq %%rax
"
        "movq %%cr0, %%rax
"
        "movq %%rax, %0
"
        "popq %%rax"
        : "=r"(cr0)
        :
        :);
    // gdprintk(XENLOG_WARNING, "gdprintk:XENMEM_show_cache_status! CR0 value is
    // %#018lx
", cr0);
    printk("<1>printk: XENMEM_show_cache_status! CR0 value is %#018lx
", cr0);
    return (long)cr0;

The code can compile and run. After I run the disable cache code, the system becomes extremely slow, which confirms the cache is disabled. In addition, the value of CR0 shows the CD bit is set when I run the disable cache code.

However, when I run the show cache code, the output shows the CD bit of CR0 is 0, no matter I disable/enable cache.

My question is:

Is the CD bit(30bit) of CR0 register always set 1 when cache is disabled?

If not, there must be something wrong with my code, could you please help me point out the error I made?

ANSWER:

The above code only set the CD bit of the CR0 register on the core where the code is running. We need to use the smp_call_function() to call the code on all cores!

My new question is:

If I disable cache and then enable cache using the above code, the CD bit of CR0 is cleared. However, the system's performance is still very very slow, just like when I disable the cache. So it seems to me that enabling the cache code does NOT work? However, since CD bit has been cleared, the enabling cache code should have worked! So the question is: How long should I wait after I enable cache so that I can have the same performance just like the performance before I disable cache?

BTW, when I run the enble cache code, the printk output shows that the CR0's CD bit is 0.

Answer 1

If you're on an SMP system, you should invoke the disable-cache code for every core with smp_call_function(), since it is theoretically possible that your show-cache code is running on a different processor. To use that function, #include <include/linux/smp.h>.

EDIT: smp_call_function() invokes the function pointer it is given only on other cores, not on the current one. Make sure to run the function on all cores by invoking the function yourself on the core that invokes smp_call_function().