SCU on ARM11MPCore will not give you coherency

Well, we thought otherwise. But, that was not correct. This was a very interesting experience. We started writing the communication module to make two different cores of ARM11MPCore communicate and found that things are not going fine as we were expecting them to. We write from one core and we cannot read it back from other. First suspect was the new virtual addresses that had happened in VxWorks. However, that was not the issue. After reading about it a bit, we were convinced that it must have been a linear mapping in our case. Now, what was the reason? The next suspect was the cache. However, the process was not simple. No one except me in the whole group believed that it can be caused by cache. Also, I myself was doubtful as I felt SCU should take care of it.

To clear our doubt, we talked to a manager in our company who was an expert in JTAG. He appeared completely assured of the fact that we are doing something dumb with JTAG as we cannot see this memory even in JTAG. When I asked about cache coherency issues, he brushed aside my doubts saying that it can never happen.

Well,  after giving the advice, he walked away without turning back. But, I was the tech lead for this project and I had to get this working somehow. And, something kept telling me that this is a cache issue. So, I decided to keep hammering at that point. Now, we were writing from VxWorks and reading from Linux. I decided to look for code which will flush cache. But, being a noob in VxWorks, I could not find this code. We tried some silly tricks in the beginning like increasing the memory size in the hope that cache will get flushed auto-magically. Fortunately or unfortunately, these tricks did not work. However, one guy in our team was able to dig out VxWorks function to flush cache. It was a simple function that took the cache type, address and length as the arguments and it did its job! (Yes that was simple compared to what I found for Linux. We had to get physical address of the memory, get page structure and then flush the page from cache!) Yes, after flushing the cache, we were able to see what was being written by the second core into the memory location from the code that was running in first core. Later, another member in my team found that SCU will not do this in case of ARM11MPCore. It does this in case of some other newer chips (Cortex-A9). We also read that allocating DMA memory might do the flushing directly. We are yet to try this.