Streamer

This is some quick notes on cross compiling xawtv. I had to get a webcam working on an embedded device this week and after not getting much success with a simple v4l app, (derived from simplewebcam android app's native library), I decided to cross compile xawtv. The only interesting part was the streamer console application.

I cross compiled this for arm. There was not much support for cross compilation as far as I could see.

XawTv depended on libv4l for which I had to compile v4l utils. This in turn needed libjpeg support. SourceForge gave me version 6 of libjpeg which had issues with libtool while cross compiling. Eventually I built version 9 which was obtained from some other location. To get the library, i had to use --enable-shared flag at the time of configuring. Then I compiled v4l utils. All this was done by setting LDFLAGS, CFLAGS and CXXFLAGS appropriately. As I remember, compiling v4l utils needed -Wl,-rpath-link=path_to_jpeglib in compiled flags.

There was some code that needed x11 libraries and I just commented them using # if 0. Another point to note is that output goes to .lib for these packages.

Now, for xawtv, I set the compiler and linker flags and configured using --host option to indicate that it is a cross compilation. However, I ran into problems as LDFLAGS was not honoured. After a long struggle, I figured out that flag was called DLFLAGS or something similar in Makefile. I set that to my current LDFLAGS value. Things went fine to some extend and I got streamer compiled. However, I was getting many warnings and errors since the build kept looking at /usr/include though I was cross compiling. Using -nostdinc only worsened the pain as I had to then use -I flag for each standard include folder and sub folders. So, I gave up on that and looked at what was causing the error. I could see that among many warnings there was only one error. It was related to FD_XXX macros of select.

Though I first ignored all these and decided to use only streamer (which was already compiled at this stage), my happiness was short lives as I found that the streamer used libng and the plugins present inside that folder.

I then commented many places where select appeared. However, same was appearing in v4l drivers. There alone I replaced select with Poll. My streamer and drivers were now ready. But, while running, streamer was not able to load drivers due to some unresolved symbols in drivers.

After a feel in entire source code, I found that they were present in libng and I only had a static library for libng. At this point, a word about makefiles in xawtv. The top folder has main Makefile. It then had Subdir.mk in each sub-folder which is included by main make file. Quick look at Subdir.mk Will give one a good idea about how to build an application/library. After seeing the objects that go into libng.a, I wrote a simple make file to get a corresponding .so file (arm-xxx-gcc -fPIC -shared needed.o -o libng.so). Then I wrote a similar make file for streamer and used this so to build streamer. (Streamer was using .a earlier). Then I copied the streamer and new libng.so to target.

Viola! "./streamer -o image.ppm -D v4l2" gave me my first image and I ended my day's work after seeing it via ppm2tiff. Well, I was really refreshing to see the same old mundane scenes via that image :)

Android logging

I was looking at building a framework for logging which is expected to be platform independent. However, truth is that it cannot be really platform independent. If I say that all processes send the log to the logging framework using IPC and stop there, I will not be doing justice to the job of defining the architecture. I need to see what type of IPC is better for our QAs. (Here the main quality attribute that we consider is performance).

Now comes another problem. If I evaluate the IPC for one of the platform (custom made Linux) against IPC that in Android, (two of the five or six platforms that I need to consider) it might so happen that IPC mechanism chosen for Linux is not available under Android. So, I eventually decided to go bit deeper and look at the details of IPC mechanisms available in Android.

It was not that easy to understand the whole framework in a short duration. However, the following page was really helpful.

http://elinux.org/Android_Logging_System

So, every application uses a library function and writes to some buffers in kernel. I checked the corresponding code. Lets start with Log.java

The file is present in frameworks/base/core/java/android/util/Log.java
No magic seems to be happening there. It just calls corresponding native function println_native
println_native is in frameworks/base/core/jni/android_util_Log.cpp. However, the real work happens in __android_log_buf_write which is inside system/core/liblog/logd_write.c (or logd_write_kern.c depending on compile time flag). It calls write_to_log which is a function pointer which very intelligently implements a singleton.

Let us look at logd_write.c first:
At first write_to_log is set to __write_to_log_init which calls __write_to_log_initialize. Which opens a file "/dev/pmsg0" and a Unix socket /dev/scoket/logdw. The pointer is then modified to point to __write_to_log_daemon and subsequent calls reach that function.

In this function (__write_to_log_daemon), the log is written to both the socket and the device. We will first see what happens to socket.

The listener is in system/core/logd/LogListener.cpp. However, this class extends SocketListener. The real work is done in SocketListener::runListener. The socket is read for each client here. It took me a while to find out how the generic SocketListener class passes data to LogListener. SocketListener class has a virtual function onDataAvailable(SocketClient *c) which is implemented by LogListener. When the SocketListener finds that there is data to be read from the socket, it calls the onDataAvailable function  which maps to corresponding function of LogListener class.

What happens in LogListener::onDataAvailable?
The message is read and it is put into a LogBuffer. It also informs the reader when a new message arrives. All the connected clients can now get the data from the reader.

Apps-->Log-->/dev/socket/logdw-->LogListener-->LogBuffer-->LogReader-->/dev/socket/logdr-->Clients of log reader!

Well, I will update the log what happens to /dev/pmsg0 later. Time to sleep. Anyway, my takeaway from this is that I can safely use unix sockets for IPC in both Android and Linux without bothering much about performance and my code might be portable across these two platforms if I take a little bit of care.

Note: I have looked at Anroid 6.0.1 source to get this information.

Getting back to blogging

For a long time I was not blogging anything. However, recently I had to study Lucene for some project and I also started work on a application that is being developed on Linux. This made me think that I should also restart this blog. As part of the new application that I am working on, I have started learning python and I will post more about it later. I have lot of things planned for future. On the Linux front, I am looking at getting a test machine setup and spend sometime on some of the industry standard software (mainly from Apache software foundation).

I will take sometime, but, I will surely restart these blogs!