Using Emacs as a better Source Insight

I wrote an article about using Emacs and its free software friends for code reading, but it is all in Chinese. So here I will show you how I used Emacs as a better Source Insight, in English😊

1 Use grep-mode relentlessly

For most functions of source insight, we need only 1 function in Emacs, and that is grep-mode. Need search for definition of a function? Use grep-mode. Need search for where a function is called? Use grep-mode. You just need several very different, very fast GREP(1) commands. In my setup, there are a couple of them: beagrep, gtags, ctags-exuberant, grep-gtags, grep-func-call, grep-func-call-all.

1.1 Grepping for something random

Here's the result of using beagrep for searching where something occurs. This is very slow in Source Insight, and especially if the project is very large (for e.g., the whole Android project), Source Insight will melt down. But I can grep a very familiar string ("hello world\n") in 9 gigabytes of Android source code in one third of a second. Here's the result: 

-*- mode: grep; default-directory: "~/src/android/external/chromium_org/tools/gyp/test/make/" -*-
Grep started at Wed Aug 20 09:28:31

beagrep -e "(\"hello world\\\\n\")"
/home/bhj/src/android/external/chromium_org/tools/gyp/test/make/main.cc:10:  printf("hello world\n");
/home/bhj/src/android/external/clang/bindings/python/tests/cindex/INPUTS/hello.cpp:4:    printf("hello world\n");
/home/bhj/src/android/external/clang/www/get_started.html:213:int main(int argc, char **argv) { printf("hello world\n"); }
/home/bhj/src/android/external/compiler-rt/BlocksRuntime/tests/localisglobal.c:19:void (^global)(void) = ^{ printf("hello world\n"); };
/home/bhj/src/android/external/compiler-rt/BlocksRuntime/tests/shorthandexpression.c:18:    void (^b)(void) = ^(void)printf("hello world\n");
/home/bhj/src/android/prebuilts/python/darwin-x86/2.7.5/lib/python2.7/test/test_memoryio.py:81:        buf = self.buftype("hello world\n")
/home/bhj/src/android/prebuilts/python/linux-x86/2.7.5/lib/python2.7/test/test_memoryio.py:81:        buf = self.buftype("hello world\n")

Grep finished (matches found) at Wed Aug 20 09:28:31

Beagrep source code is here and blog is here.

Note how I didn't use a screenshot for the above grep output! I customized my org-mode setup so that grep-mode and compilation-mode can be htmlized too😝

Also note how I put 4 backslashes before the n to grep the newline character \n in C. This is automatic, I wrote some Emacs Lisp for grep-mode, so that the correct grep regeex can be generated automatically.

1.2 Grepping for definitions

Here's how to grep for a function definition: 

-*- mode: grep; default-directory: "~/src/android/" -*-
Grep started at Wed Aug 20 09:33:29

grep-gtags -e "getExternalStorageDirectory" -v .xml
Found total 5/13 definitions: Entering directory `/home/bhj/src/android'
frameworks/base/core/java/android/os/Environment.java:177: method: <android.os.Environment.UserEnvironment.getExternalStorageDirectory> : public File getExternalStorageDirectory() {
frameworks/base/core/java/android/os/Environment.java:383: method: <android.os.Environment.getExternalStorageDirectory> : public static File getExternalStorageDirectory() {
frameworks/base/core/java/android/os/Environment.java:387: method: <android.os.Environment.getExternalStorageDirectory> : public static File getExternalStorageDirectory() {
external/robolectric/src/main/java/com/xtremelabs/robolectric/shadows/ShadowEnvironment.java:26: method: <com.xtremelabs.robolectric.shadows.ShadowEnvironment.getExternalStorageDirectory> : public static File getExternalStorageDirectory() {
external/chromium_org/base/android/java/src/org/chromium/base/PathUtils.java:114: method: <org.chromium.base.PathUtils.getExternalStorageDirectory> : public static String getExternalStorageDirectory() {

Grep finished (matches found) at Wed Aug 20 09:33:30

Note how I hacked grep-mode to show some respect to the Entering directory `/home/bhj/src/android'# thing, which used to be for compilation-mode only😊 I did this so that the grep output can be a bit shorter: instead of /home/bhj/src/android/frameworks/base/core/java/android/os/Environment.java:177:, I need only start with frameworks/.

grep-gtags is a shell/perl script that I wrote which integrated ctags-exuberant and gtags from GNU global. You may want to check out Ajoke.el which is a package for doing Java completion in Emacs (code here and blog here).

1.3 Grepping for where a definition is used

I use grep-func-call-all and grep-func-call for finding out how a function/variable/class etc. is used.

Here's an example: 

-*- mode: grep; default-directory: "~/src/android/" -*-
Grep started at Wed Aug 20 09:52:01

grep-func-call -e "ACTION_MEDIA_SCANNER_SCAN_FILE" --nc -a


Entering directory `/home/bhj/src/android/development/sdk'
Entering directory `/home/bhj/src/android/external/chromium_org/ui/android/java/src/org/chromium/ui'
SelectFileDialog.java:141: <=     public void onIntentCompleted(WindowAndroid window, int resultCode,
SelectFileDialog.java:158: =>                     Intent.ACTION_MEDIA_SCANNER_SCAN_FILE, mCameraOutputUri));
Entering directory `/home/bhj/src/android/frameworks/base/core/java/android/content'
Entering directory `/home/bhj/src/android/packages/apps/Backup/src/com/smartisanos/backup/activity'
BackupActivity.java:430: <=     private void notifyMediaScan(File file) {
BackupActivity.java:431: =>         Intent intent = new Intent(Intent.ACTION_MEDIA_SCANNER_SCAN_FILE);
Entering directory `/home/bhj/src/android/packages/apps/BrowserSmartisan/src/com/android/browser'
UploadHandler.java:65: <=     void onResult(int resultCode, Intent intent) {
UploadHandler.java:90: =>                         new Intent(Intent.ACTION_MEDIA_SCANNER_SCAN_FILE, result));
Entering directory `/home/bhj/src/android/packages/apps/Browser/src/com/android/browser'
UploadHandler.java:59: <=     void onResult(int resultCode, Intent intent) {
UploadHandler.java:84: =>                         new Intent(Intent.ACTION_MEDIA_SCANNER_SCAN_FILE, result));
Entering directory `/home/bhj/src/android/packages/apps/ContactsSmartisan/src/com/android/contacts/vcard'
ExportProcessor.java:96: <=     private void runInternal() {
ExportProcessor.java:211: =>             mService.sendBroadcast(new Intent(Intent.ACTION_MEDIA_SCANNER_SCAN_FILE,
Entering directory `/home/bhj/src/android/packages/apps/MmsSmartisan/src/com/android/mms/ui'
ComposeMessageActivity.java:2310: <=     private boolean copyPart(PduPart part, String fallback) {
ComposeMessageActivity.java:2402: =>                 sendBroadcast(new Intent(Intent.ACTION_MEDIA_SCANNER_SCAN_FILE,
Entering directory `/home/bhj/src/android/packages/apps/MmsSmartisan/src/com/android/mms/ui'
MessageUtils.java:804: <=     public static String saveAttachment(Context context, Uri data, String name) {
MessageUtils.java:853: =>             context.sendBroadcast(new Intent(Intent.ACTION_MEDIA_SCANNER_SCAN_FILE,
Entering directory `/home/bhj/src/android/packages/apps/Mms/src/com/android/mms/ui'
ComposeMessageActivity.java:1669: <=     private boolean copyPart(PduPart part, String fallback) {
ComposeMessageActivity.java:1751: =>                 sendBroadcast(new Intent(Intent.ACTION_MEDIA_SCANNER_SCAN_FILE,
Entering directory `/home/bhj/src/android/packages/apps/SoundRecorderSmartisan/src/com/smartisanos/recorder/provider'
RecorderProvider.java:86: <=     public Uri insert(Uri uri, ContentValues values) {
RecorderProvider.java:100: =>                     Intent.ACTION_MEDIA_SCANNER_SCAN_FILE,
RecorderProvider.java:136: <=     public int update(Uri uri, ContentValues values, String selection,
RecorderProvider.java:155: =>                                     Intent.ACTION_MEDIA_SCANNER_SCAN_FILE,
Entering directory `/home/bhj/src/android/packages/apps/SoundRecorder/src/com/android/soundrecorder'
SoundRecorder.java:897: <=     private Uri addToMediaDB(File file) {
SoundRecorder.java:944: =>         sendBroadcast(new Intent(Intent.ACTION_MEDIA_SCANNER_SCAN_FILE, result));
Entering directory `/home/bhj/src/android/packages/apps/UnifiedEmailSmartisan/src/com/android/mail/providers'
EmlAttachmentProvider.java:255: <=     private int copyAttachment(Uri uri, ContentValues values) {
EmlAttachmentProvider.java:340: =>                 final Intent intent = new Intent(Intent.ACTION_MEDIA_SCANNER_SCAN_FILE);
Entering directory `/home/bhj/src/android/packages/apps/UnifiedEmail/src/com/android/mail/providers'
EmlAttachmentProvider.java:253: <=     private int copyAttachment(Uri uri, ContentValues values) {
EmlAttachmentProvider.java:338: =>                 final Intent intent = new Intent(Intent.ACTION_MEDIA_SCANNER_SCAN_FILE);
Entering directory `/home/bhj/src/android/packages/apps/VideoEditor/src/com/android/videoeditor/service'
ApiService.java:4297: <=     private Uri exportToGallery(String filename) {
ApiService.java:4305: =>         sendBroadcast(new Intent(Intent.ACTION_MEDIA_SCANNER_SCAN_FILE,
Entering directory `/home/bhj/src/android/packages/providers/MediaProvider/src/com/android/providers/media'
MediaScannerReceiver.java:35: <=     public void onReceive(Context context, Intent intent) {
MediaScannerReceiver.java:64: =>                 } else if (Intent.ACTION_MEDIA_SCANNER_SCAN_FILE.equals(action) &&
Entering directory `/home/bhj/src/android/vendor/qcom/opensource/fm/FMRecord/src/com/codeaurora/fmrecording'
FMRecordingService.java:391: <=     private Uri addToMediaDB(File file) {
FMRecordingService.java:431: =>         sendBroadcast(new Intent(Intent.ACTION_MEDIA_SCANNER_SCAN_FILE, result));

Grep finished (matches found) at Wed Aug 20 09:52:06

Here's how it works. First I use beagrep (which is very fast) to find out all the files that contain ACTION_MEDIA_SCANNER_SCAN_FILE, then I use ctags-exuberant to parse each of these files. For e.g., in the file FMRecordingService.java above, ctags-exuberant tells me private Uri addToMediaDB(File file) { appears on line 391 and beagrep tells me that ACTION_MEDIA_SCANNER_SCAN_FILE is on line 431 in this file, and ctags-exuberant also tells me that there is no other function definition in between line 391 and 431, so I can safely assume that ACTION_MEDIA_SCANNER_SCAN_FILE is called by (actually used by, because it's not a callable thing but you know what I mean😊​) addToMediaDB.

2 Use sawfish to customize more key binding

I don't know why I don't do it in Emacs itself. Anyway, I have customized my sawfish desktop so that when I press F1, it will synthesize some events to the focused window (i.e. Emacs):

sawfish-binding-for-f1.png

And the synthesized keys will invoke grep-func-call for me in Emacs. 

(bind-keys global-keymap "F1" '(synthesize-multiple-events "ESC" "g" "f" "RET"))

I used the ScrLk key in sawfish to toggle a code reading mode for Emacs (think ECS/i toggle in vim😊) so that when this mode is on, I can press F1-F5 for all the above mentioned grepping commands, with just a single key, so this is very convinient, just like Source Insight, right?

And in the code reading mode, I can use Home/End keys to jump to the prev/next grep result, Left/Right to back/forth where I started grepping. The idea is from xcscope.el where the tags can be popped, I took it a step further, that they can also be pushed. So that when code reading, I can follow very deep down where the functions are called one by one using grep-func-call again and again, and then I can press Left key again and again to come back where I started. Then I can use the Right key to go down the call stack again.

For e.g., here's the binding in sawfish for Home/End: 

(bind-keys global-keymap "Home" '(synthesize-multiple-events "ESC" "g" "p"))
(bind-keys global-keymap "End" '(synthesize-multiple-events "ESC" "g" "n"))

And here's the help for the synthesized key in Emacs: 

M-g p (translated from <escape> g p) runs the command previous-error
(found in global-map), which is an interactive compiled Lisp function
in `simple.el'.

3 How to try out my setup

I must apologize that I have not put too much thinking (actually I put a lot but not enough) into how other people can use my setup. On github some Emacs gurus have published a complete emacs.d which is well organized. My setup is one step further, I not only want to mess with your .emacs and your emacs.d, but also your .bashrc and ~/system-config/bin, sigh…

But, in case you are not daunted by my mess, here's how to use my setup:

  1. Get debian. Ubuntu will do (I tested). Maybe get a new user account (or back up all of your account's config files). Get sudo for your account.
  2. cd ~; git clone --recursive https://github.com/baohaojun/system-config
  3. ~/system-config/bin/Linux/after-check-out.sh

Here's how to try out beagrep after you do the above setup:

  1. cd ~/system-config/; for-code-reading
  2. cd ~/system-config/; beagrep -e "(\"hello world\\\\n\")" #

Bug reports are welcome😊 Even more welcome are success stories of you using my set up😊 The most welcome is patches😊

Last but not least, can you read my emojis in this article? 😝