How I have fixed a lot of memory leaks in low-mem embedded system

Introduction
1. Using post-mortem analysis on low-mem board: Analysis of DDR RAM dumps
2. Fixing of compiler warnings
3. Static code analysis
4. Using HW mocking to debug memory leaks
Conclusion

Introduction

So we have some tiny famous commercially-success device. Device is Linux based build on tightly integrated ARM platform. Also there are a bunch of memory leaks. Device has 32 Megabytes of DDR RAM. On device start there are spike of memory consumption up to 30 MBytes but with following rollback to 24Mb. But after some time we crash with OOM. So now I describe few trivial debug techniques which helped me to improve situation significantly.

Following material is targeted for juniors but some mature programmers also find it useful.

As far as it was known on this time we had really huge amount of leak, so on first iter I refused to deal with complex and magical debugging and used simple and trivial things but many-many times.

It was:
1. Analysis of DDR RAM dumps
2. Fixing of compiler warnings
3. Static code analysis
4. Running of Valgrind on mocked application.

And now some details:

1. Using post-mortem analysis on low-mem board: analysis of DDR RAM dumps

So in steps to debug this we put memory dumping mechanism in kernel OOM handler and also pass-through it to userspace. It was implemented in the following way: when OOM was raised we set appropriate flag at certain predefined memory address, flushed caches and called the reset IRQ handler. As first stage of reset handler we checked memory flag and, if setted, we put a copy of register file and DDR to file on SD CARD. On finish of coping we got data suitable for T32, crash tool and other similar utilities. Now I will describe how I dealt with crash tool:

1. Open ramdump in crash
2. Do “files” command
3. Look for repeatable patterns like:

...  
 19  c3675080  c3902d80  c3823000  REG   /dev/ccinet0                           
 20  c36cfb80  c3902d80  c3823000  REG   /dev/ccinet0                           
 21  c3ede600  c3902d80  c3823000  REG   /dev/ccinet0                           
 22  c36cf780  c3902d80  c3823000  REG   /dev/ccinet0
...

    $ find -name “*.[chS]” -print0 | xargs -0 grep “/dev/ccinet0”

By fact on this step we have got ~1 Mbyte of additional free memory. And, also, you are not limited by “files” functionality of crash. I have illustrated my investigations via “files” because it was the most effective one in my situation. Don’t hesitate to use all power of this excellent tool.

2. Fixing of compiler warnings

We all are programmers so I will be short here:

Keep on sawing, Shura, keep on sawing! They are surely golden!

Here I have saved similar amount of free memory.

3. Static code analysis

I have used cppcheck:

1. cd /path/to/codebase/
2. cppcheck \
        -I"$BIONIC_INC" \
        -I"$BIONIC_ARCH_INC" \
        --enable=all -v --force . 2>&1 \
        | tee log.cppcheck
3. vim log.cppcheck and have gone in a way similar to prev chapter.

Achieved results are similar to prevs chapters.

4. Using HW mocking to debug memory leaks

How about using our lovely valgrind with low-mem systems? If we are large company and have enough time we can sold additional special debug version of board with enough qty of resources to ran valgrind. But if we can’t? Or if we don’t have enough time? Or if we are small company?

If we were enough smart we already have, as i have illustrated in my previous article, HAL and HW simulator. So we can run our SW in simulator under valgrind on enough power host. With enough good and right HAL and HW simulator we will be able to fix in this way all or almost all memory related issues.

But if we are still learning, lazy or not enough smart? We can try to mock HW in some way, for example using some mocking framework like GoogleMock. But follow I will illustrate it using limited mocking functionality of GoogleTest because it was enough in my situation.

So we have an application implemented as an instance of PacketManager class:

$ cat Makefile
.PHONY: all
all: App1-gtest_run

App1-gtest: $(Common_OBJECTS) \
		${CURDIR}/tests/PacketManager-test.cpp
	g++ -o $@ \
		${CURDIR}/gtest/gtest-all.cc \
		${CURDIR}/gtest/gtest_main.cc \
		${CURDIR}/tests/PacketManager-test.cpp \
		$(App1_Static_LIBS) \
		$(LIBS)

.PHONY: App1-gtest_run check test
App1-gtest_run check test: App1-gtest
	mkdir -p valgrind-gtest; \
	valgrind  -v --read-var-info=yes \
		--leak-check=full \
		--track-origins=no \
		--log-file=valgrind-gtest/App1-gtest.`date +%F_%T`.valgrind \
			./App1-gtest

.PHONY: check-strace
check-strace: App1-gtest
	export DIR="strace-gtest/App1.$(shell date +%F_%T).trc"; \
	mkdir -p "$${DIR}"; \
	strace -ff -tt -s 1024 -o "$${DIR}/App1-gtest.trc" \
		./App1-gtest

.PHONY: App1-gtest_run-all check-all test-all
App1-gtest_run-all check-all test-all: App1-gtest
	mkdir -p valgrind-gtest; \
	valgrind  -v --read-var-info=yes \
		--leak-check=full \
		--track-origins=yes \
		--show-reachable=yes \
		--log-file=valgrind-gtest/App1-gtest_all.`date +%F_%T`.valgrind \
			./App1-gtest --gtest_also_run_disabled_tests

.PHONY: check-strace-all
check-strace-all: App1-gtest
	export DIR="strace-gtest/App1.$(shell date +%F_%T).trc"; \
	mkdir -p "$${DIR}"; \
	strace -ff -tt -s 1024 -o "$${DIR}/App1-gtest.trc" \
		./App1-gtest --gtest_also_run_disabled_tests

.PHONY: cppcheck
cppcheck:
	mkdir -p cppcheck; \
	cppcheck --enable=all -v . 2>&1 | tee cppcheck/App1.`date +%F_%T`.cppcheck

$ cat tests/PacketManager-test.cpp
#include <cstdlib>
#include <gtest/gtest.h>
#include <UI/UItest.h>

#include <PacketManager.h>


namespace app1 {

class PacketManagerTest: public ::testing::Test {
protected:
	PacketManagerTest()
	{
		enum { argc = 2 };
		char argv_0[] = "./App1";
		char argv_1[] = "-DESTINATION=./tests/";
		char* argv[argc] = { argv_0, argv_1 };

		ui = new UItest();
		ac = new AppConfig(1, "", argc, argv, ui, NULL);
		bs = new BinStream(NULL);
		pm = new PacketManager(bs, NULL, ac, ui, NULL);
	}

	~PacketManagerTest() {
		delete pm; pm = NULL;
		delete ac; ac = NULL;
		delete bs; bs = NULL;
		delete ui; ui = NULL;
	}

	PacketManager *pm;
	AppConfig     *ac;
	BinStream     *bs;
	UI            *ui;

	enum
	{
		PcktInitState       = PacketManager::PcktInitState      ,
		PcktTypeState       = PacketManager::PcktTypeState      ,
		PcktLenState        = PacketManager::PcktLenState       ,
		PcktEndState        = PacketManager::PcktEndState       ,
	};


	inline CircularBuffer* GetRingBuffer() const
		{ return pm->ringBuffer; }

	inline Statistics* GetApp1Statistics() const
		{ return &pm->app1Statistics; }

	inline int GetPcktState() const
		{ return pm->pcktState; }

	inline void IterStatesPcktTable(Frame& frame) const
		{ pm->IterStatesPcktTable(frame); }
};

TEST_F(PacketManagerTest, Init_Read_EndPacket){
	int i;

	/// It's real data captured from device some time before
	unsigned char const rxBuff[DATA_PACKET_SIZE] = {
		'D',  'A',  'T',  'A',
		DATA_PACKET,
		0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, //< CRC
		0x00, 0x01, 0x02, 0x03,                   //< Timestamp
		0x00, 0x00, 0x00, 0x00,                   //< Data ID
		0x00, 0x00, 0x00, 0x01,                   //< header length (24 bytes)
		0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
	};

	EXPECT_EQ(DATA_PACKET_SIZE, sizeof(rxBuff));

	/// Here we deliver our data to application logic
	EXPECT_EQ(sizeof(rxBuff), pm->SavePacketReceived(rxBuff, sizeof(rxBuff)));
	EXPECT_EQ(sizeof(rxBuff), GetRingBuffer()->SpaceUsed());

	/// Here we run main application class on our data
	Frame frame(GetApp1Statistics(), NULL, NULL);
	EXPECT_EQ(sizeof(rxBuff), frame.Read(GetRingBuffer()));
	EXPECT_EQ(0, GetRingBuffer()->SpaceUsed());
	EXPECT_EQ(PcktInitState, GetPcktState());
	EXPECT_FALSE(frame.is_processed());

	IterStatesPcktTable(frame);
	EXPECT_EQ(4, frame.GetIndex());
	EXPECT_EQ(PcktTypeState, GetPcktState());
	EXPECT_FALSE(frame.is_processed());

	IterStatesPcktTable(frame);
	EXPECT_EQ(5, frame.GetIndex());
	EXPECT_EQ(PcktLenState, GetPcktState());
	EXPECT_FALSE(frame.is_processed());

	IterStatesPcktTable(frame);
	EXPECT_EQ(PACKET_HEADER_SIZE, frame.GetIndex());
	EXPECT_EQ(PcktEndState, GetPcktState());
	EXPECT_FALSE(frame.is_processed());

	IterStatesPcktTable(frame);
	EXPECT_EQ(PACKET_HEADER_SIZE, frame.GetIndex());
	EXPECT_EQ(PcktInitState, GetPcktState());
	EXPECT_FALSE(frame.is_processed()); //< @warning WA for undefined reason
	//EXPECT_TRUE(frame.is_processed());
}
}
$ make
...
$ cat valgrind-gtest/App1-gtest.2013-04-18_18:17:10.valgrind
==17112== Memcheck, a memory error detector
==17112== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==17112== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==17112== Command: ./App1-gtest
==17112== Parent PID: 17106
==17112== 
--17112-- 
--17112-- Valgrind options:
--17112--    --suppressions=/usr/lib/valgrind/debian-libc6-dbg.supp
--17112--    -v
--17112--    --read-var-info=yes
--17112--    --leak-check=full
--17112--    --track-origins=no
--17112--    --log-file=valgrind-gtest/App1-gtest.2013-04-18_18:17:10.valgrind
--17112-- Contents of /proc/version:
--17112--   Linux version 3.2.0-40-generic (buildd@allspice) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #64-Ubuntu SMP Mon Mar 25 21:22:10 UTC 2013
--17112-- Arch and hwcaps: AMD64, amd64-sse3-cx16
--17112-- Page sizes: currently 4096, max supported 4096
--17112-- Valgrind library directory: /usr/lib/valgrind
--17112-- Reading syms from /home/vitaly/Projects/App1/App1/App1-gtest (0x400000)

parse_type_DIE: confused by:
 <4><32fc36>: DW_TAG_structure_type
     DW_AT_declaration : 1	

--17112-- WARNING: Serious error when reading debug info
--17112-- When reading debug info from /home/vitaly/Projects/App1/App1/App1-gtest:
--17112-- parse_type_DIE: confused by the above DIE
--17112-- Reading syms from /lib/x86_64-linux-gnu/ld-2.15.so (0xd181000)
--17112--   Considering /lib/x86_64-linux-gnu/ld-2.15.so ..
--17112--   .. CRC mismatch (computed eabdc7b7 wanted 3ee54b4e)
--17112--   Considering /usr/lib/debug/lib/x86_64-linux-gnu/ld-2.15.so ..
--17112--   .. CRC is valid
--17112-- Reading syms from /usr/lib/valgrind/memcheck-amd64-linux (0x38000000)
--17112--   Considering /usr/lib/valgrind/memcheck-amd64-linux ..
--17112--   .. CRC mismatch (computed fd32bc40 wanted 1b1bde8c)
--17112--    object doesn't have a symbol table
--17112--    object doesn't have a dynamic symbol table
--17112-- Reading suppressions file: /usr/lib/valgrind/debian-libc6-dbg.supp
--17112-- Reading suppressions file: /usr/lib/valgrind/default.supp
==17112== embedded gdbserver: reading from /tmp/vgdb-pipe-from-vgdb-to-17112-by-vitaly-on-???
==17112== embedded gdbserver: writing to   /tmp/vgdb-pipe-to-vgdb-from-17112-by-vitaly-on-???
==17112== embedded gdbserver: shared mem   /tmp/vgdb-pipe-shared-mem-vgdb-17112-by-vitaly-on-???
==17112== 
==17112== TO CONTROL THIS PROCESS USING vgdb (which you probably
==17112== don't want to do, unless you know exactly what you're doing,
==17112== or are doing some strange experiment):
==17112==   /usr/lib/valgrind/../../bin/vgdb --pid=17112 ...command...
==17112== 
==17112== TO DEBUG THIS PROCESS USING GDB: start GDB like this
==17112==   /path/to/gdb ./App1-gtest
==17112== and then give GDB the following command
==17112==   target remote | /usr/lib/valgrind/../../bin/vgdb --pid=17112
==17112== --pid is optional if only one valgrind process is running
==17112== 
--17112-- REDIR: 0xd1999e0 (strlen) redirected to 0x380625c7 (???)
--17112-- Reading syms from /usr/lib/valgrind/vgpreload_core-amd64-linux.so (0xdba6000)
--17112--   Considering /usr/lib/valgrind/vgpreload_core-amd64-linux.so ..
--17112--   .. CRC mismatch (computed 8f05ac98 wanted c6d0ab1a)
--17112--    object doesn't have a symbol table
--17112-- Reading syms from /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so (0xdda8000)
--17112--   Considering /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so ..
--17112--   .. CRC mismatch (computed fc059af4 wanted ffc668e8)
--17112--    object doesn't have a symbol table
--17112-- REDIR: 0xd199850 (index) redirected to 0xddacc60 (index)
--17112-- REDIR: 0xd1998d0 (strcmp) redirected to 0xddadc20 (strcmp)
--17112-- Reading syms from /lib/x86_64-linux-gnu/libpthread-2.15.so (0xdfb3000)
--17112--   Considering /lib/x86_64-linux-gnu/libpthread-2.15.so ..
--17112--   .. CRC mismatch (computed 8e430e5b wanted e619a829)
--17112--   Considering /usr/lib/debug/lib/x86_64-linux-gnu/libpthread-2.15.so ..
--17112--   .. CRC is valid
--17112-- Reading syms from /lib/x86_64-linux-gnu/libc-2.15.so (0xe1d0000)
--17112--   Considering /lib/x86_64-linux-gnu/libc-2.15.so ..
--17112--   .. CRC mismatch (computed 3af7ebbf wanted 50fc58fa)
--17112--   Considering /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.15.so ..
--17112--   .. CRC is valid
--17112-- warning: addVar: unknown size (to_buf)
--17112-- warning: addVar: unknown size (p)
--17112-- warning: addVar: unknown size (info)
--17112-- REDIR: 0xe25ce30 (strcasecmp) redirected to 0xdba6610 (_vgnU_ifunc_wrapper)
--17112-- REDIR: 0xe2591d0 (strnlen) redirected to 0xdba6610 (_vgnU_ifunc_wrapper)
--17112-- REDIR: 0xe25f100 (strncasecmp) redirected to 0xdba6610 (_vgnU_ifunc_wrapper)
--17112-- REDIR: 0xe25abc0 (__GI_strrchr) redirected to 0xddaca80 (__GI_strrchr)
--17112-- REDIR: 0xe252f40 (malloc) redirected to 0xddac660 (malloc)
--17112-- REDIR: 0xe25bc10 (memset) redirected to 0xdba6610 (_vgnU_ifunc_wrapper)
--17112-- REDIR: 0xe25bc50 (__GI_memset) redirected to 0xddaf080 (memset)
--17112-- REDIR: 0xe2617e0 (memcpy@@GLIBC_2.14) redirected to 0xdba6610 (_vgnU_ifunc_wrapper)
--17112-- REDIR: 0xe317f90 (__memcpy_ssse3_back) redirected to 0xddadf30 (memcpy@@GLIBC_2.14)
--17112-- REDIR: 0xe253580 (free) redirected to 0xddab7c0 (free)
--17112-- REDIR: 0xe25b5f0 (bcmp) redirected to 0xdba6610 (_vgnU_ifunc_wrapper)
--17112-- REDIR: 0xe32d500 (__memcmp_sse4_1) redirected to 0xddaeca0 (bcmp)
--17112-- REDIR: 0xe25bbc0 (memcpy@GLIBC_2.2.5) redirected to 0xdba6610 (_vgnU_ifunc_wrapper)
--17112-- REDIR: 0xe31d530 (__memmove_ssse3_back) redirected to 0xddadd20 (memcpy@GLIBC_2.2.5)
--17112-- REDIR: 0xe2590f0 (__GI_strlen) redirected to 0xddacfc0 (__GI_strlen)
--17112-- REDIR: 0xe2575b0 (strcmp) redirected to 0xdba6610 (_vgnU_ifunc_wrapper)
--17112-- REDIR: 0xe301e10 (__strcmp_sse42) redirected to 0xddadb60 (strcmp)
--17112-- REDIR: 0xe2542a0 (calloc) redirected to 0xddaacf0 (calloc)
--17112-- REDIR: 0xe2590a0 (strlen) redirected to 0xdba6610 (_vgnU_ifunc_wrapper)
--17112-- REDIR: 0xe332bd0 (__strlen_sse2_pminub) redirected to 0xddacfa0 (strlen)
--17112-- REDIR: 0xe2592f0 (__GI_strncmp) redirected to 0xddad480 (__GI_strncmp)
--17112-- REDIR: 0xe2574f0 (index) redirected to 0xdba6610 (_vgnU_ifunc_wrapper)
--17112-- REDIR: 0xe301d60 (__strchr_sse42) redirected to 0xddacb20 (index)
--17112-- REDIR: 0xe262d00 (strchrnul) redirected to 0xddaf3b0 (strchrnul)
--17112-- REDIR: 0xffffffffff600000 (???) redirected to 0x380625b3 (???)
--17112-- REDIR: 0xe3043b0 (__strcasecmp_sse42) redirected to 0xddad500 (strcasecmp)
--17112-- REDIR: 0xe262af0 (__GI___rawmemchr) redirected to 0xddaf400 (__GI___rawmemchr)
--17112-- REDIR: 0xe25b2a0 (memchr) redirected to 0xddadce0 (memchr)
--17112-- REDIR: 0xe258a80 (__GI_strcpy) redirected to 0xddad0b0 (__GI_strcpy)
--17112-- REDIR: 0xe2575f0 (__GI_strcmp) redirected to 0xddadbc0 (__GI_strcmp)
==17112== 
==17112== HEAP SUMMARY:
==17112==     in use at exit: 7,848 bytes in 95 blocks
==17112==   total heap usage: 1,896 allocs, 1,801 frees, 87,643,726 bytes allocated
==17112== 
==17112== Searching for pointers to 95 not-freed blocks
==17112== Checked 209,931,568 bytes
==17112== 
==17112== 38 bytes in 1 blocks are possibly lost in loss record 8 of 61
==17112==    at 0xDDAC6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==17112==    by 0x64CDBC: operator new(unsigned long) (in /home/vitaly/Projects/App1/App1/App1-gtest)
...
==17112==    by 0x4115A1: app1::protobuf_AddDesc_app1lite_2eproto() (app1lite.pb.cc:305)
==17112==    by 0x406998: _GLOBAL__sub_I__ZN3app135protobuf_AssignDesc_app1lite_2eprotoEv (app1lite.pb.cc:326)
==17112==    by 0x65E2CC: __libc_csu_init (in /home/vitaly/Projects/App1/App1/App1-gtest)
==17112==    by 0xE1F16FF: (below main) (libc-start.c:185)
==17112== 
...
==17112== 
==17112== LEAK SUMMARY:
==17112==    definitely lost: 384 bytes in 4 blocks
==17112==    indirectly lost: 0 bytes in 0 blocks
==17112==      possibly lost: 1,496 bytes in 24 blocks
==17112==    still reachable: 5,968 bytes in 67 blocks
==17112==         suppressed: 0 bytes in 0 blocks
==17112== Reachable blocks (those to which a pointer was found) are not shown.
==17112== To see them, rerun with: --leak-check=full --show-reachable=yes
==17112== 
==17112== ERROR SUMMARY: 8 errors from 8 contexts (suppressed: 2 from 2)
--17112-- 
--17112-- used_suppression:      2 dl-hack3-cond-1
==17112== 
==17112== ERROR SUMMARY: 8 errors from 8 contexts (suppressed: 2 from 2)
$

Conclusion

Sometimes using simple tools in trivial way we can achieve impressive results with small efforts. Also there are left some qty of complex and non-trivial leaks but we can ignore them for now.