dob.sk Blog

I’ve encountered unusual problem while dumping one postgresql database. Every try to run pg_dump resulted in this:


sam@cerberus:~/backup$ pg_dump -v -c js1 >x
pg_dump: Error message from server: ERROR: invalid memory alloc request size 18446744073709551613
pg_dump: The command was: COPY job.description (create_time, job_id, content, hash, last_check_time) TO stdout;
pg_dump: *** aborted because of error


or this


sam@cerberus:~/backup$ pg_dump -v -c --inserts js1 >x
pg_dump: Error message from server: ERROR: invalid memory alloc request size 18446744073709551613
pg_dump: The command was: FETCH 100 FROM _pg_dump_cursor
pg_dump: *** aborted because of error


Few weeks ago I’ve encountered almost fatal failure of two(!) disks in my RAID5 array – well really funny situation, and a bit uncomfortable way how to find out why not having monitoring and alarming system is a really bad idea
Fortunately no serious data damage happened, postgresql seemed to recover from this, without any problems and my databases worked fine – until I’ve tried to perform a database dump.

And now comes the important question, how to find out which table rows are incorrect (and pg_filedump says everything is OK) ?
And the answer is, use custom function:


CREATE OR REPLACE FUNCTION
find_bad_row(tableName TEXT)
RETURNS tid
as $find_bad_row$
DECLARE
result tid;
curs REFCURSOR;
row1 RECORD;
row2 RECORD;
tabName TEXT;
count BIGINT := 0;
BEGIN
SELECT reverse(split_part(reverse($1), '.', 1)) INTO tabName;

OPEN curs FOR EXECUTE ‘SELECT ctid FROM ‘ || tableName;

count := 1;
FETCH curs INTO row1;

WHILE row1.ctid IS NOT NULL LOOP
result = row1.ctid;

count := count + 1;
FETCH curs INTO row1;

EXECUTE ‘SELECT (each(hstore(‘ || tabName || ‘))).* FROM ‘
|| tableName || ‘ WHERE ctid = $1′ INTO row2
USING row1.ctid;

IF count % 100000 = 0 THEN
RAISE NOTICE ‘rows processed: %’, count;
END IF;
END LOOP;

CLOSE curs;
RETURN row1.ctid;
EXCEPTION
WHEN OTHERS THEN
RAISE NOTICE ‘LAST CTID: %’, result;
RAISE NOTICE ‘%: %’, SQLSTATE, SQLERRM;
RETURN result;
END
$find_bad_row$
LANGUAGE plpgsql;

It goes over all records in given table, expands them one by one – what will results in exception if some expansion occurs. Exception will also contain CTID of last correctly processed row. And next row with higher CTID will be the corrupted one.
Like this:


js1=# select find_bad_row('public.description');
NOTICE: LAST CTID: (78497,6)
NOTICE: XX000: invalid memory alloc request size 18446744073709551613
find_bad_row
--------------
(78497,6)
(1 row)

js1=# select * from job.description where ctid = ‘(78498,1)’;
ERROR: invalid memory alloc request size 18446744073709551613
js1=# delete from job.description where ctid = ‘(78498,1)’;
DELETE 1
js1=# select find_bad_row(‘job.description’);
NOTICE: rows processed: 100000
NOTICE: rows processed: 200000
NOTICE: rows processed: 300000
NOTICE: rows processed: 400000
NOTICE: rows processed: 500000
NOTICE: rows processed: 600000
NOTICE: rows processed: 700000
NOTICE: rows processed: 800000
find_bad_row
————–

(1 row)

Note: this function requires hstore postgresql extension – it is part of postgresql distribution, you may need to create it with:


CREATE EXTENSION hstore;


Records in this table are not that important, and I can restore them from external source – so I could delete this corrupted row. If you can’t you will have to play directly with data files – like described here – good luck

I’ve made two simple (but useful) disk drive monitoring scripts for collectd exec plugin. You can find them on http://devel.dob.sk/collectd-scripts/.

smartmon.sh

This script monitors SMART attributes of given disks using smartctl (smartmontools).

megamon.sh

This one monitors some interesting values of MegaRaid adapter physical drives using MegaCli tool.

Description how to use them can be found within scripts itself – enjoy

I’ve actually released log-malloc2 library for linux, that logs calls to memory allocation functions and should be very helpful when trying to locate memory leaks. It can be used without recompiling application, simply by preloading it using LD_PRELOAD .

Logged functions:

• malloc
• realloc
• calloc
• free
• memalign

Every function call  is logged with their parameters, amount of allocated/deallocated memory, total amount of allocated memory, copy of /proc/self/statm content and backtrace (call stack) if possible. Additionally function call counts is logged and printed out when application exits.

Typical usage:

LD_PRELOAD=../liblog-malloc2.so PROGRAM ARGUMENTS 1022>/tmp/malloc.log

More complete description of library, its usage, logging format and internals can be found in README file.

Library can be downloaded from project homepage. Actually there is no script helping with malloc log file analysis included, but logfile format is very simple and there should be no big problem write your own doing exactly what you want.

You should note that using this library harms application performance, is intensively uses allocate/deallocate functions. Consider logging to file located on tmpfs (in memory) filesystem, to improve logging IO throughput

Just created a quick patch against perl XML::LibXML module, that adds global external entity loader support. Till now it was only possible to have per instance entity loader, but this is not enough if you want i.e., XML::LibXSLT to also use yours entity loader for imports, and input callbacks doesn’t suit all your needs.

Usage is simple:

XML::LibXML::externalEntityLoader(\&_entity_handler);

where _entity_handler is subroutine like by option ext_ent_handler described in (http://search.cpan.org/dist/XML-LibXML/lib/XML/LibXML/Parser.pod#PARSER_OPTIONS).

Note: when you define global entity loader, per instance entity loader is simply ignored.

You can download this patch  from http://devel.dob.sk/patches/perl-XML::LibXML+global_entity_loader-0.1.diff. Just download XML::LibXML from cpan, patch it and install. I’ll try to push it to CPAN if possible

Just a short example C implementation of Oracle VARNUM/NUMBER type encoding, in case you could not use library functions. I didn’t found it anywhere when needed (and afterwards I found it doesn’t helps me at all due to little ‘problem’ in ora ) and Oracle documentation is pretty unclear about how-to do it.

typedef	struct
{
	uint8_t len;
	uint8_t exp;
	uint8_t man[20];
} VARNUM_t;
static void setVARNUM(VARNUM_t *varnum, int64_t value)
{
	int8_t digits = 0;
	uint64_t u_value = 0;
	/* init */
	memset(varnum, 0, sizeof(*varnum));
	varnum->len	= 1;	//start len
	/* check sign */
	if(value >= 0)
	{
		u_value		= value;
		varnum->exp	= trunc(log(u_value) / log(100)) + 128 + 65;
	}
	else
	{
		u_value		= value * - 1;
		varnum->exp	= trunc(log(u_value) / log(100)) + 128 + 65;
		varnum->exp	= ~varnum->exp;
	}
	/* count value digits */
	digits = trunc(log(u_value) / log(10));
	digits = ((digits / 2) + 1);
	/* mantisa */
	for(; digits >= 0 && varnum->len <= 20; digits--)
        {
                uint64_t v_tmp = 0;
                /* prevent INT overflow for too much digits */
                if(digits > 0)
		{
			uint64_t v_pow = powl(100, digits - 1);
			v_tmp = ((u_value / v_pow) / 100) % 100;
		}
		else
			v_tmp = u_value % 100;
		if(!v_tmp)
			continue;
		/* +1 for positive, subtract 101 for negative */
		v_tmp = (value < 0 ? (101 - v_tmp) : (v_tmp + 1));
                varnum->man[varnum->len - 1] = v_tmp;
		varnum->len++;
	}
	/* terminator byte for negative value */
	if(value < 0 && varnum->len <= 20) 	{
                varnum->man[varnum->len - 1] = 102;
		varnum->len++;
	}
	return;
}

NUMBER is just the same as VARNUM but without len member, thus is one byte shorter.

Note: this is just a quick implementation that works, there are some easy optimizations possible…