My NSLU2 is running Debian Etch. Ive installed Asterisk from the Debian repositories (apt-get install asterisk), however its a memory hog out of the box. When (re)starting Asterisk, the load average would go up to 10.
Read about it here
You can reduce Asterisk memory footprint by unloading unnecessary modules (located in /usr/lib/asterisk/modules).
You can load/unload modules in /etc/asterisk/modules.conf.