(Note: These programs are described as being for "AR7/Adam2" because that's what I wrote/hacked them for. It is possible that they may also be of use for more general cases. It is also possible that they don't actually work at all except for my particular version of the DSL600E. I don't know. But I think that at any rate they are potentially useful in enough cases to make it worth putting them up.)
Weird Squashfs Unfucker - Squashfs Extractor and Weird Kernel Image Extractor/Decompressor - ROM MTD Partition and Env Data Extractor - Penisiser - Kernel Binary Image Generator (and Checksum Calculator) - Tiny version of swapon - uClibcised nbd-client
This program might be of some use to owners of the Aztech DSL600E, or the Solwise SAR-600E, which is the same thing, who are trying to figure out just where the juddering fuck you can get a version of unsquashfs from that is capable of unpacking the mutant squashfs filesystem in its ROM. (See this page for details of how to extract that ROM data via JTAG in the first place.)
There are two major problems with such an endeavour. One is that the kernel support for squashfs in any version of Linux anyone is likely to be running these days smells like a budgie's penis, and when you try and mount the filesystem directly it moans that it can't handle a squashfs version less than 4.0. The other is that there seem to be as many custom variants of the LZMA compression algorithm floating about as there are applications that use it, and the one used for compressing the squashfs in a DSL600E ROM seems to have had something weird done to it apparently for no more reason than the pure sake of being a cunt.
There are also two more problems, one a minor one, and one a huge massive dog's cunt of a problem. The minor one is that it's an arseache to find out what historic version of squashfs includes an unsquashfs capable of unpacking a version 1.3r3 filesystem. Squashfs version 1.3r3 itself doesn't have unsquashfs at all, and when it does appear in the tree it is not at first capable of reading older filesystems (what the fuck is it with the squashfs guys being so shite at legacy support?) and you have to go up to version 3.4 to find one that does the trick.
The huge massive dog's cunt problem is so described because it is really fucking annoying: at one point there were programs out there that could handle this shit, but now they have all disappeared for stupid reasons. You find stuff like someone's website with various links to AR7-related software on it which all work except the links to the weird-squashfs-unfucker software, even though they're not even external links. The links to irrelevant stuff on other unrelated websites all work, but the ones to relevant stuff on the guy's own site don't. FFS. And most infuriating of all are the multitudinous search results that end up on the openwrt wiki at a page on which all the links to weird-squashfs-unfucker software - and only those links - are not actually links any more. Searching archive.org for older versions of the page in the hope of finding working links brings no joy, but it does find older versions explaining that the maintainers of the openwrt wiki are so bleeding clueless that when they moved from their old wiki to their current one, they weren't up to writing a script to move the attachments along with the text, and inviting random users to go to the old wiki (which still existed at that point) and copy the attachments to the new one by hand. Which of course didn't fucking happen, oh what a surprise. So you then search archive.org for the relevant pages of the old wiki, and find that yes, those pages did have valid links on them... but the utter cunts blocked archive.org from crawling them so all you get off them now is a redirect to the front page. And at this point you could really fucking strangle someone.
After hunting around for ages and finding that every apparent lead ended up following the same fucking useless pattern, I found I had exceeded my swear limit for the internet for the time being. But I still had a full quota of swears for programming, so I hacked and glued together a weird-squashfs-unfucker myself using bits of squashfs and AR7 firmware source code, plus other buggering about to do stuff like fix numerous compilation warnings and errors, and fix bugs like time_t having been a 4-byte type when the original code was written but it's 8 bytes now. The result seems to be OK both for gcc 5.2.1 on amd64, and 4.3.2 on i386.
I THINK the weirdnesses of this squashfs are a function of the use of the Adam2 bootloader on the AR7 platform, so there is a decent chance that it'll work for the various Netgear, Linksys, etc. devices that also use Adam2 on AR7, as well as for the DSL600E. And if it doesn't, there are debugging directives in the source files which can be enabled to make it spew all sorts of cack about its inner workings which ought to be some help in figuring out why not.
Usage is the same as the equivalent standard unsquashfs version; see unsquashfs -h for details.
Note that this hack does not provide a weird-squashfs mksquashfs, a weird-squashfs kernel module, or anything like that; it's unsquashfs only. This is because my attitude to weird non-standard shit for no reason that needs weird non-standard tools to do stuff with is to get rid of it and replace it with a standard version that you can use any old tool on, whether it's filesystems or screw-heads.
AR7/Adam2 Weird Squashfs Unfucker: unsquashfs-lzma-sar600e.tar.gz
The squashfs code is taken from squashfs versions 1.3r3, 3.0 and 3.4, copies also here: squashfs1.3r3.tar.gz, squashfs3.0.tar.gz, squashfs3.4.tar.gz. The LZMA decoder is taken from the Acorp AR7 firmware version 3.6.0D_GPL, copy also here: Acorp_3.6.0D_GPL.tar.bz2
This program extracts the squashfs and the kernel image from a dump of an MTD partition containing these items. Which one this is probably changes from device to device, but it is probably the longest one. It definitely begins with the string 4DTM. There is also a smaller partition which begins with the same string; that is a truncated version of the larger partition, and it won't be any good.
There is only the most perfunctory checking for the partition's validity, because it's only a quick hack, so expect weirdness if you give it the wrong file. However there is just about enough that it is OK to run mtdx *.bin on a whole set of MTD partition dumps and it'll just use the valid ones and ignore all the ones that aren't valid without chewing anything up or eating its own feet.
The kernel image in these things again uses some weird fucked-up version of LZMA compression which is similarly resistant to being decompressed with any standard tools. It is basically the same as the version used for the weird squashfs, but is weird in the headers as well as in the compression algorithm. This program attempts to decompress it using a hack of the same Acorp LZMA decoder mentioned above, and writes out the decompressed image. It does this by making potentially dodgy assumptions about information that would normally be extracted from the headers if they weren't fucking missing. If the decompression fails, it just writes out the compressed image anyway.
To make sense of the extracted squashfs, the Weird Squashfs Unfucker is required.
Usage is simply:
mtdx filename [filename...]
where filename refers to an MTD partition file. There are no switches or anything. Output filenames are derived by adding .(squashfs|lzma|kernel-img|unknown).bin to the input filename (shorn of its .bin suffix, if it has one), according to whether the data in the file is a squashfs, an LZMA compressed kernel, an uncompressed kernel image, or some other bloody thing that I don't know about.
AR7/Adam2 squashfs extractor and weird kernel image decompressor: mtdx-sar600e.tar.gz
(Note: "ROM extractor" means it extracts data from a ROM dump, not that it extracts a ROM dump from the AR7 device. That latter operation is described here.)
This is a quick and dirty program that lists the various items of Adam2 env data stored in the ROM, and (optionally) extracts the various MTD partitions, each to a separate file. (Note that these are weird and overlap in strange ways because of the way the interpretation of a "partition" changes as the boot process progresses.) Its usage summary is:
solx -f filename [-p prefix] [-a address]
filename (required) is the filename of
the ROM dump to read.
prefix is an optional prefix for the
filenames of the files into which the MTD partitions are extracted. It
acts as an output control: if it is not supplied, the partitions will not be extracted; the
program will simply list the Adam2 env data and exit.
address is the address of the
Adam2 env table, in hex. The default is 0x1f0000,
which is what the DSL600E ROM uses. Supplying -a 0
is useful for reading the env data from an already-extracted env data
partition.
The extracted MTD partition files are named prefix.mtdx-n.ssssss-ffffff.bin, where prefix is the prefix supplied with the -p option, x is the MTD partition number, n is the ordinal of this partition's entry in the env data table, and ssssss-ffffff are the start and finish+1 addresses of the partition, in hex.
I don't know whether any other bootloaders use the same env data format as Adam2. If they do, this program will probably work with them as well. The Adam2 env data format assumed by the program is a contiguous sequence of 128-byte blocks, each containing:
There is very little error checking, so if you enter silly parameters it is possible that your keyboard will melt and your genitals will turn into a cabbage.
AR7/Adam2 ROM extractor: solx.c.gz
This program penisises files in order to deal with the urjtag dog's cunt. For a full explanation see that link.
mkbin.sh is a shell script to package up a freshly-compiled ELF kernel image into the format required by the Adam2 bootloader, which goes like this:
Note that it is possible that the value to be subtracted from 0x100000000 when calculating checksum2 should be entry_point, not load_addr. The sample data I am using to work this shit out from is the ROM dump of my SAR-600E, in which those values are both the same. Still, it probably doesn't matter, since the Adam2 source code which is my other reference for working it out does not provide the information to work it out either; it stops processing once it knows entry_point and just ignores checksum2 - never even bothers to read it.
The notation Σ(image) can be expanded into the following more explicit C-function-style form:
unsigned int calc_sigma(unsigned char *image, unsigned int length) { unsigned int x; unsigned int sigma = 0; for (x = 0; x < length; x++) sigma += (unsigned int)(image[x]); return(sigma); }
chksum.c is a program to perform this calculation on a file, or part of a file, because it is slow as fuck doing it in shellscript. It needs to be compiled and the resulting binary placed in your PATH so that mkbin.sh can find it. Its usage summary is:
chksum [-s start] [-e end] [-l length] [-b bytes] [filename [filename...]]
If no filename is given, stdin is read.
Optional parameters start, end and length define a subsection of the file, over which the checksum is calculated. If all three of start, end and length are given, length is ignored. Values are in hex; the 0x prefix is optional.
bytes is the number of bytes of checksum required (ie. half the number of hex digits), starting from the low end. The checksum is printed in hex, without prefix, one line per input file.
The usage summary of mkbin.sh itself is:
mkbin.sh infile [outfile]
If outfile is not given, the output filename is derived from infile with any existing suffix chopped off and .bin tacked on to the end.
mkbin.sh uses PREFIXobjcopy and PREFIXnm to convert the image and to extract the load and execute addresses, where PREFIX is the usual gubbins tacked on to the front of the names of architecture-specific utilities to distinguish which architecture they are for; it is assumed to be mipsel-linux-gnu- by the script.
Note that this script does not include any facility to compress the kernel image. The reason it doesn't include it is that I couldn't be arsed, and the reason I couldn't be arsed is because the intended use is to produce kernel images for storage not in the limited space of the DSL600E's onboard ROM, but on the hard drive of a host machine from which a hacked version of the Adam2 bootloader (for which I will post the source code once I am sufficiently convinced it is usably stable) downloads the kernel image via FTP. It is a whole flaming bastard sight quicker and easier for experimenting with custom-built kernels to have the thing hoik them in to RAM over ethernet and not have to bother uploading them to flash ROM as normal, which takes bleeding forever.
As with most of these utilities I haven't been arsed to treble its size with error-checking code, so the usual risk of genital gangrene secondary to invalid input applies.
Kernel binary image generator: mkbin.tar.gz
This is a very small and rudimentary version of the swapon utility such as one might desire for use on highly-constrained systems - for example to allow such a system running one of the abovementioned network-loaded kernels to activate a swap device (using NBD or NFS, with the appropriate kernel patches applied to make these actually work properly). Normally this would be done using the swapon module of Busybox, or something of the kind, but the version of Busybox in the DSL600E's firmware unsurprisingly does not include this module, and it decided to be a cunt when I tried to compile it, so I figured the simplest answer was to put together the necessary few lines to invoke the swapon() system call myself, rather than pissing about for ages in the internals of some great pile of build scripts, which invariably smell of wee.
It does not read /etc/fstab; it just activates or deactivates the swap devices specified on the command line. If the binary, or a symlink via which it is called, has a name ending in swapon, it activates the swap devices; if it is called via a name ending in swapoff, it deactivates them. That's all there is to it. So it goes like this:
swapon devicename [devicename...] swapoff devicename [devicename...]
Included with the source code is a Makefile to cross-compile it, and to link it against uClibc (ordinary libc being far too fat). It doesn't install it; you need to copy it into place and create the appropriately-named symlink(s) yourself. It may be necessary to edit the first three lines of the Makefile to correspond with your setup.
CROSSPREFIX is the gubbins you need to tack on to the beginning of gcc and ld to get the cross-compilation versions instead of the native ones. In my case this is mipsel-linux-gnu.
UCLIBDIR is the path to whatever directory the uClibc shared libraries are in. In this directory, ld expects to find the uClibc versions of: libc itself, libc.so; the dynamic linker, ld.so; and the libc startup/shutdown files, crt[0in].o.
SOSUFF is uClibc's suffix for the abovementioned shared libraries, in my case .so.0.
(Installing cross-compilation tools is something I intend to write about later. For the moment suffice it to say that the current Debian multiarch thing is a fucking mess and a pain in the arse, but fortunately it is possible to dodge having to deal with it.)
Tiny version of swapon: uswapon.tar.gz
Of course, before you can activate a remote swap device, you have to have one in the first place. There does exist a set of patches to provide an NBD client in the kernel, but I haven't got them to work yet. The chap who developed them (on later kernels than I'm using) did so because the amount of extra code that needs to be added to the kernel is very small, but obtaining a similar result using a user-space client in an initrd means using a statically-linked client which comes out at not far short of a meg. And this is true. But nbd-client in dynamically linked form is only 22k odd, and if you dynamically link it against uClibc instead of the ordinary libc, the total amount of code required is still way smaller than with static linking.
So what we have here is a Makefile to compile nbd-client and link it against uClibc, along with a patch which fixes a couple of bugs that don't show up when you build it normally but do when you do this, adds a couple of bits and pieces that would normally be taken care of by the package configuration and build scripts, and also does some general tidying up. (The original nbd-client code mis-spells "queue" as "que". This was doing my nut because every time it said "que" at me I thought some bug was happening that made it drop bits of output. Also I kept thinking of Manuel. So I corrected the spelling in all the places the error occurred, and then of course having done that I had to correct it in all the preprocessor directives that also spelt it wrong just for the sake of consistency.)
So, the first thing to do is get the nbd-client source code. I got it from here: Debian Source Package: nbd (1:3.2-4~deb7u5). That link will probably break once stretch becomes stable, but another copy exists here: nbd_3.2-4~deb7u5_source.tar.gz. Only a handful of the source files are actually required for the present purpose, since we're not building the complete Debian package but just one binary.
Unpack that lot with dpkg-source -x nbd_3.2-4~deb7u5.dsc and apply the patch, then drop the Makefile into the build root. Check that the paths and shit in it are appropriate for your setup; as well as the same three definitions described for the uswapon Makefile above, there is a fourth one: INCDIR=/usr/$(CROSSPREFIX)/include. This is the directory which contains the cross-compilation versions of the standard C header files (stdio.h and all that lot), which is /usr/mipsel-linux-gnu/include on my setup.
Check also that your uClibc libraries include the ones needed for the linker stage. As well as libc.so, this requires the uClibc versions of libpthread.so and libdl.so.
(NOTE: on some versions of uClibc the whole bloody lot is part of libc.so and those extra two libraries don't exist as separate entities. If yours is like this, the obvious modification needs to be made to the Makefile.)
Having checked all that, it is now just a matter of typing make to create the uClibc-linked nbd-client binary. Do not use any of the package's own configuration scripts, or build the Debian package, or anything like that. They are not required for this very simple but nonstandard task; this hack's Makefile does all that is needed. (It probably won't break if you do run them, but it might do, and it's a waste of time and doesn't achieve anything. Conversely, if you do want to build the Debian package, do it from a clean and freshly-unpacked source tree, not from this hacked one.)
You now have an nbd-client binary which is ready to be added to your boot system. Tbe boot system must of course also include the abovementioned uClibc components and dynamic linker.
uClibcisation hack for nbd-client: nbd-client-uclibc.tar.gz
Back to AR7 Index
Back to Pigeon's Nest
Be kind to pigeons